This is the refactor for v1.0.0 that includes the major changes described below.
⚠ BREAKING CHANGES
cmd/development/substation refactored into cmd/client/file/substation
condition.Inspector is no longer in the public API
condition inspectors refactored into individual functions
JSON Schema inspector removed
Inspectors no longer directly support negation
config.Capsule refactored into message package
config.Channel is no longer in the public API
process package refactored into transform package
Count processor removed
Flatten processor removed
IP Database processor removed
Sumo Logic support removed (replaced)
Group processor removed (replaced)
Processors (now Transfoms) no longer directly support conditions
internal/transform package removed
internal/sink package removed
proto removed
Secrets are now explicitly retrieved and put into the Secrets Store using the utility_secret transform
The enrich_kv_store_set transform had it's object.key and object.set_key behavior flipped (key is now the value put into the KV, set_key is now the key used in the KV)
The send_http transform is now send_http_post
All TTLOffset settings are now strings instead of integers (e.g., "15m")
Metrics are now generated using the meta_metric_duration and utility_metric_count transforms
Features
Added substation package
Added message package
Updated applications to use new concurrency and data processing model
Updated all send transforms to support batching data
Added Kinesis Data Firehose support to cmd/aws/lambda/substation
Added meta_negate inspector to condition package
Added meta_err transform to transform package
Added meta_switch transform to transform package
Added string_append transform to transform package
Added string_uuid transform to transform package
Added utility_delay transform to transform package
Added utility_err transform to transform package
Added support for non-aggregated data to AWS Kinesis Data Stream transform
Added region and assume role support to all AWS transforms
Removed IAM modules in build/terraform/aws/
Added build/scripts/config/format.sh
Added build/scripts/terraform/format.sh
Added shorthand to build/config/substation.libsonnet
Added build/config/substation_test.jsonnet
Downgraded go.mod and development containers to Go 1.19
Upgraded application containers to Go 1.21
Refactored all examples/
Added utility_secret transform
All transform object handling patterns (object.source_key) return the input message if the retrieved key value does not exist
Secrets Store AWS Secrets Manager backend supports AWS and retry configuration
KV Store AWS DynamoDB backend supports AWS and retry configuration
Added example for summarizing multiple events into a single event
Added example for using MaxMind with the KV transform
Added meta_metric_duration and utility_metric_count transforms with examples
All Time values now target nanosecond precision as the default
Added array_zip transform
IAM roles and policies in Terraform use randomized names
Added Terraform modules and examples for ingesting data from CloudWatch log groups
Motivation and Context
See this discussion for motivation and context and the updated documentation for more information. The focus of this release is to simplify the public API and introduce quality of life improvements. I don't think it's possible to provide a migration path from pre-release to this release, but I'm open to ideas.
Merging this PR will bump the main release to v1.0.0.
How Has This Been Tested?
All unit tests are passing
All examples have been integration tested in AWS
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
Description
This is the refactor for v1.0.0 that includes the major changes described below.
⚠ BREAKING CHANGES
cmd/development/substationrefactored intocmd/client/file/substationcondition.Inspectoris no longer in the public APIconditioninspectors refactored into individual functionsconfig.Capsulerefactored intomessagepackageconfig.Channelis no longer in the public APIprocesspackage refactored intotransformpackageinternal/transformpackage removedinternal/sinkpackage removedprotoremovedutility_secrettransformenrich_kv_store_settransform had it's object.key and object.set_key behavior flipped (key is now the value put into the KV, set_key is now the key used in the KV)send_httptransform is nowsend_http_postTTLOffsetsettings are now strings instead of integers (e.g., "15m")meta_metric_durationandutility_metric_counttransformsFeatures
substationpackagemessagepackagecmd/aws/lambda/substationmeta_negateinspector toconditionpackagemeta_errtransform totransformpackagemeta_switchtransform totransformpackagestring_appendtransform totransformpackagestring_uuidtransform totransformpackageutility_delaytransform totransformpackageutility_errtransform totransformpackagebuild/terraform/aws/build/scripts/config/format.shbuild/scripts/terraform/format.shbuild/config/substation.libsonnetbuild/config/substation_test.jsonnetgo.modand development containers to Go 1.19examples/utility_secrettransformmeta_metric_durationandutility_metric_counttransforms with examplesarray_ziptransformMotivation and Context
See this discussion for motivation and context and the updated documentation for more information. The focus of this release is to simplify the public API and introduce quality of life improvements. I don't think it's possible to provide a migration path from pre-release to this release, but I'm open to ideas.
Merging this PR will bump the main release to v1.0.0.
How Has This Been Tested?
Types of changes
Checklist: