This is the refactor for v1.0.0 that includes the major changes described below.
⚠ BREAKING CHANGES
cmd/development/substation refactored into cmd/client/file/substation
condition.Inspector is no longer in the public API
condition inspectors refactored into individual functions
JSON Schema inspector removed
Inspectors no longer directly support negation
config.Capsule refactored into message package
config.Channel is no longer in the public API
process package refactored into transform package
Count processor removed
Flatten processor removed
IP Database processor removed
Sumo Logic support removed (replaced)
Group processor removed (replaced)
Processors (now Transfoms) no longer directly support conditions
internal/transform package removed
internal/sink package removed
proto removed
Secrets are now explicitly retrieved and put into the Secrets Store using the utility_secret transform
The enrich_kv_store_set transform had it's object.key and object.set_key behavior flipped (key is now the value put into the KV, set_key is now the key used in the KV)
The send_http transform is now send_http_post
All TTLOffset settings are now strings instead of integers (e.g., "15m")
Metrics are now generated using the meta_metric_duration and utility_metric_count transforms
Features
Added substation package
Added message package
Updated applications to use new concurrency and data processing model
Updated all send transforms to support batching data
Added Kinesis Data Firehose support to cmd/aws/lambda/substation
Added meta_negate inspector to condition package
Added meta_err transform to transform package
Added meta_switch transform to transform package
Added string_append transform to transform package
Added string_uuid transform to transform package
Added utility_delay transform to transform package
Added utility_err transform to transform package
Added support for non-aggregated data to AWS Kinesis Data Stream transform
Added region and assume role support to all AWS transforms
Removed IAM modules in build/terraform/aws/
Added build/scripts/config/format.sh
Added build/scripts/terraform/format.sh
Added shorthand to build/config/substation.libsonnet
Added build/config/substation_test.jsonnet
Downgraded go.mod and development containers to Go 1.19
Upgraded application containers to Go 1.21
Refactored all examples/
Added utility_secret transform
All transform object handling patterns (object.source_key) return the input message if the retrieved key value does not exist
Secrets Store AWS Secrets Manager backend supports AWS and retry configuration
KV Store AWS DynamoDB backend supports AWS and retry configuration
Added example for summarizing multiple events into a single event
Added example for using MaxMind with the KV transform
Added meta_metric_duration and utility_metric_count transforms with examples
All Time values now target nanosecond precision as the default
Added array_zip transform
IAM roles and policies in Terraform use randomized names
Added Terraform modules and examples for ingesting data from CloudWatch log groups
Motivation and Context
See this discussion for motivation and context and the updated documentation for more information. The focus of this release is to simplify the public API and introduce quality of life improvements. I don't think it's possible to provide a migration path from pre-release to this release, but I'm open to ideas.
Merging this PR will bump the main release to v1.0.0.
How Has This Been Tested?
All unit tests are passing
All examples have been integration tested in AWS
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
Description
This is the refactor for v1.0.0 that includes the major changes described below.
⚠ BREAKING CHANGES
cmd/development/substation
refactored intocmd/client/file/substation
condition.Inspector
is no longer in the public APIcondition
inspectors refactored into individual functionsconfig.Capsule
refactored intomessage
packageconfig.Channel
is no longer in the public APIprocess
package refactored intotransform
packageinternal/transform
package removedinternal/sink
package removedproto
removedutility_secret
transformenrich_kv_store_set
transform had it's object.key and object.set_key behavior flipped (key is now the value put into the KV, set_key is now the key used in the KV)send_http
transform is nowsend_http_post
TTLOffset
settings are now strings instead of integers (e.g., "15m")meta_metric_duration
andutility_metric_count
transformsFeatures
substation
packagemessage
packagecmd/aws/lambda/substation
meta_negate
inspector tocondition
packagemeta_err
transform totransform
packagemeta_switch
transform totransform
packagestring_append
transform totransform
packagestring_uuid
transform totransform
packageutility_delay
transform totransform
packageutility_err
transform totransform
packagebuild/terraform/aws/
build/scripts/config/format.sh
build/scripts/terraform/format.sh
build/config/substation.libsonnet
build/config/substation_test.jsonnet
go.mod
and development containers to Go 1.19examples/
utility_secret
transformmeta_metric_duration
andutility_metric_count
transforms with examplesarray_zip
transformMotivation and Context
See this discussion for motivation and context and the updated documentation for more information. The focus of this release is to simplify the public API and introduce quality of life improvements. I don't think it's possible to provide a migration path from pre-release to this release, but I'm open to ideas.
Merging this PR will bump the main release to v1.0.0.
How Has This Been Tested?
Types of changes
Checklist: