Adds a tap command to the CLI tool (replaces cmd/development/substation-kinesis-tap)
Adds a read command to the CLI tool (replaces cmd/development/substation-file)
Updates all Jsonnet configs to use the sub external variable
Refactors other CLI command for consistency
Motivation and Context
This replaces multiple legacy tools with new CLI tool commands:
The substation-kinesis-tap app is now substation tap and is designed to be forward compatible with other data stream sources. For example, support for GCP Pub/Sub would use the flag --gcp-pub-sub-subscription and Kafka would use the flags --kafka-brokers--kafka-topic.
The substation-file app is now substation read.
All Jsonnet configurations now use an external variable, which fixes issues with relative imports and version inconsistency. substation build (and other commands) now directly import the content of substation.libsonnet without the user needing to provide an import. The build command should be considered the recommended method for building configs due to these changes.
Other CLI commands were refactored to reduce code and fix some minor issues discovered during testing.
This requires an update to the website docs.
How Has This Been Tested?
These new commands were tested:
% ~/go/bin/substation tap -h
% ~/go/bin/substation tap help
% ~/go/bin/substation tap --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx
% ~/go/bin/substation tap --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx
% ~/go/bin/substation tap test.libsonnet --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx
% ~/go/bin/substation tap test.libsonnet --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx --offset earliest
% SUBSTATION_DEBUG=1 ~/go/bin/substation tap test.libsonnet --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx --offset earliest
Description
tapcommand to the CLI tool (replacescmd/development/substation-kinesis-tap)readcommand to the CLI tool (replacescmd/development/substation-file)subexternal variableMotivation and Context
This replaces multiple legacy tools with new CLI tool commands:
substation-kinesis-tapapp is nowsubstation tapand is designed to be forward compatible with other data stream sources. For example, support for GCP Pub/Sub would use the flag--gcp-pub-sub-subscriptionand Kafka would use the flags--kafka-brokers--kafka-topic.substation-fileapp is nowsubstation read.substation build(and other commands) now directly import the content ofsubstation.libsonnetwithout the user needing to provide an import. Thebuildcommand should be considered the recommended method for building configs due to these changes.This requires an update to the website docs.
How Has This Been Tested?
These new commands were tested:
% ~/go/bin/substation tap -h% ~/go/bin/substation tap help% ~/go/bin/substation tap --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx% ~/go/bin/substation tap --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx% ~/go/bin/substation tap test.libsonnet --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx% ~/go/bin/substation tap test.libsonnet --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx --offset earliest% SUBSTATION_DEBUG=1 ~/go/bin/substation tap test.libsonnet --aws-kinesis-data-stream arn:aws:kinesis:xx:xx:stream/xx --offset earliest% ~/go/bin/substation read --file main.go% ~/go/bin/substation read --http https://github.com% ~/go/bin/substation read --aws s3://xx/...Refactored commands were also individually checked.
Types of changes
Checklist: