This PR adds a data warehouse stream client that can be used for multiple things, but is presently and solely concerned with publishing billing events to the stream.
The client is unused by any running service in builds using this commit. However, it is ready to be used and will likely be done so with opt-in configuration from SDF's own configuration options.
When a client is not present, the billing events server will run in "no-op" mode, which consumes the message off the NATS Jetstream stream (preventing overload), logs that the message was received and then does a whole lotta nothing.
As a result of these changes, this commit adds aws-sdf-firehose and aws-config third party dependencies to the repository. They are scoped purely to the new data-warehouse-stream-client crate, which is intentional.
Disclaimer
We are avoiding creating Server and Config structs for now because the billing events "server" is really a "task". It's immediately created from its configuration options and doesn't own its long running future.
That being said, when and if it becomes its own service, we should employ the pattern established by veritech, pinga and rebaser with si-service.
Additional Changes
This PR also adds a location for billing events. For now, we hardcode this value, but will tune it and use the correct value when we add configuration options to SDF for the billing events server to deliver events.
Description
This PR adds a data warehouse stream client that can be used for multiple things, but is presently and solely concerned with publishing billing events to the stream.
The client is unused by any running service in builds using this commit. However, it is ready to be used and will likely be done so with opt-in configuration from SDF's own configuration options.
When a client is not present, the billing events server will run in "no-op" mode, which consumes the message off the NATS Jetstream stream (preventing overload), logs that the message was received and then does a whole lotta nothing.
As a result of these changes, this commit adds
aws-sdf-firehoseandaws-configthird party dependencies to the repository. They are scoped purely to the newdata-warehouse-stream-clientcrate, which is intentional.Disclaimer
We are avoiding creating
ServerandConfigstructs for now because the billing events "server" is really a "task". It's immediately created from its configuration options and doesn't own its long running future.That being said, when and if it becomes its own service, we should employ the pattern established by veritech, pinga and rebaser with
si-service.Additional Changes
This PR also adds a location for billing events. For now, we hardcode this value, but will tune it and use the correct value when we add configuration options to SDF for the billing events server to deliver events.