Messaging for processing events

[JimCircadian]

To support external integration, we need to extend the ETL processing so that we can trigger external systems that subscribe to events in the ETL layer.

• [x] Provision event grid / integration layer that takes events from the following
  • [x] Arrival of a new input in blob storage (custom topic etl-forecast-arrival)
  • [x] Commencement and completion of processing (custom topic etl-forecast-processing)

The use of the event grid is slightly Azure specific, but should be relatively portable and is probably better than deferring to service bus usage at this stage, as it can be limited to Terraform provisioning and does not rely on a predefined message structure. The client is solely responsible for subscribing and choosing how to handle events.

[JimCircadian]

The output of events can't really be via a system topic as is currently implemented, so this should feed a service bus which can buffer the messages as well.

[JimCircadian]

These items are provisioned as infrastructure, need to be implemented in the code / expanded as per the limitations identified. Weird tracking of issues but it'll all come out in the wash 😄