Investigate Azure Event Grid Integration

[arnejduranovic]

DevSecOps Issue

Summary

16237 created a new Azure Function using Blob Trigger to support SFTP ingestion into RS. The Blob Trigger documentation (linked above) notes the following:

Warning Storage logs are created on a "best effort" basis. There's no guarantee that all events are captured. Under some conditions, logs may be missed.

While we don't think we will see this issue any time soon, due to our relatively low volume, we would like to move to the more reliable Azure Event Grid alternative as we scale up.

The Azure Event Grid documentation shows the following diagram, the top flow being our exact use case:

The request for DevOps is to:

1. Look into what it would take to create Azure Event Grid resource for all environments and give an estimate of size of work
2. Estimate how long it would take to set Azure Event Grid up for staging (and connect it to the external SFTP bucket) so app devs can try connecting to it
3. Calculate cost estimates

Environment

• [ ] Local
• [ ] Dev
• [x] Stage
• [x] Prod

Priority

• [ ] Critical - affecting prod systems
• [ ] Major - blocking major functionality, deployment, etc
• [ ] Minor - improvements, bug fixes
• [x] Nice-to-have - feature request

Blocks the following

Blocked by the following

Contact

Team: Platform

Slack Team Channel: prime-reportstream-pipeline

Slack Contact (Full Name): Arnej

Technical Team Lead: Arnej

Definition of Done

• [ ] Resource cost and full implementation time estimate created and shared
• [ ] If viable: Event Grid resource created in Staging and hooked up to SFTP

[devopsmatt]

@arnejduranovic - https://docs.google.com/document/d/1sdgEx9ssbQypzo8naMkEhBL9NLjvWzTA4KrYeRYZ44w/edit?usp=drive_link