icenet-ai / icenet-event-processor

IceNet event processing engine
MIT License
0 stars 0 forks source link

Replace `azurerm_eventgrid_system_topic_event_subscription` for data hookup #3

Closed JimCircadian closed 1 year ago

JimCircadian commented 1 year ago

The icenet-etl issue is going to dispatch via a service bus (or similar replacement) that can support the subscription on this side. terraform/processing/subscriptions.tf then needs changing appropriately, as the system topic subscription just refuses to provision, so it's likely that it is limited to in-resource group provisioning (unlike custom and domain topics):

│ Error: creating/updating System Topic Event Subscription: (Event Subscription Name "sub-etl-icenetetldev-forecast-topic" / System Topic Name "/subscriptions/[[REDACTED]]/resourceGroups/rg-icenetetldev-data/providers/Microsoft.EventGrid/systemTopics/etl-icenetetldev-forecast-topic" / Resource Group "rg-icenetevtdev-evtproc"): eventgrid.SystemTopicEventSubscriptionsClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceNotFound" Message="The Resource 'Microsoft.EventGrid/systemTopics/subscriptions' under resource group 'rg-icenetevtdev-evtproc' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"
│ 
│   with module.processing.azurerm_eventgrid_system_topic_event_subscription.data_subs["/subscriptions/[[REDACTED]]/resourceGroups/rg-icenetetldev-data/providers/Microsoft.EventGrid/systemTopics/etl-icenetetldev-forecast-topic"],
│   on processing/subscriptions.tf line 1, in resource "azurerm_eventgrid_system_topic_event_subscription" "data_subs":
│    1: resource "azurerm_eventgrid_system_topic_event_subscription" "data_subs" {
JimCircadian commented 1 year ago

Testing the subscription with the following filters, as too many events are getting through:

JimCircadian commented 1 year ago

This has been solved within the icenet-etl repository, with additional filters ensuring that topics don't receive unnecessary events.