Track processed events in Azure provider

MarcAstr0 commented 4 months ago

Description

The changes in this PR aim to correct a behavior noticed in the Azure provider, where the Cosmos DB change feed would get triggered more than once for the same event, resulting in the same event being processed twice. This behavior has been seen to happen during load-balancing scenarios.

To correct this duplicate-event behavior, a new processed events container is added to Cosmos DB to store the IDs of events that the event processor has processed. The processor will query this container for the event IDs and verify the event has not been processed (i.e., it can't find its ID in this new processed-events container). If an event with the same ID is found, to avoid duplication, the event processor will ignore it and log a warning indicating the duplication.

The IDs store in the processed events container have a time-to-live (TTL) and will get cleared automatically.

Changes

Modifies framework-provider-azure-infrastructure package adding the new container for processed events.
Adds a configurable processedEventsTtl to BoosterConfig for setting the TTL of the event IDs in the processed events container (defaults to 300 seconds, i.e., 5 minutes).
Makes the necessary changes to the BoosterEventProcessor in the framework-core package for querying and storing the processed event IDs.
Modifies the adapters in the framework-provider-azure package for interacting with the processed events container.
For backwards compatibility, adds dummy functions to the (now deprecated) AWS and the local provider packages that will skip any verification for duplicate events since this behavior hasn't been reported in these providers.

Checks

[x] Project Builds
[x] Project passes tests and checks
[x] Updated documentation accordingly

what-the-diff[bot] commented 4 months ago

PR Summary

Tracking of Processed Events Added a new JSON file to keep track of processed events in Azure. This ensures the repetition of events is avoided, enhancing performance and efficiency.
Update in the Core Framework Added new functionalities in the core framework which allow for filtering and storing of unprocessed events. These changes ensure that only new events are dispatched to event handlers.
Addition of Event Storage Utility Added new methods in the event storage service to handle storing and searching for processed events. This facilitates efficient event tracking and retrieval.
New Tests for Event Processing Incorporated a new test case to verify if already processed events are filtered before dispatch to event handlers, enhancing the accuracy of event dispatches.
AWS and Azure Infrastructure Update For both providers, added dummy methods that return an empty array representing processed events. These methods ensure compatibility across different providers.
Azure Infrastructure Enhancement By creating a container for storing processed events, the improved Azure infrastructure facilitates the tracking and retrieval of processed events.
Improved Event Adapters For both AWS and Azure, new methods were introduced for storing and fetching processed events. This results in better management of events and ensures processed events can be easily retrieved.
Configurations and Interface Enhancements Several new methods, interfaces, and properties have been added to different files. These additions provide new features such as processed events storage and search, enhancing the flexibility and functionality of the booster configuration class.
Minor Changes Small changes have been made to some files, assuring a better and more refined coding structure.

MarcAstr0 commented 4 months ago

/integration sha=cffd25137016b4b4eae2032990dfc81c8e8fee6f