Provide an anomaly detection mechanism in the dashboard

[GoutsmitSam]

Scenario

• Alert the user if an unusual amount of errors and / or messages occur during a certain timeperiod.
• Altering can be either a direct visualisation in the dashboard or a new type of Custom Alert within Dashboard (or both)

Input:

• Usual number of errors and messages per flow per period (to refine)

Output

• Visualisaton in the dashboard
• New type of Custom Alert

[GoutsmitSam]

07/11 During meeting with Ruud en Perrine, two broad options were discussed how this could be set up

• Through ADX (Azure Data Explorer)
• Through TIMEGEN-1 model in Azure AI Studio (https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-timegen-1)

Some more research is needed to determine which one would be best feature- and pricewise.

[grik001]

I have attached two files: MergedParentFlow and WorkFlowEvent.

For the current alerts, we push the flowId and timestamp to LogAnalytics. A heartbeat signal is sent every x minutes to indicate that the flow is active. Other logs are only sent when an error is detected in the ImportJob for each flow individually.

We have highlighted the most important fields in both files, but we recommend reviewing the entire schema in detail. Please don’t hesitate to reach out if you have any questions.

MergedParentFlow:

This is the parent flow which in the database would be a single document in the Flows Collection(single row in the dashboard Flow page). We currently ONLY use the merged flows for both heartbeat and error detection

StatusID - used to identify if the flow has failed, completed or is active ChainID - used as a unique identifer for the document and to group Workflow events together EventTimestamp - the start time that the flow was triggered FlowId - an array of matching FlowIds

WorkFlowEvent:

This represents the chain of Logic Apps associated with a specific parent flow. We’ve included this sample in case you would like to base your anomaly detection on more granular data.

EventTimestamp - the start time that the LogicApp was triggered StatusID - used to identify if the specific LogicApp has failed, completed or is active

Feel free to ping us if you have further questions.

WorkFlowEvent.txt MergedParentFlow.txt