correlation ID

[stishkin]

Have correlation ID travel through out the system.

• On every HTTP request from CLI - create a new correlation ID on service side and pass it through all requests until it is returned back to CLI. (we don't trust CLI, since it runs on user's machine - so don't let it control correlation ID)
• On every HTTP request from Agent - agent creates Correlation ID and sets it a request header. Service side will retrieve correlation ID and use it through out request until returning back to the agent. Agent uses same correlation ID as part of custom dimensions for all the logging as part of the execution flow.
• On every Timer request - create new Correlation ID in backend and use it throughout timer execution
• On every Queue request - Correlation ID should be part of the queue message, and it should be set as part of Custom Dimensions

AB#36057

[Porges]

Potentially we should be looking at OpenTelemetry for (distributed) tracing as well:

• https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview
• https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=net

[Porges]

And I think AppInsights might give us some of this for free, since it knows about each request. However we need to revisit this after we get rid of the -net suffix as there is a problem currently where it thinks that there are two .NET applications.

[Porges]

And I think AppInsights might give us some of this for free, since it knows about each request. However we need to revisit this after we get rid of the -net suffix as there is a problem currently where it thinks that there are two .NET applications.

The problem mentioned here was fixed by #2597

[tevoinea]

We should also output CorrelationId from CLI on error responses or when using verbose flag