Redo export of metrics, traces and logs

[arealmaas]

Introduction

A short and concise description of the functionality this issue addresses.

Description

After a meeting with platform about the steps we need to take to test out Grafana, we discussed some other points related to that.

• We should discuss whether we should fully transition to OpenTelemetry with the collector pattern and remove App Insights-specific components from our code. With this approach, we can use the Container Apps OpenTelemetry agent to send metrics and logs to App Insights and potentially other sources.
• They’re interested in metrics from us, so the on-call team, for instance, can get a holistic view of the status across all systems in Altinn.
• We likely don’t need to do much with traces and logs, as App Insights is already handling these effectively. However, for metrics, we should consider testing the Grafana instance they have. A prerequisite for this is setting up an Azure Monitor Workspace to send metrics to, which they would then use as a datasource.

Implementation

• Add an Azure Monitor Workspace via IaC.
• Use the OT agent in Container Apps (see documentation) to send metrics to Azure Monitor and logs to App Insights.
• Gain access to Grafana and add Azure Monitor as a datasource.
• Increase the coverage of metrics and alerting. Ensure that we get notifications about crashing applications etc.

### Tasks
- [ ] https://github.com/digdir/dialogporten/issues/1462
- [ ] https://github.com/digdir/dialogporten/issues/1463
- [ ] https://github.com/digdir/dialogporten/issues/1465
- [ ] https://github.com/digdir/dialogporten/issues/1466
- [ ] Investigate the possibility to export the logs/traces from container apps(system logs and stdout) into application insights

### Threat modelling
- [ ] Does this change introduce any potential security issues?

Acceptance criteria

GIVEN ... WHEN .... THEN ...

GIVEN ... WHEN .... THEN ...