search

Altinn / app-template-dotnet

Altinn Studio application template
1 stars 8 forks source link

Missing logs in Application Insights #215

Closed mlqn closed 2 weeks ago

mlqn commented 3 months ago

Description of the bug

Exceptions that occur in Program.cs, during the deployment of an application, are not logged in Application Insights, making it difficult for users to debug their app.

These logs can still be found in the ContainerLog table of the Kubernetes service, but our users don't have access to AKS.

Steps To Reproduce

  1. Create a new application in Studio
  2. Open the repository in Gitea
  3. Add throw new System.Exception(); in App/Program.cs
  4. Commit changes
  5. Fetch changes in Studio
  6. Create a new release
  7. Deploy the new release. This release should fail to deploy and throw an exception

https://github.com/Altinn/app-frontend-react/assets/24462611/5b9b488a-b37a-441b-b872-6b05b4ab8db9

  1. Open logs in Application Insights and notice that the exception is missing
ai-logs
  1. Open logs in AKS
aks-logs
bjosttveit commented 3 months ago

Does this issue belong in app-frontend-react? Maybe it should be moved to app-template-dotnet or app-lib-dotnet instead, depending on where the problem lies?

mlqn commented 3 months ago

@bjosttveit Nice catch, thanks! I moved it to app-template-dotnet as I mentioned the Program.cs file, but it might be an issue for app-lib-dotnet, I'm not sure. Feel free to move it to correct repo

martinothamar commented 3 months ago

I don't think it is possible to make logs be shipped through App Insights SDK (or OTel SDK for that matter) consistently in this way. The only place we know that logs will appear regardless of where or how early the code crashes is stdout/stderr from the container/process. In k8s these are stored on the host filesystem, and this is typically what log-scrapers (such as the one shipping to ContainerLog) uses to reliably capture all output. The AppInsights SDK (and OTel SDK) does flush telemetry on process exit, but only for clean exits.

Examples include

The case our users will most commonly run into I think are exceptions originating from configuration on startup. For example code that is run in RegisterCustomAppServices currently runs before AddAltinnAppServices which is currently what adds AppInsights/OTel.

Some options I can think of (that are not mutually exclusive):

mlqn commented 1 month ago

Owners will soon have access to two new Log Analytics workspaces: application-{org}-{env}-law and operational-{org}-{env}-law, which will give them access to all logs. App logs will go to application-{org}-{env}-law, and infra/system logs will go to operational-{org}-{env}-law. So this issue might no longer be particularly relevant 🤔, except perhaps for the suggestion to move the initialization of AppInsights/OTel SDK earlier in Program.cs. 🤷‍♂️

martinothamar commented 1 month ago

Yeah I guess the original issue is reoslved. I think it would be beneficial to pull more app observability into the Studio UI over time, perhaps something for 2025 😄