Feedback on Container Apps Monitoring

slavizh commented 2 years ago

Please provide us with the following information:

This issue is a: (mark with an x)

[x] bug report -> please search issues before submitting
[x] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description

• Monitoring is set at Container Apps environment level rather at container app. I do not know the technical details why that is, but it would be better if it can be set per container app. • Instead of using diagnostic settings or Data collection rules to configure ingestion of logs this is done via integration within environment resource API. It should be via diagnostic settings or better via data collection rule at container app resource. This also prevents option to choose which logs to send. • _ResourceId column is missing from the logs, so you do not know the ID of the container app resource that is emitting these logs. Every log that comes from Azure resource should have the _ResourceId column in the logs that emits. • In the Container Apps System logs I am missing the which is the source of the event which is described here: https://docs.microsoft.com/en-us/azure/container-apps/observability?tabs=bash#container-apps-system-logs but not present as column in the logs • Instead of using tables that end with prefix _CL which are custom tables why don’t you have your own tables in Log Analytics that are pre-defined and not custom? • For the console logs I couldn’t generate stderr log. The only log that was generated even when there were errors was stdout. Either this is not possible, or it is not documented how to do that. My expertise is not in developing apps but still it should be clear enough how to do that. • Metrics on container apps cannot be exported to Log Analytics or third party as there is no option to configure diagnostic settings. This also means that documentation wrongly states that metrics are exportable: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftappcontainerapps • As I understand with container apps, we set specific number of cores and GBs for each app. If that is a static value, why do we do not have CPU and Memory usage in percentage values? It is a lot easier to alert on percentage rather specifying a slightly lower number of nano cores than what is assigned to the app. • Errors in ContainerAppSystemLogs_CL have value info for column Level even when it is error which makes https://docs.microsoft.com/en-us/azure/container-apps/observability?tabs=bash#container-apps-system-logs misleading. • Sometimes time_t is filled other times it is time_s column.

• You might also want to avoid naming columns by starting with _ character. Usually standard/hidden columns in LA start that way: https://docs.microsoft.com/en-us/azure/azure-monitor/logs/log-standard-columns • ContainerAppName_s is not populated in all records in ContainerAppSystemLogs_CL table.

Steps to reproduce

N/A

Expected behavior [What you expected to happen.] N/A

Actual behavior [What actually happened.] N/A

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

Ex. Did this issue occur in the CLI or the Portal?

cwe1ss commented 2 years ago

I also see errors with Level=info in my ContainerAppSystemLogs_CL. Here's an example:

Column	Value
TenantId	***
SourceSystem	RestAPI
TimeGenerated [UTC]	2022-09-05T18:22:14.12Z
time_t [UTC]	2022-09-05T18:18:37.165Z
Error_s	2/2 Failed:daprdContainerCrashing
_timestamp_d	1662401917.16564
Level	info
RevisionName_s	***
Log_s	Error provisioning revision.
EnvironmentName_s	***
Type	ContainerAppSystemLogs_CL

kendallroden commented 2 years ago

• Instead of using diagnostic settings or Data collection rules to configure ingestion of logs this is done via integration within environment resource API. It should be via diagnostic settings or better via data collection rule at container app resource. This also prevents option to choose which logs to send. Diagnostic settings coming in the near term • _ResourceId column is missing from the logs, so you do not know the ID of the container app resource that is emitting these logs. Every log that comes from Azure resource should have the _ResourceId column in the logs that emits. _ResourceId column will be included when we move to diagnostic settings which is coming very soon • Metrics on container apps cannot be exported to Log Analytics or third party as there is no option to configure diagnostic settings. The doc referenced is not specific to aca therefore it will not align to our specific implementation but as mentioned we are moving to diagnostic settings

We will look into your other comments. thanks for reporting

ezYakaEagle442 commented 1 year ago

I tried to deploy an App with

// https://docs.microsoft.com/en-us/azure/templates/microsoft.insights/diagnosticsettings?tabs=bicep
resource appInsDgsAdminServer 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'dgs-${appName}-send-${adminServerContainerAppName}-logs-and-metrics-to-log-analytics'
  scope: AdminServerContainerApp
  properties: {
    logAnalyticsDestinationType: 'AzureDiagnostics'
    workspaceId: logAnalyticsWorkspace.id
    logs: [
      {
        category: 'ApplicationConsole'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
      {
        category: 'SystemLogs'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
      {
        category: 'IngressLogs'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }    
    ]
    metrics: [
      {
        category: 'AllMetrics'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
    ]
  }
}

Hit "The resource type 'microsoft.app/containerapps' does not support diagnostic settings"


{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"BadRequest\",\r\n            \"message\": \"{\\r\\n  \\\"code\\\": \\\"ResourceTypeNotSupported\\\",\\r\\n  \\\"message\\\": \\\"The resource type 'microsoft.app/containerapps' does not support diagnostic settings.\\\"\\r\\n}\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"BadRequest\\\",\\r\\n    \\\"message\\\": \\\"\\\"\\r\\n  }\\r\\n}\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"}]}}

slavizh commented 1 year ago

@ezYakaEagle442 container apps do no support currently diagnostic settings. Integration happen trough property on the app environment resource currently.

ezYakaEagle442 commented 1 year ago

@slavizh https://learn.microsoft.com/en-us/azure/container-apps/log-options#diagnostic-settings shows it indeed in the Portal, What property to set in Bicep in the 'Microsoft.App/managedEnvironments@2022-06-01-preview'?

Bicep doc shows: Cluster configuration which enables the log daemon to export app logs to a destination. Currently only "log-analytics" is supported.

cwe1ss commented 1 year ago

You can use the new azure-monitor based logging like this in Bicep:

The managed environment must use properties.appLogsConfiguration.destination: 'azure-monitor':

resource appEnv 'Microsoft.App/managedEnvironments@2022-03-01' = {
  name: appEnvName
  location: location
  tags: tags
  properties: {
    appLogsConfiguration: {
      destination: 'azure-monitor'
    }
    daprAIConnectionString: monitoringAppInsights.properties.ConnectionString
    daprAIInstrumentationKey: monitoringAppInsights.properties.InstrumentationKey
    vnetConfiguration: {
      internal: false
      infrastructureSubnetId: networkSubnetApps.id
    }
  }
}

And these are the diagnostic settings to send console logs and system logs to a Log Analytics workspace:

resource appEnvDiagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: diagnosticSettingsName
  scope: appEnv
  properties: {
    workspaceId: platformLogs.id
    logs: [
      {
        category: 'ContainerAppConsoleLogs'
        enabled: true
      }
      {
        category: 'ContainerAppSystemLogs'
        enabled: true
      }
    ]
  }
}

Here's an entire Bicep file demonstrating it: https://github.com/cwe1ss/msa-template/blob/db6f134dddf78f682b2cebd681fe11ebfb68026e/infrastructure/environment/app-environment.bicep

ezYakaEagle442 commented 1 year ago

see also:

ezYakaEagle442 commented 1 year ago

@kendallroden

About Metrics on container apps cannot be exported to Log Analytics or third party as there is no option to configure diagnostic settings. This also means that documentation wrongly states that metrics are exportable: https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftappcontainerapps

I did test it with :

@description('Cluster configuration which enables the log daemon to export app logs to a destination. Currently only "log-analytics" is supported https://learn.microsoft.com/en-us/azure/templates/microsoft.app/managedenvironments?pivots=deployment-language-bicep#managedenvironmentproperties')
param logDestination string = 'azure-monitor' // 'log-analytics'

resource corpManagedEnvironment 'Microsoft.App/managedEnvironments@2022-06-01-preview' = {
  name: azureContainerAppEnvName
  location: location
  properties: {
    appLogsConfiguration: {
      destination: logDestination // azure-monitor
      logAnalyticsConfiguration: {
        customerId: logAnalyticsWorkspace.properties.customerId
        sharedKey: logAnalyticsWorkspace.listKeys().primarySharedKey
      }
    }
    zoneRedundant: zoneRedundant
    daprAIInstrumentationKey: appInsights.properties.InstrumentationKey
    daprAIConnectionString: appInsights.properties.ConnectionString
  }
}

output corpManagedEnvironmentId string = corpManagedEnvironment.id 
output corpManagedEnvironmentDefaultDomain string = corpManagedEnvironment.properties.defaultDomain
output corpManagedEnvironmentStaticIp string = corpManagedEnvironment.properties.staticIp

// https://github.com/microsoft/azure-container-apps/issues/382#issuecomment-1278623205
// https://github.com/cwe1ss/msa-template/blob/db6f134dddf78f682b2cebd681fe11ebfb68026e/infrastructure/environment/app-environment.bicep

// https://github.com/microsoft/azure-container-apps/issues/382
resource appInsightsDiagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'dgs-${appName}-send-${azureContainerAppEnvName}-logs-and-metrics-to-log-analytics'
  scope: corpManagedEnvironment
  properties: {
    logAnalyticsDestinationType: 'AzureDiagnostics'
    workspaceId: logAnalyticsWorkspace.id
    logs: [
      {
        category: 'ContainerAppConsoleLogs'
        enabled: true
      }
      {
        category: 'ContainerAppSystemLogs'
        enabled: true
      }      
      {
        category: 'ApplicationConsole'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
      {
        category: 'SystemLogs'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
      {
        category: 'IngressLogs'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }    
    ]
    metrics: [
      {
        category: 'AllMetrics'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
    ]
  }
}

fails with Metric export is not enabled

ERROR: ***"status":"Failed","error":***"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[***"code":"Conflict","message":"***\r\n  \"status\": \"Failed\",\r\n  \"error\": ***\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      ***\r\n        \"code\": \"DeploymentFailed\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n        \"details\": [\r\n          ***\r\n            \"code\": \"BadRequest\",\r\n            \"message\": \"***\\r\\n  \\\"code\\\": \\\"BadRequest\\\",\\r\\n  \\\"message\\\": \\\"Metric export is not enabled.\\\"\\r\\n***\"\r\n          ***\r\n        ]\r\n

BigMorty commented 1 year ago

Thanks @cwe1ss for pointing out Azure Container Apps now support AzMon integration and Diagnostic settings.

@sanchitmehta, I was just creating a Diagnostic setting for my environment and noticed Metrics don't show up as something I can export.

slavizh commented 1 year ago

the env resource does not have metrics. The metrics on the container app resource but there there is no support for setting diagnostic settings.

BigMorty commented 1 year ago

@sanchitmehta, thoughts on this?

giovannifl commented 1 year ago

Any update on this issue ?

BigMorty commented 1 year ago

@giovannifl, are you talking about being able to redirect metrics via a diagnostic setting for and ACA app?

I get the option to do that now.

anuragvaishnava commented 10 months ago

@giovannifl, are you talking about being able to redirect metrics via a diagnostic setting for and ACA app?

I get the option to do that now.

I tried same ("Checked "ALL METRICS") but still i cant see the Diagnostics logs for container Apps. Was it working for you?

vaspopinex commented 9 months ago

We have tried "all metrics" aswell, but no metric data is shipped to the destination.

giovannifl commented 9 months ago

Yes, now that option is finally available but documentation is still outdated...

https://learn.microsoft.com/en-us/azure/templates/microsoft.app/managedenvironments?pivots=deployment-language-bicep#managedenvironmentproperties

vaspopinex commented 9 months ago

Yes, now that option is finally available but documentation is still outdated...

https://learn.microsoft.com/en-us/azure/templates/microsoft.app/managedenvironments?pivots=deployment-language-bicep#managedenvironmentproperties

I assume you are referring to appLogsConfiguration? From our understanding "azure-monitor" is required for enabling diagnostic settings? or atleast thats our conclusion based on what happens in the portal UI. The reason is that we want the logs shipped to a storage account. But requiring us to change from azure-monitor to log-analytics would disable diagnostic settings and thus not being able to ship logs to storage account (not directly atleast, sure we could export to storage account from the log workspace but we would prefer not doing it). Is this a correct interpretation of the current state of things?

That in short, we either choose logs+metric to log analytics workspace OR logs to storage account? Or are diagnostic settings still usable if we use "log-analytics" even though that not being possible using the UI?

Also since we cant see the metrics maybe we are looking in the wrong place. In which table are the metrics expected to end up in in Log Analytics Workspace?

microsoft / azure-container-apps