Microsoft Entra Authentication integration with azure-monitor-opentelemetry to send logs/traces does not work.

[summahto]

• Package Name: azure-monitor-opentelemetry
• Package Version: 1.6.2
• Operating System: Linux
• Python Version: 3.11

Describe the bug Unable to authenticate post requests to App-insights using user assigned Managed Identity for my app-service deployed in Azure. Managed Identity has been assigned the role of Monitoring Metrics Publisher

Further, I checked the documentation entra authentication in case of App-Insights (https://learn.microsoft.com/en-us/azure/azure-monitor/app/azure-ad-authentication?tabs=python#unsupported-scenarios), there are some Contradicting statements here:

1. It says, ingestion for AutoInstrumentation for Python on Azure App Service is unsupported.
2. However, here it says it is supported along with the configuration :Enable Microsoft Entra ID (formerly Azure AD) authentication.

To Reproduce Steps to reproduce the behavior:

1. Create an app-service in Azure.
2. Create a user-assigned managed identity for the above app-service.
3. Enable app-insights for the app-service. Disable localAuth to stop unauthorized telemetry Ingestion
4. Configure Azure monitor for the Webapp.
5. Pass the managed identity along with the client ID for Managed Identity, when initializing Azure Monitor.
6. All the post requests to Application Insights instance fail with 401 error (unauthorized). Here is an example log from the Web-app:

DEBUG    urllib3.connectionpool
[https://eastus2-0.in.applicationinsights.azure.com:443](https://eastus2-0.in.applicationinsights.azure.com/)
"POST //v2.1/track HTTP/1.1" 401 None

Expected behavior Telemetry with Managed Identity should be authorized and sent across to Application Insights.

Screenshots

Web app with Managed identity configured :

Role assigned to Managed Identity :

Logs:

Azure monitor config

def configure_basic_azure_monitor():

    conn_string = settings.APPLICATIONINSIGHTS_CONNECTION_STRING

    docgen_logger = logging.getLogger("test")

    managed_credential = ManagedIdentityCredential(client_id=settings.AZURE_CLIENT_ID)

    configure_azure_monitor(
        connection_string= conn_string,
        logger_name=docgen_logger.name,
        credentials=managed_credential
    )

Additional context Webapp : Fast API with uvicorn

[kashifkhan]

Thank you for the feedback @summahto . We will investigate and get back to you asap.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @gulopesd @Haiying-MSFT @jairmyree @joshfree @KarishmaGhiya @KevinBlasko @kurtzeborn @nisha-bhatia @pvaneck @sarangan12 @scottaddie @srnagar @ToddKingMSFT.

[summahto]

Hi @jeremydvoss , @lzchen

Any Update on this ?

[lzchen]

@summahto

Apologies for the late reply.

Further, I checked the documentation entra authentication in case of App-Insights (https://learn.microsoft.com/en-us/azure/azure-monitor/app/azure-ad-authentication?tabs=python#unsupported-scenarios), there are some Contradicting statements here:

It says, ingestion for AutoInstrumentation for Python on Azure App Service is unsupported. However, here it says it is supported along with the configuration :Enable Microsoft Entra ID (formerly Azure AD) authentication.

These are not conflicting statements. The first link is for AUTOINSTRUMENTATION via Azure App service attach (which is currently unsupported and is planning to be released this year). The second is MANUAL instrumentation (the one you are interested in) via the azure monitor distro.

With that being said, we need to clear up a few things regarding your setup/configuration. If you deploy a Python application into AppService, and you choose to ENABLE appinsights through the portal (upon creation), this is using AUTOINSTRUMENTATION. This means that you DO NOT have to change your actual Python application to use the Azure monitor sdk (configure_azure_monitor() call) and your telemetry should flow automatically. It looks like from your setup that you are using BOTH, so it is difficult to know which telemetry is generating those 401 errors. Please follow the below guidance:

If you want to keep using AppService with entraID auth, you can either:

1. Wait until this feature is supported via AUTO instrumentation in a couple of months. When it is supported (via the checkbox on the UX create page), you can remove the call to configure_azure_monitor in your code and your telemetry should flow automatically.
2. REMOVE telemetry collection via AUTO instrumentation by setting ApplicationInsightsAgent_EXTENSION_VERSION=disabled in your environment variable and KEEP your code as is with configure_azure_monitor() call. This is effectively now using MANUAL instrumentation.

[summahto]

Hi @lzchen ,

Thank you so much for clarifying this. It makes much more sense now. Let me try the 2nd solution you mentioned and come back with my results.

[summahto]

@lzchen ,

Yes, we had both AUTOINSTRUMENTATION and Manual instrumentation enabled for our application. I tried the 2nd solution which you provided i.e. disabling auto-instrumentation using the setting ApplicationInsightsAgent_EXTENSION_VERSION=disabled. However, even now the telemetry does not go through with Entra Authentication, that is the original problem remains.

Here is the stacktrace after enabling Debug logs for the web-app:

If you notice App-insights returns 401 unauthorized whereas request to livediagnostics https://eastus2.livediagnostics.monitor.azure.com/ got through.

All the other configurations are same as above.

[lzchen]

@summahto

A couple of questions:

1. Did you turn on live metrics feature via the sdk?

configure_azure_monitor(enable_live_metrics=True)

2. Can you pip freeze your dependencies and paste that here?
3. Are you looking at the traces table in your application insights resource?
4. Can you open up the 401 log and show a screen shot of fields? Specifically I want to see sdkVersion field.

[summahto]

@lzchen ,

Here are the answers :

1. Yes, I have enabled live metrics feature via the sdk.
2. Following are the dependencies :

azure-core==1.32.0
azure-core-tracing-opentelemetry==1.0.0b11
azure-cosmos==4.7.0
azure-identity==1.17.1
azure-monitor-opentelemetry==1.6.2
azure-monitor-opentelemetry-exporter==1.0.0b31
azure-purview-catalog==1.0.0b4
azure-storage-blob==12.23.1
opentelemetry-api==1.27.0
opentelemetry-distro==0.48b0
opentelemetry-instrumentation==0.48b0
opentelemetry-instrumentation-aiohttp-client==0.48b0
opentelemetry-instrumentation-asgi==0.48b0
opentelemetry-instrumentation-asyncio==0.48b0
opentelemetry-instrumentation-dbapi==0.48b0
opentelemetry-instrumentation-django==0.48b0
opentelemetry-instrumentation-fastapi==0.48b0
opentelemetry-instrumentation-flask==0.48b0
opentelemetry-instrumentation-httpx==0.48b0
opentelemetry-instrumentation-logging==0.48b0
opentelemetry-instrumentation-psycopg2==0.48b0
opentelemetry-instrumentation-requests==0.48b0
opentelemetry-instrumentation-urllib==0.48b0
opentelemetry-instrumentation-urllib3==0.48b0
opentelemetry-instrumentation-wsgi==0.48b0
opentelemetry-resource-detector-azure==0.1.5
opentelemetry-sdk==1.27.0
opentelemetry-semantic-conventions==0.48b0
opentelemetry-util-http==0.48b0 

3. Yes, I am looking at the traces table in Application Insights resource :

4. The opened log does not have the sdkVersion field in it. Here is the screenshot.

[lzchen]

@summahto

It does appear that these logs are being generated by an SDK in AppService and not from the Azure monitor distro (I see the type is AppServiceConsoleLogs). The distro is not responsible for sending those logs.

I think we need to start from testing your application locally first, verifying your entra ID works by verifying that you do indeed see telemetry in you ApplicationInsights resource and THEN deploy to Appservice after.

Could you run your code locally with the same settings (try first disabling live metrics and then if that works, enable it). You want to be looking at the logs under your APPLICATIONINSIGHTS resource, not log analytics. Please also paste a code snippet of how you are using the SDK and any autocollected logs/traces/metrics you are EXPECTING to appear but have not seen because of the 401s.

[summahto]

@lzchen ,

It does appear that these logs are being generated by an SDK in AppService and not from the Azure monitor distro (I see the type is AppServiceConsoleLogs). The distro is not responsible for sending those logs.

I enabled Debug logs for the app-service. We do have logs from the azure monitor distro in the AppServiceConsoleLogs. as you can see in the following screenshot.

One small observation here,

We initially had these variables defined in App-Service.

After removing all of the above except the APPLICATIONINSIGHTS_CONNECTION_STRING, following logs were seen.

I am not sure, why app-service was trying to resolve 'westus-0.in.applicationinsights.azure.com' ? we have given 'eastus-0.in.applicationinsights.azure.com' in connection string. One point to note here is that all our app-insights resources and LAW are behind AMPLS.

Our cloud resources are behind a firewall, so currently connecting to Application insights from local system is not possible. Sorry about this. Our company's policies don't allow this.

I think we do not have enough logger statements in the distro source code to see what is going wrong when app-service tries connecting to App-insights.

[lzchen]

@summahto

We need to step back a bit. In order to identify what logs are sent by the distro, you need to look at Applicaiton Insights logs and not AppServiceConsoleLogs. It is too difficult right now to decipher whether appservice is responsible for creating those logs or if the distro is. It is also difficult to diagnose your original issue of not seeing logs when entra id is enabled if we don't look at the actual raw logs that the distro is responsible for, and not be distracted by all the noise that is being added by changing configurations. First, focus more attention on Application Insights Logs by going to "Logs" section under your ApplicationInsights resource. These logs + log levels are configured ONLY by in code configuration. Please modify your code snippet that you supplied:

def configure_basic_azure_monitor():

    conn_string = settings.APPLICATIONINSIGHTS_CONNECTION_STRING

    docgen_logger = logging.getLogger("test")
    docgen_logger .setLevel(logger.DEBUG)

    managed_credential = ManagedIdentityCredential(client_id=settings.AZURE_CLIENT_ID)

    configure_azure_monitor(
        connection_string= conn_string,
        logger_name=docgen_logger.name,
        credentials=managed_credential,
        enable_live_metrics=False,
    )

Only after we have seen that your logs in your AppInsights resource looks correct we can begin ruling out what is going on with your AppService logs.

SEPARATELY IN ADDITION TO ABOVE, please re-add all the variables that you had defined previously in AppService. They are all needed in setting up your application to be the most bare bones, and removing them may cause unnecessary noise/telemetry that may affect our investigation.

I am not sure, why app-service was trying to resolve 'westus-0.in.applicationinsights.azure.com' ? we have given 'eastus-0.in.applicationinsights.azure.com' in connection string.

In debug mode, you will get logs related to retriable scenarios in which telemetry attempted to send but couldn't, so we retry for later. These are legitimate use cases. You are seeing westus.... logs because you removed APPLICATIONINSIGHTS_DISABLED_STATSBEAT_ALL environment variable. If you do not want to see those, you can re-add the env var back.