Irregular failing to connect to Logfire API errors

[yovelcohen]

Description

I have a bunch of python microservices, currently running on azure container apps. they write logs to two major projects in Logfire. For some reason, even though I'm using the same configuration code and the exact same write token, some of the services can't connect to logfire, with a weird open telemetry error msg:

/usr/local/lib/python3.11/site-packages/logfire/_internal/config.py:842: UserWarning: Logfire API is unreachable, you may have trouble sending data. Error: HTTPConnectionPool(host='k8se-otel.k8se-apps.svc', port=4317): Max retries exceeded with url: /v1/info (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x71713aae7e90>: Failed to resolve 'k8se-otel.k8se-apps.svc' ([Errno -2] Name does not resolve)"))
  warnings.warn(f'Logfire API is unreachable, you may have trouble sending data. Error: {e}')

I haven't been able to find consistency as to why some services can connect and some connect (double-checked the logfire configuration in each)

Python, Logfire & OS Versions, related packages (not required)

logfire="0.42.0"
platform="Linux-5.15.153.1-1.cm2-x86_64-with"
python="3.11.7 (main, Jan 27 2024, 07:04:48) [GCC 13.2.1 20231014]"
[related_packages]
requests="2.32.3"
pydantic="2.7.4"
fastapi="0.111.0"
protobuf="4.25.3"
rich="13.7.1"
executing="2.0.1"
opentelemetry-api="1.25.0"
opentelemetry-exporter-otlp-proto-common="1.25.0"
opentelemetry-exporter-otlp-proto-http="1.25.0"
opentelemetry-instrumentation="0.46b0"
opentelemetry-proto="1.25.0"
opentelemetry-sdk="1.25.0"
opentelemetry-semantic-conventions="0.46b0"

[alexmojaki]

k8se-otel.k8se-apps.svc is not the Logfire API. If you didn't configure that yourself, then I think something in Azure is setting the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. You can override it with base_url='https://logfire-api.pydantic.dev' in logfire.configure().

[yovelcohen]

@alexmojaki thanks, that was indeed the issue, I'm just a bit curios as to why would this happens on some container apps and not others, but thanks :)

[alexmojaki]

I see OTEL_EXPORTER_OTLP_ENDPOINT mentioned in https://learn.microsoft.com/en-us/azure/container-apps/opentelemetry-agents?tabs=arm#environment-variables, but I don't know the details. If you learn more, please share :)

I'm closing since this seems resolved, but if other people run into the same issue then we might want to consider not paying attention to OTEL_EXPORTER_OTLP_ENDPOINT to avoid confusion.