9.0.0-rc.1.24511.1 does not receive telemetry from podman containers

[paulomorgado]

The same app host can receive telemetry from a container running an ASP.NET Core 8 application when using Docker Rancher (with persistent containers), but is not able to receive telemetry from a container running an ASP.NET Core 8 application when using podman.

Environment variables in the running container

app@085f4773b125:/app$ printenv | egrep "OTEL|DOTNET"
DOTNET_RUNNING_IN_CONTAINER=true
OTEL_BLRP_SCHEDULE_DELAY=1000
OTEL_BSP_SCHEDULE_DELAY=1000
OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES=true
OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://host.containers.internal:19155
OTEL_EXPORTER_OTLP_HEADERS=x-otlp-api-key=4970d0f523a311be078713892cdc280a
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_METRIC_EXPORT_INTERVAL=1000
OTEL_METRICS_EXEMPLAR_FILTER=trace_based
OTEL_RESOURCE_ATTRIBUTES=service.instance.id=nubejazf
OTEL_SERVICE_NAME=hub-container
OTEL_TRACES_SAMPLER=always_on

[davidfowl]

Can you make a repro? I am testing with a later Aspire 9.0 build and podman and everything works just fine.

[paulomorgado]

Can you make a repro? I am testing with a later Aspire 9.0 build and podman and everything works just fine.

I can wait a few days for the release and get back at this.

[davidfowl]

Well I it’s be great if you can provide me with a repro so I can try it before you get it to see if it’s indeed fixed

[paulomorgado]

Well I it’s be great if you can provide me with a repro so I can try it before you get it to see if it’s indeed fixed

Sure.

I've created an Aspire starter project and added Docker support to the ApiService project using .NET SDK.

Then made this change to the app host:

//var apiService = builder.AddProject<Projects.AspireApp1_ApiService>("apiservice");

var apiServiceContainer = builder.AddContainer("apiservice", "localhost/aspireapp1apiservice", "latest")
    .WithHttpEndpoint(targetPort: 8080)
    .WithEnvironment("Logging__LogLevel__Default", "Debug")
    .WithEnvironment("Logging__LogLevel__Microsoft.AspNetCore", "Debug")
    .WithOtlpExporter()

    ;var apiService = apiServiceContainer.GetEndpoint("http");

Just in case, I set the ASPIRE_ALLOW_UNSECURE_TRANSPORT to true and ran everything in HTTP:

I can see the console logs from the container but not structured logs, traces or metrics.

The OTEL_EXPORTER_OTLP_ENDPOINT is set to http://host.containers.internal:19085 (the same port as the Web project).

[joperezr]

@paulomorgado Thanks for logging the issue. We released Aspire 9.0 GA earlier today, can you please update to that to see if this fixes it for you?

Also, thanks for providing repro steps, but would you be able to actually share a GH repo with the minimal repro so we can take a closer look?

[paulomorgado]

@joperezr,

Still not working.

I'm targeting .NET 8.0. Does it work on 9.0?

Do you have a working sample?

It works on Rancher Desktop (Aspire requires persistent containers).

[paulomorgado]

@joperezr, have you been able to repro this?

[joperezr]

We have not, so it would be helpful if you could share a GH Repo with the minimal repro so we can take a look.

[paulomorgado]

Well I it’s be great if you can provide me with a repro so I can try it before you get it to see if it’s indeed fixed

Sure.

I've created an Aspire starter project and added Docker support to the ApiService project using .NET SDK.

Then made this change to the app host:

//var apiService = builder.AddProject("apiservice");

var apiServiceContainer = builder.AddContainer("apiservice", "localhost/aspireapp1apiservice", "latest") .WithHttpEndpoint(targetPort: 8080) .WithEnvironment("LoggingLogLevelDefault", "Debug") .WithEnvironment("LoggingLogLevelMicrosoft.AspNetCore", "Debug") .WithOtlpExporter()

;var apiService = apiServiceContainer.GetEndpoint("http");

Just in case, I set the ASPIRE_ALLOW_UNSECURE_TRANSPORT to true and ran everything in HTTP:

I can see the console logs from the container but not structured logs, traces or metrics.

The OTEL_EXPORTER_OTLP_ENDPOINT is set to http://host.containers.internal:19085 (the same port as the Web project).

@joperezr does it not repro for you from this? Or couldn't you build this repro? Because if you couldn't I need to know why, so that I can set up a repo with a repro.

[joperezr]

I don't have a machine with podman and I also are not likely the person that will investigate this, but we have seen many cases in the past where we make some incorrect assumptions when trying to reproduce, so it's always best when we have a repro solution so we can tell exactly what is going on. For instance, one simple example is that I'm not 100% sure I understand what this statement means: I've created an Aspire starter project and added Docker support to the ApiService project using .NET SDK. Does that mean that as part of building the project you generate a docker image for it?

We have found that the best way to make sure we are investigating the right thing, is to avoid having a layer of interpretation in between 😃.