Open JoeGaggler opened 1 week ago
Hi @JoeGaggler, sorry to hear you're having issues.
The symptom you're describing is typically related to using v2 of the automatic instrumentation with v3 of the manual instrumentation. Would it be possible to show your dockerfile? Is it possible you're installing the 2.x tracer there?
Sure, here is our dockerfile
:
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG PAT
WORKDIR /source
COPY containerapp containerapp
COPY NuGet.config .
RUN dotnet nuget update source "Internal Feed" -u pat -p "${PAT}" --store-password-in-clear-text --configfile NuGet.config ; \
dotnet restore "containerapp/Project.csproj" \
--locked-mode \
-v n \
--configfile NuGet.config \
--ignore-failed-sources \
--runtime linux-x64 \
/p:PublishReadyToRun=true
RUN dotnet publish "containerapp/Project.csproj" \
--no-restore \
--configuration Release \
--output /publish \
--no-self-contained \
--runtime linux-x64 \
/p:PublishReadyToRun=true
FROM mcr.microsoft.com/dotnet/aspnet:8.0 as base
COPY --from=datadog/serverless-init:1.2.8 /datadog-init /app/datadog-init
WORKDIR /app
COPY --from=build /publish .
RUN /app/datadog/createLogPath.sh
USER $APP_UID
ENTRYPOINT ["/app/datadog-init"]
CMD ["dotnet", "Project.dll"]
Thanks @JoeGaggler, that's really useful, I'm a little confused about one aspect - in order to use the Datadog.Trace.Bundle Nuget, you need to set a bunch of environment variables (CORECLR_PROFILER_PATH
, DD_DOTNET_TRACER_HOME
etc), but AFAICT, your dockerfile isn't setting any of those 🤔
Is there something I'm missing, with that dockerfile as far as I can tell, it shouldn't work at all i.e. you shouldn't get auto-instrumentation or manual traces 😅
Ah, yes, sorry about that. Those environment variables are being set in the Azure Container App via an ARM template, below with minor redactions:
resourceGroup: 'REDACTED_SECRET_VALUE'
location: 'centralus'
name: 'REDACTED_SECRET_VALUE'
type: Microsoft.App/containerApps
properties:
template:
containers:
- name: 'REDACTED_SECRET_VALUE'
image: REDACTED_SECRET_VALUE
env:
- name: CORECLR_ENABLE_PROFILING
value: '1'
- name: CORECLR_PROFILER
value: '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}'
- name: CORECLR_PROFILER_PATH
value: '/app/datadog/linux-x64/Datadog.Trace.ClrProfiler.Native.so'
- name: LD_PRELOAD
value: '/app/datadog/linux-x64/Datadog.Linux.ApiWrapper.x64.so'
- name: DD_API_KEY
secretRef: datadog-api-key
- name: DD_AZURE_RESOURCE_GROUP
value: 'REDACTED_SECRET_VALUE'
- name: DD_AZURE_SUBSCRIPTION_ID
value: 'REDACTED_SECRET_VALUE'
- name: DD_DBM_PROPAGATION_MODE
value: service
- name: DD_DOTNET_TRACER_HOME
value: '/app/datadog'
- name: DD_ENV
value: 'esone-dev'
- name: DD_LOG_LEVEL
value: 'info'
- name: DD_LOGS_ENABLED
value: 'true'
# Container Apps already use the docker integration for log submission
# - name: DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS
# value: ''
- name: DD_LOGS_INJECTION
value: 'true'
- name: DD_RUNTIME_METRICS_ENABLED
value: 'true'
- name: DD_SERVICE
value: 'REDACTED_SECRET_VALUE'
- name: DD_SITE
value: 'datadoghq.com'
- name: DD_TRACE_ENABLED
value: 'true'
- name: DD_TRACE_DEBUG
value: 'false'
- name: DD_VERSION
value: '24.1009.7.3_merge'
- name: DD_TRACE_SAMPLE_RATE
value: '1.0'
- name: DD_PROFILING_ENABLED # NOTE: Datadog requires cpu >= 1.0 for profiling
value: '1'
- name: DD_PROFILING_ALLOCATION_ENABLED
value: '1'
- name: DD_PROFILING_EXCEPTION_ENABLED
value: '1'
- name: DD_PROFILING_HEAP_ENABLED
value: '1'
- name: DD_PROFILING_LOCK_ENABLED
value: '1'
- name: DD_PROFILING_WALLTIME_ENABLED
value: '0'
resources: # for consumption plan, memory must be 2x cpu (in Gi)
cpu: '1.0'
memory: '2.0Gi'
Thanks for that @JoeGaggler! Unfortunately I can't see anything obvious at all, that all looks ok to me 🤔 Would it be possible to raise a ticket with support so that we can look at collecting more details like the tracer log files etc? Thanks!
Will do, thanks @andrewlock!
Hey @JoeGaggler, just wanted to follow up to say that those logs you sent in the ticket were really useful, thank you! We'll should keep general communication there, but just wanted to say that we have managed to find a scenario that reproduces the issue you have. Specifically, we can reproduce the issue if you have a static constructor in your Program
class, e.g. something like this:
public static class Program
{
private static readonly Datadog.Trace.Tracer _tracer;
static Program()
{
_tracer = Datadog.Trace.Tracer.Instance;;
}
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/", () => {
using var scope = _tracer.StartActive("custom-operation-name");
return "Hello World!";
});
app.Run();
}
}
Do you have code similar to that in your application? If not, would you mind sharing an example of the code (on the Zendesk ticket is fine!). Thanks!
Hi @andrewlock, We're not using an explicit static constructor in Program
(Minimal API), but I will update the Zendesk ticket with the actual source code.
Describe the bug We upgraded
Datadog.Trace.Bundle
from2.59.0
to3.3.1
and no longer have custom traces in Datadog. Automatic traces still are working, i.e. ASP.NET Core Requests.To Reproduce / Runtime environment Steps to reproduce the behavior:
FROM mcr.microsoft.com/dotnet/aspnet:8.0
1.2.8
Datadog.Trace.Bundle 3.3.1
Datadog.Trace.Tracer.Instance.StartActive
to create custom traces.Expected behavior Expecting to still have custom traces in addition to automatic traces. Downgrading to
2.59.0
immediately resolved the issue.Additional context Nothing in the v2 to v3 migration guide seemed to point to the issue. Wondering if the
serverless-init
is somehow not compatible with the latest NuGet package.