Derived method 'TryCreateLogger' in type 'OpenTelemetry.Logs.LoggerProviderSdk' from assembly 'OpenTelemetry, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c' cannot reduce access.

ChuanGoing commented 4 months ago

Package

OpenTelemetry

Package Version

Package Name	Version
OpenTelemetry.Api	1.9.0
OpenTelemetry.Api.ProviderBuilderExtensions	1.9.0
OpenTelemetry.Instrumentation.EntityFrameworkCore	1.0.0-beta.12

Runtime Version

.NET6.0

Description

When recompiling and running my application, an error occurs during the startup phase: The derived method 'TryCreateLogger' in type 'OpenTelemetry.Logs.LoggerProviderSdk' from assembly 'OpenTelemetry, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c' cannot reduce access.

Description: The application relies on OpenTelemetry related packages to provide APM functionality. After encountering the error, I compared it with another application (which starts normally), which references OpenTelemetry.Api version 1.8.1. This led me to suspect that there might have been changes in the OpenTelemetry.Api version 1.9.0 update.

Steps to Reproduce

Upgrading from OpenTelemetry.Api 1.8.1 to 1.9.0 can reproduce the issue.

Expected Result

I hope to locate the problem as soon as possible.

Actual Result

The above exception message is encountered.

Additional Context

No response

Kielek commented 4 months ago

@ChuanGoing, could you please provide us Minimal, Reproducible Example? It can be separate repository with the problematic code. It should not include any internal business logic, just scaffolding of code to reproduce issue.

jovnas-inet commented 4 months ago

We have the same problem, using the following packages. (Downgrade to 1.8.X fixes the problem.) OpenTelemetry.Instrumentation.Runtime v1.9.0 OpenTelemetry.Instrumentation.SqlClient v1.9.0-beta.1

My guess is that this change is the reason: https://github.com/open-telemetry/opentelemetry-dotnet/commit/2710c427f032fb22208827cbcf9d9459eaaf1f76#diff-725c4458d2e5e52fc9f80dc087141999cce9d3b213c2fa30ed2488e1cec3fa06L92-R104

EDIT: If it matters, for us, it is when Autofac tries to build the services that the error occurs.

Unhandled exception. Autofac.Core.DependencyResolutionException: An exception was thrown while activating ?:Microsoft.Extensions.Hosting.IHost -> Microsoft.Extensions.Hosting.Internal.ApplicationLifetime -> Microsoft.Extensions.Logging.Logger`1[[Microsoft.Extensions.Hosting.Internal.ApplicationLifetime, Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]] -> Microsoft.Extensions.Logging.LoggerFactory -> ?:Microsoft.Extensions.Logging.ILoggerProvider[] -> ?:Microsoft.Extensions.Logging.ILoggerProvider.
 ---> System.TypeLoadException: Derived method 'TryCreateLogger' in type 'OpenTelemetry.Logs.LoggerProviderSdk' from assembly 'OpenTelemetry, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7bd6737fe5b67e3c' cannot reduce access.

jiawbao commented 4 months ago

I also met this issue after we added this package : <PackageVersion Include="OpenTelemetry.Instrumentation.Quartz" Version="1.0.0-beta.3" />

Below are all packages we are using :

    <PackageVersion Include="OpenTelemetry.Audit.Geneva" Version="2.1.1" />

    <PackageVersion Include="OpenTelemetry.Exporter.Console" Version="1.5.1" />

    <PackageVersion Include="OpenTelemetry.Exporter.Geneva" Version="1.5.1" />

    <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.5.1" />

    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.8.1" />

    <PackageVersion Include="OpenTelemetry.Instrumentation.GrpcNetClient" Version="1.8.0-beta.1" />

    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.8.1" />

    <PackageVersion Include="OpenTelemetry.Instrumentation.Quartz" Version="1.0.0-beta.3" />`

damien-c-d commented 4 months ago

We are having the same problem with the following packages installed:

OpenTelemetry.Instrumentation.AspNetCore v1.9.0 OpenTelemetry.Instrumentation.Http v1.9.0 OpenTelemetry.Instrumentation.Runtime v1.9.0

Downgrading them ALL to 1.8.1 fixes the problem. Likewise, upgrading any one of them to 1.9.0 reintroduces the problem.

If this method is not called, then the problem does not occur:

    public static WebApplicationBuilder WithOpenTelemetry(this WebApplicationBuilder builder)
    {
        const string serviceName = "my-web-app";

        builder.Logging.AddApplicationInsights();

        builder.Logging.AddOpenTelemetry(options =>
        {
            options.IncludeScopes = true;
            options.IncludeFormattedMessage = true;

            options.SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService(serviceName));
        });

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resource =>
            {
                resource.AddService(serviceName);
            })
            .WithTracing(tracing =>
            {
                tracing.AddAspNetCoreInstrumentation();
            })
            .WithMetrics(metrics =>
            {
                metrics.AddMeter("My.WebApp");
                metrics.AddAspNetCoreInstrumentation()
                       .AddRuntimeInstrumentation();
            })
            .UseAzureMonitor(options =>
            {
                options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
            });

        return builder;
    }

TimothyMothra commented 4 months ago

I've narrowed this to an incompatibility between OpenTelemetry v1.8.1 and OpenTelemetry.Api.ProviderBuilderExtensions v1.9.0. On the surface that may look obvious, but these can be transient dependencies from other combinations of packages.

Here's a minimal repro:

WebApplication1.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="OpenTelemetry" Version="1.8.1" />
    <PackageReference Include="OpenTelemetry.Api.ProviderBuilderExtensions" Version="1.9.0" />
  </ItemGroup>

</Project>

Program.cs

namespace WebApplication1
{
    using Microsoft.AspNetCore.Builder;
    using Microsoft.Extensions.DependencyInjection;
    using Microsoft.Extensions.Logging;

    public class Program
    {
        public static void Main(string[] args)
        {
            var builder = WebApplication.CreateBuilder(args);

            builder.Services.AddLogging(logging => logging.AddOpenTelemetry());

            var app = builder.Build();

            app.MapGet("/", () => "Hello World!");

            app.Run();
        }
    }
}

Additional Repro

This combination may be more common for users:

csproj

In this example, these are the latest versions of these two packages.

  <ItemGroup>
    <PackageReference Include="Azure.Monitor.OpenTelemetry.AspNetCore" Version="1.2.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
  </ItemGroup>

Program.cs

In this example, UseAzureMonitor() internally uses builder.Services.AddLogging(logging => logging.AddOpenTelemetry());

builder.Services.AddOpenTelemetry().UseAzureMonitor();

vishweshbankwar commented 4 months ago

Upgrading OpenTelemetry to 1.9.0 should fix the issue.

@CodeBlanch - FYI

ChuanGoing commented 4 months ago

I'm sorry for the late response. After encountering the issue last time, I resolved it by downgrading OpenTelemetry.Instrumentation.EntityFrameworkCore to version 1.0.0-beta.11, but I didn't personally investigate the root cause

damien-c-d commented 4 months ago

@vishweshbankwar That's not a fix. Upgrading the packages I mentioned above to 1.9.0 is what causes this issue for us.

Edit:

These 3 are the only ones we have installed: OpenTelemetry.Instrumentation.AspNetCore v1.9.0 OpenTelemetry.Instrumentation.Http v1.9.0 OpenTelemetry.Instrumentation.Runtime v1.9.0

All other packages are up to date in the solution.

CodeBlanch commented 4 months ago

Lots to unpack here! 🤣

Let's start with root cause. In 1.9.0 the LoggerProvider class inside OpenTelemetry.Api was promoted to public from internal but the Logger class was left internal. This creates issues for LoggerProvider.TryCreateLogger which hands back a Logger instance. TryCreateLogger had to be made internal:

API 1.9.0:

public class LoggerProvider : BaseProvider
{
    internal virtual bool TryCreateLogger(string? name, out Logger? logger) {}
}

API previously:

internal class LoggerProvider : BaseProvider
{
    protected virtual bool TryCreateLogger(string? name, out Logger? logger) {}
}

The SDK (OpenTelemetry.dll) contains an implementation of this type.

SDK 1.9.0:

internal sealed class LoggerProviderSdk : LoggerProvider
{
    internal override bool TryCreateLogger(string? name, out Logger? logger) {}
}

SDK previous:

internal sealed class LoggerProviderSdk : LoggerProvider
{
    protected override bool TryCreateLogger(string? name, out Logger? logger) {}
}

The issue is going to manifest if you have OpenTelemetry.Api.dll >= 1.9.0 and OpenTelemetry.dll < 1.9.0.

The fix is to add a reference to OpenTelemetry (SDK) version 1.9.0 (or greater). A direct reference will win over transitive ones.

This report:

These 3 are the only ones we have installed: OpenTelemetry.Instrumentation.AspNetCore v1.9.0 OpenTelemetry.Instrumentation.Http v1.9.0 OpenTelemetry.Instrumentation.Runtime v1.9.0

I'm having trouble reconciling that. Because there is no OpenTelemetry (SDK) reference at all in that graph. The goal with all OTel implementations is explicitly that instrumentation libraries should NOT need an SDK reference. If you only have those 3 packages, there is no "Sdk.Create*" API available or AddOpenTelemetry API available to even turn on OTel. There must be some other reference brining in OpenTelemetry at an older version.

My guess would be it is something more like...

OpenTelemetry.Extensions.Hosting v1.8.1 <- Issue is here
OpenTelemetry.Instrumentation.AspNetCore v1.9.0
OpenTelemetry.Instrumentation.Http v1.9.0
OpenTelemetry.Instrumentation.Runtime v1.9.0

Getting the SDK transitively is the issue here.

OpenTelemetry.Extensions.Hosting brings in the SDK. So if you are using that type of reference structure (getting the SDK transitively), make sure to bump OpenTelemetry.Extensions.Hosting:

OpenTelemetry.Extensions.Hosting v1.9.0
OpenTelemetry.Instrumentation.AspNetCore v1.9.0
OpenTelemetry.Instrumentation.Http v1.9.0
OpenTelemetry.Instrumentation.Runtime v1.9.0

Or do the direct reference.

What you want to see in your graph is SDK + API versions matching:

Bad:

Good:

Also good (direct reference added to SDK):

What could be done to improve this?

We have run into other issues before with users on newer instrumentation libraries and older SDKs. @rajkumar-rangaraj @vishweshbankwar @Kielek The way our CI is set up we always test SDK using latest API. Perhaps we should have some steps which test latest API packages using previous SDK? We sort of assume users will be on same version of API & SDK but the way instrumentation libraries work it is easy for graphs to become mismatched.

Kielek commented 4 months ago

@CodeBlanch, correct me if I am wrong, but it is the root cause of the issue: https://github.com/open-telemetry/opentelemetry-dotnet/blob/dbe2ce35e7066a6d8b712c3491c39cd99061cdcc/src/OpenTelemetry.Api/AssemblyInfo.cs#L6.

I suppose that we could add some msbuild task to detect such cases and rise warning if there is incompatibility version between OpenTelemetry.Api and OpenTelemetry. The other option is to have more strict requirements (support for exact versions) on the nuget package level. In such case we should cover all new releases.

xaviergxf commented 4 months ago

I'm also having this issue, can this be reopened?

CodeBlanch commented 4 months ago

@Kielek

@CodeBlanch, correct me if I am wrong, but it is the root cause of the issue:

InternalsVisibleTo is part of it but isn't the root cause. If we didn't use InternalsVisibleTo we wouldn't have this issue, but we also wouldn't have experimental APIs and we would need a lot of other internal APIs to be exposed public. We definitely abuse InternalsVisibleTo 😄

The issue is something which was previously protected became internal in 1.9.0 so the type system is complaining. Basically a breaking change to an API contract but it slipped through our tooling because of the internal nature.

The other option is to have more strict requirements (support for exact versions) on the nuget package level. In such case we should cover all new releases.

I'm not opposed to this but I would be concerned the net result long term would be users getting stuck on old versions. We can discuss more on SIG?

@xaviergxf

I'm also having this issue, can this be reopened?

We could reopen this but I'm not sure what could be done. To fix update your graph so API + SDK packages are on the same version. Let's say we release a new OpenTelemetry.Api with some fix. You would need a package upgrade for that so the process would look pretty much the same 😄 I'll mess with things to see if there is a possible way to even fix it through API upgrade.

ggimenes commented 1 month ago

@Kielek

@CodeBlanch, correct me if I am wrong, but it is the root cause of the issue:

InternalsVisibleTo is part of it but isn't the root cause. If we didn't use InternalsVisibleTo we wouldn't have this issue, but we also wouldn't have experimental APIs and we would need a lot of other internal APIs to be exposed public. We definitely abuse InternalsVisibleTo 😄

The issue is something which was previously protected became internal in 1.9.0 so the type system is complaining. Basically a breaking change to an API contract but it slipped through our tooling because of the internal nature.

The other option is to have more strict requirements (support for exact versions) on the nuget package level. In such case we should cover all new releases.

I'm not opposed to this but I would be concerned the net result long term would be users getting stuck on old versions. We can discuss more on SIG?

@xaviergxf

I'm also having this issue, can this be reopened?

We could reopen this but I'm not sure what could be done. To fix update your graph so API + SDK packages are on the same version. Let's say we release a new OpenTelemetry.Api with some fix. You would need a package upgrade for that so the process would look pretty much the same 😄 I'll mess with things to see if there is a possible way to even fix it through API upgrade.

I just faced this problem trying to use OpenTelemetry.Instrumentation.StackExchangeRedis version 1.9.0-beta.1 The application completely stopped sending traces to Jaeger 😢

ChuanGoing commented 3 weeks ago

A few months later, I came back because I found that some older projects were referencing OpenTelemetry.Instrumentation.EntityFrameworkCore-1.0.0-beta.12, and re-releasing also had this issue. Downgrading to version 1.0.0-beta.11 resolved it. So, why does this package have such an issue, and no one has answered this question?

open-telemetry / opentelemetry-dotnet