[.NET] AmbiguousMatchException on initializing ServiceClient class

[OlehUdovytskyi]

Hi guys.

I have an issue with initializing my service client created by AutoRest and inherited from ServiceClient.

It throws AmbiguousMatchException and it arises in ServiceClient class.

[AmbiguousMatchException: Multiple custom attributes of the same type found.]
   System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) +122
   Microsoft.Rest.ServiceClient`1.get_FrameworkVersion() +103
   Microsoft.Rest.ServiceClient`1.SetDefaultUserAgentInfo() +177
   Microsoft.Rest.ServiceClient`1.SetUserAgent(String productName, String version) +50
   Microsoft.Rest.ServiceClient`1..ctor(DelegatingHandler[] handlers) +61

It only appears on my Remote machine with Windows Server 2008 R2 Standart .NET Framework 4.6.1. Locally on Windows 10 Anniversary 4.6.2 Framework everything is ok.

In Microsoft.Rest.ServiceClient I've found next part of code in ClientVersion property getter:

try
          {
            AssemblyInformationalVersionAttribute customAttribute1 = assembly.GetCustomAttribute(typeof (AssemblyInformationalVersionAttribute)) as AssemblyInformationalVersionAttribute;
            this._clientVersion = customAttribute1 != null ? customAttribute1.InformationalVersion : (string) null;
            if (string.IsNullOrEmpty(this._clientVersion))
            {
              AssemblyFileVersionAttribute customAttribute2 = assembly.GetCustomAttribute(typeof (AssemblyFileVersionAttribute)) as AssemblyFileVersionAttribute;
              this._clientVersion = customAttribute2 != null ? customAttribute2.Version : (string) null;
            }
          }
          catch (AmbiguousMatchException ex)
          {
          }

I think it is source of my problem but I don't understand why it occurs and to handle it. I see that current exception must be caught but I don't have any other ideas what else can be wrong.

Guys, any ideas ?

[shaunol]

I'm also having this same problem. Working fine on my Windows 10 development workstation but when I deploy to my Server 2012 environment, I get the above exception. I also can't seem to find the source code for the Microsoft.Rest.ClientRuntime package to try to debug. I can't really make heads or tails of this issue.

[shaunol]

Downgrading Microsoft.Rest.ClientRuntime from 2.3.3 to 2.3.2 has fixed the issue for me.

[opterios]

I had exactly the same issue with shaunol (on Windows 8 it was fine, on Windows 2012 it was not working) while trying to make calls to AzureBatch. I was getting the following exception:

Message='Multiple custom attributes of the same type found. at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit)
 at Microsoft.Rest.ServiceClient1.get_FrameworkVersion()
 at Microsoft.Rest.ServiceClient1.SetDefaultUserAgentInfo()
 at Microsoft.Rest.ServiceClient1.SetUserAgent(String productName, String version)
 at Microsoft.Rest.ServiceClient1.InitializeHttpClient(HttpClientHandler httpClientHandler, DelegatingHandler[] handlers)


and downgrading to 2.3.2 resolved the issue!

Thanks for your help!

[robpex]

Adding info from my troubleshooting this problem.

In my case, the "Application Insights Monitor" (https://github.com/Azure/azure-content/blob/master/articles/application-insights/app-insights-monitor-performance-live-website-now.md) was injecting an additional "AssemblyFileVersionAttribute" value into mscorlib.dll. So, the Microsoft.Rest.ClientRuntime error makes sense because when it tries to get the file version using that attribute, it is, in fact, ambiguous.

1) Does it ever make sense to have more than one "AssemblyFileVersionAttribute" on any file? If not, there's at least a bug in the tool I just mentioned.

2) Regardless of the answer to number 1, you'll see that the code for Microsoft.Rest.ClientRuntime tries to use "AssemblyFileVersionAttribute" more than once. In one instance, it catches and ignores AmbiguousReferenceException and in other instances, it is not. I believe it should be consistent. Either it's always safe to ignore it, or it's not.

[shaunol]

Noting that I also have Application Insights Monitor installed on my Server 2012 instance that is exhibiting this behaviour. Really good catch.

[SergeyKanzhelev]

@robplex, Application Insights do not inject any attributes into mscorlib. What made you make this conclusion?

Application Insights do make some code injections. But I cannot think of any that will make the custom attributes reading logic to fail. Do you have a small repro I can try out?

[robpex]

@SergeyKanzhelev the way I reproduced it was by debugging my code and at the breakpoint (I don't think the breakpoint location is important), running the following in an immediate window: System.Attribute.GetCustomAttributes((typeof(object)).Assembly).

From there, I was able to see multiple {System.Reflection.AssemblyFileVersionAttribute} with different values and when I uninstall Application Insights, this was no longer the case. I'm sure there are other ways of getting at those properties for mscorlib, but this is what caused me to make this conclusion.

I saw Microsoft in your profile...are you directly involved with Application Insights?

[SergeyKanzhelev]

@robpex yes, I am working on Application Insights. Do you happen to have a small repro I can try out? I'll check the collection System.Attribute.GetCustomAttributes((typeof(object)).Assembly) on my apps now and report back.

[SergeyKanzhelev]

Ok. I got a repro. We will investigate

[SergeyKanzhelev]

BTW, this is the magic we are trying to make with the status monitor if you are interested: http://apmtips.com/blog/2016/11/18/how-application-insights-status-monitor-not-monitors-dependencies/

If you want to keep using application insights - you can disable profiler and only use Application Insights SDK. It will still collect interesting telemetry. Let me know if you need help configuring it

[robpex]

@SergeyKanzhelev ok, cool. I can't share the code (company policies), but I'm happy to discuss further since I dug pretty deep on this one. :) I suspect there's still something to be fixed in the autorest side as well as I mentioned above in number 2. My initial thought was to catch/ignore the exception like it's doing elsewhere, but I don't know if that's the best approach. I don't know if you're involved with this project though, but perhaps you know someone. I wasn't sure who to tell. :)

[SergeyKanzhelev]

I looked at the list of contributors and I only know one person there. I think the issue is clearly related to Application Insights and needs to be fixed on our side. After that no additional exception handling will be needed. But I didn't dug deep, just initial thoughts

[brjohnstmsft]

Possibly related: https://github.com/Azure/azure-sdk-for-net/issues/2516

[brjohnstmsft]

@SergeyKanzhelev Are you investigating this issue? I'm getting more reports of customers being affected.

[SergeyKanzhelev]

We know the root cause and working on a fix. We expect the fix shipped next week. Meanwhile the work around is to disable Application Insights Azure WebSite extension or StatusMonitor. Please escalate using CSS channel if you need a hotfix sooner.

[jimt4593]

Status Monitor somehow causes AmbiguousMatchException exception on AutoRest apps #241 is the issue that @SergeyKanzhelev is referring to.

The issue is marked closed and mentions Application Insights 2.2.0. Can someone confirm that updating to Application Insights 2.2.0 fixes this issue?

[SergeyKanzhelev]

@jimt4593 the issue you mentioned is still open. We released a fix fro Azure Web Apps and working on on-premisis Status Monitor.

We will also appreciate a confirmation that the fix actually worked.

[vdevappa]

Never worked. Only way I could get it to work is to have a new asp.net project which does not use app insights and when deploying the cloud service - do not enable sending telemetry. If you do, error still happens.

SOMEONE FIX THIS SOON PLEASE!!!

[arnaudauroux]

Any update on this @SergeyKanzhelev ? I am also experiencing the same issue with the Azure Search Library and Asp.Net Core "Microsoft.Azure.Search": "3.0.1", "Microsoft.ApplicationInsights.AspNetCore": "1.0.2",

[vdevappa]

These bozos at Microsoft do not reply. They screw everyone up and they shut their mouths and we are suffering because of it - marketing says - hey! everyone c'mon use our services!

Guys, don't keep referencing these s!@#$%y links which take you on a merry go around. We don't have the time to crawl the entire internet trying to figure out how to fix the s#$t you broke.

This is supposed to be enterprise search capability - not corner mom & pop shop search which you can break like this and keep broken for 17 #@#$%^& days.

Anyway...

What I ended up doing - this is the only way to get this c@#p to work - created another project - without application insights referenced in it and then put my search code there and it works.

If you remove, uninstall app insights packages - still does not work - I already tried it.

Only caveat, the wizard keeps prompting to select the app insights when you try to deploy the cloud service everytime - but I can live with that.

[arnaudauroux]

@vdevappa I have downgraded Microsoft.Azure.Search from 3.0.1 to 1.1.3 and it solved the problem ;)

[vdevappa]

Ya, not a great solution I think to downgrade search for a reason which is not even remotely related to it (app insights). I went the other way.

[SergeyKanzhelev]

@vdevappa @arnaudauroux in what environment you are running the application? You don't need to know these details, but there are two pieces of Application Insights - SDK that you install with the application and agent that you install. In Azure Web Sites - agent is deployed by WebSite extension, in Cloud Services by WAD and in other cased for regular IIS - by Application Insights Status Monitor. We updated extension 17 days ago and Status Monitor yesterday. WAD update is pending as we need to test is as well.

So you'd need to update an agent, not only uninstall Application Insights. I still didn't get confirmation from anybody that the issue was actually fixed after update and will appreciate if you'll try it out.

[vdevappa]

I use cloud services. I know the last release did not work as you mention (WAD) because it is pending.

[SergeyKanzhelev]

In cloud services - did you enable Application Insights using WAD (Diagnostics Settings window in Visual Studio) or using a startup task?

[arnaudauroux]

I use Azure Web App

[vdevappa]

In cloud services, Application Insights was enabled a weeks ago (before I implemented search) using right click > configure/ install application insights.

[SergeyKanzhelev]

@vdevappa - can you check whether you have Application Inisghts Sink configured in wadcfg file. See this

@arnaudauroux - in the list of azure Web App extensions you should have an Application Insights. And it should say that update is available.

[vdevappa]

I see the config in the diagnostics.wadcfgx file. No instrumentation key in the below config though.

`

  </SinksConfig>`

[SergeyKanzhelev]

Please remove this sink completely and try it. I believe instrumentation key will be substituted on deployment.

[vdevappa]

I can't do this right now. Already moved on and put search in another service.

[SergeyKanzhelev]

@vdevappa thanks!

@arnaudauroux have you had a chance to try it with the updated extension?

[arnaudauroux]

@SergeyKanzhelev The update failed with the following error trace:

Failed to update web app extension Application Insights. {"Message":"ExitCode: 1, Output: 1/9/2017 9:17:10 AM: [Error] Failure happened during instrumentation: Application Insights extension doesn't support ASP.NET Core applications. Please enable Application Insights through the application project code., 
Error: , Kudu.Core.Infrastructure.CommandLineException: 
\r\nD:\\home\\SiteExtensions\\Microsoft.ApplicationInsights.AzureWebSites\\install.cmd \r\n   
at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args)\r\n   
at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args)\r\n   
at Kudu.Core.SiteExtensions.SiteExtensionManager.<>c__DisplayClass25_1.<InstallExtension>b__1()\r\n   
at Kudu.Core.Infrastructure.OperationManager.<>c__DisplayClass2_0.<Attempt>b__0()\r\n   
at Kudu.Core.Infrastructure.OperationManager.Attempt[T](Func`1 action, Int32 retries, Int32 delayBeforeRetry, Func`2 shouldRetry)\r\n   
at Kudu.Core.Infrastructure.OperationManager.Attempt(Action action, Int32 retries, Int32 delayBeforeRetry)\r\n   
at Kudu.Core.SiteExtensions.SiteExtensionManager.<InstallExtension>d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n  
at Kudu.Core.SiteExtensions.SiteExtensionManager.<TryInstallExtension>d__24.MoveNext()"}

The extension update mechanism does not apparently work with dotnet core.. :(

[SergeyKanzhelev]

@arnaudauroux this is a known limitation. I didn't realize you are using .NET Core. Anyway update of the extension will bring a new "agent" piece that will eliminate the AmbiguousMatchException problem. Is it the case for you?

[vdevappa]

Guys, I have an update here. The same error happens when application insights is installed in the cloud service web api project if you try to use any version of the MongoDB.Driver 2.4.0 or beyond.

Message="System.Reflection.AmbiguousMatchException: Multiple custom attributes of the same type found. at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit) at System.Runtime.InteropServices.RuntimeInformation.get_FrameworkDescription() at System.Lazy1.CreateValue()--- End of stack trace from previous location where exception was thrown --- at System.Lazy1.get_Value() at MongoDB.Driver.Core.Connections.ClientDocumentHelper.CreateClientDocument(String applicationName) at MongoDB.Driver.Core.Connections.BinaryConnectionFactory..ctor(ConnectionSettings settings, IStreamFactory streamFactory, IEventSubscriber eventSubscriber) at MongoDB.Driver.Core.Configuration.ClusterBuilder.BuildCluster() at MongoDB.Driver.ClusterRegistry.CreateCluster(ClusterKey clusterKey) at MongoDB.Driver.ClusterRegistry.GetOrCreateCluster(ClusterKey clusterKey) at MongoDB.Driver.MongoClient..ctor(MongoClientSettings settings) at Locus.MongoDB.Client.Initialize(String databaseName, String hostName, Int32 hostPortNo, String userName, String password) at Locus.MongoDB.LocusConfig.LocusConfigData.Database() at Locus.MongoDB.MongoConfigReader.GetConfiguration[T](String collectionName, Func2 getDefaultConfig) at Locus.Configuration.CommonDbConfig.GetLoggingConfiguration()" TraceSource="w3wp.exe"`

This is a total disaster. Now, I can't even use MongoDB because application insights won't play well with it.

Any hope of a fix anytime in the near future?

Removing the SinksConfig completely did not make any difference.

Regards, Vijay

[ishankalakshan]

@SergeyKanzhelev "We expect the fix shipped next week" Is this fix already there? We are trying to use some feature (analysers) of azure search 3.0. So there is no way to downgrade libraries. Still it works only in local environment.When deployed to azure search throws the AmbiguousMatchException.

[SergeyKanzhelev]

@Brahmnes was WAD update shipped?

[tbolon]

Hi,

We had the same problem on PowerBI Embeded. This package is using Microsoft.Rest.ClientRuntime. The package was updated to version 2.3.4 which have this problem.

I was able to restore the app by downgrading the package to 2.3.2.

This package source seems not to be hosted on github.

See stacktrace below:

System.Reflection.AmbiguousMatchException: Multiple custom attributes of the same type found.
   at System.Attribute.GetCustomAttribute(Assembly element, Type attributeType, Boolean inherit)
   at Microsoft.Rest.ServiceClient`1.get_FrameworkVersion()
   at Microsoft.Rest.ServiceClient`1.SetDefaultUserAgentInfo()
   at Microsoft.Rest.ServiceClient`1.SetUserAgent(String productName, String version)
   at Microsoft.Rest.ServiceClient`1.InitializeHttpClient(HttpClientHandler httpClientHandler, DelegatingHandler[] handlers)
   at Microsoft.PowerBI.Api.V1.PowerBIClient..ctor(ServiceClientCredentials credentials, DelegatingHandler[] handlers)

[SergeyKanzhelev]

@tbolon, what environment your application is deployed? I just got confirmation that WAD agent got updated so the bug should be gone in most environments.

• Cloud Services - WAD agent will be updated with the next deployment
• Azure Web Site - you need to update extension as shown on picture below
• Status Monitor - update via Web Platform Installer

[keremdemirer]

@SergeyKanzhelev Do we get the updated extension if we re-reploy the application from visual studio or vsts?

[SergeyKanzhelev]

@keremdemirer yes, unless you are creating a new application when deploying. Unfortunately Azure Web Apps extensions doesn't support automatic update

[keremdemirer]

@SergeyKanzhelev I'm sorry, forgot to mention. I deploy to Cloud Service web and worker roles. Does the same applies?

[SergeyKanzhelev]

@keremdemirer , do you update or redeploy and VIP swap? I believe the agent will be updated on redeploy, not on update, +@brahmnes to confirm

[brahmnes]

A new WAD update hasn't shipped yet, hopefully we can update later this week. Updating the agent won't help until that has happened.

[tbolon]

@SergeyKanzhelev You are right, extension was out of date. We have updated them to the latest revision (we are using azure web site). Thanks.

[YZahringer]

Same exception with v2.3.5 on UWP application with Xamarin. No problem with v2.3.2.

Only in Release mode with .NET Native tool chain activated.

[SergeyKanzhelev]

The root cause of original problem was the Application Insights profiler that will inject extra custom attributes. We do not enable profiler for UWP apps so the root cause should be different. Maybe Native tool chain is doing something similar to what we do for Application Insights

Does this code returns multiple values for you as well?

System.Attribute.GetCustomAttributes((typeof(object)).Assembly)

What attributes will this code return:

Console.WriteLine("Number of custom attributes:" + System.Attribute.GetCustomAttributes((typeof(object)).Assembly).Length);

foreach(var a in System.Attribute.GetCustomAttributes((typeof(object)).Assembly))
{
    Console.WriteLine("attribute: " + a.TypeId);
}

[peters]

Same defect can be observed using the MongoDB driver 2.4.2 driver: https://jira.mongodb.org/plugins/servlet/mobile#issue/CSHARP-1863

[peters]

Apparently uninstalling the application insights toolkit resolves the issue. The version installed on the servers was pretty old and since we are bundling application insights with our web app it was never required anyway.

[mpbelov]

@SergeyKanzhelev I faced with this issue using MongoDB driver. As I undesrtood, the issue was resolved and Azure services were updated with the fixed version of AppInsights. However I am still experiencing it. I've tried to redeploy my cloud service, delete and re-create it from scratch, update my VS components. Nothing helped. Only switching off Azure diagnostics when deploying resolved it. How to get fixed version of monitoring tools for cloud services?