Azure Web Role recycles when Swashbuckle.Core reference is present

[dendle]

Hello All,

I am using Swashbuckle in a WebAPI project, and everything works perfectly in IIS locally.

It Also works fine when I run in from within IISExpress.

As soon as I send it to azure the role keeps recycling. The error code I get is that the role exited with error code 0. If I remote into the machine, the AppPool just dies during startup. Nothing in the event log.

The normal cause of role recycling is missing references, but this is not the case - If I extract the web app from the package file, and point IIS at it, it again, works perfectly. I also see the Swashbuckle assembly in the package.

If I create a brand new WebAPI project, it works fine. It seems as if it doesn't play well with the existing WebAPI project for some reason.

Its rather a large one, and I am currently ruling out interaction with other nugget packages, by slowly removing each one and seeing if the problem goes away. No luck yet.

Has anyone else experienced this issue? Can anyone think of a reason why this might happen??

Thanks for any help in advance, and I will post my results of the nuget package elimination shortly.

Thanks, Matt

[kpfaulkner]

Had a very late night fighting that as well. IIS Express fine, Azure compute emulator fine..... reality.... broken.

I was previously trying to get it working with Owin + Azure Worker roles, but no luck so last night I ripped the guts out of my API and shifted it to a regular Azure Web Role, WebAPI2.... no Owin. Still no luck..... (wanted to get rid of Owin due to some dependency issues anyway).

Am literally at the keyboard now fighting this problem. If I get anywhere I'll post it.

[dendle]

I have got somewhere! It appears if I use the RoleEntryPoint class to do stuff with IIS config, (keep it "always on" so it doesn't need to wake up after a busy period) The role blows up.

If I comment it out, it works fine.

Just confirming now, and will post more detail in about an hour.

Cheers, Matt

[dendle]

Confirmed!

Here is what I had: public class WebRoleEntryPoint : RoleEntryPoint { private static volatile bool _configuredIIS; public override bool OnStart() { RoleEnvironment.StatusCheck += RoleEnvironment_StatusCheck; return base.OnStart(); }

    void RoleEnvironment_StatusCheck(object sender, RoleInstanceStatusCheckEventArgs e)
    {
        if (e.Status == RoleInstanceStatus.Ready && !_configuredIIS)
        {
            lock (typeof(WebRoleEntryPoint))
            {
                LogEvent(String.Format("{0} IIS Configuraion Starting...", DateTime.UtcNow));
                ConfigureIIS();
                _configuredIIS = true;
                LogEvent(String.Format("{0} IIS Configuration Complete", DateTime.UtcNow));
            }
        }
    }

    private void ConfigureIIS()
    {
        ServerManager iis = new ServerManager();
        foreach (Microsoft.Web.Administration.Site site in iis.Sites)
        {
            foreach (var application in site.Applications)
            {
                application.SetAttributeValue("preloadEnabled", true);
            }
            site.ApplicationDefaults.SetAttributeValue("preloadEnabled", true);
            site.ServerAutoStart = true;
        }
        foreach (ApplicationPool applicationPool in iis.ApplicationPools)
        {
            if (!applicationPool.Recycling.PeriodicRestart.Schedule.Any())
            {
                applicationPool.Recycling.PeriodicRestart.Schedule.Add(new TimeSpan(3, 0, 0));
            }
        }
        iis.CommitChanges();
    }

    private void LogEvent(string message)
    {
        const string sSource = "MyApiSource";
        const string sLog = "Application";
        string sEvent = message;

        if (!EventLog.SourceExists(sSource))
        {
            EventLog.CreateEventSource(sSource, sLog);
        }
        EventLog.WriteEntry(sSource, sEvent, EventLogEntryType.Information);
    }
}

If I comment that out, everything works.

I'm narrowing things down now by slowly adding stuff till it breaks again.

Cheers, Matt

[dendle]

It appears as though all calls to the EventLog class below cause the AppPool crash and therefore the role to recycle.

If I comment the below method, and its calls, Swashbuckle works fine.

I can see no correlation between this code and the Swashbuckle source. Its worth noting that I am not using WebActivatorEx, and I am not configuring or enabling Swashbuckle - im just adding the reference to Swashbuckle.Core - that is enough to crash it.

private void LogEvent(string message)
{
    const string sSource = "MyApiSource";
    const string sLog = "Application";
    string sEvent = message;

    if (!EventLog.SourceExists(sSource))
    {
        EventLog.CreateEventSource(sSource, sLog);
    }
    EventLog.WriteEntry(sSource, sEvent, EventLogEntryType.Information);
}

Not sure where to go from here, so for now, I'll just leave it commented, and hope this ticket helps others with similar problems.

Cheers, Matt

[kpfaulkner]

Very very interesting. I know I'm not directly writing any event logs.... but am willing to bet something under the covers is doing that. Will try it out locally and see what happens.

Cheers

Ken

On Wed, Jun 10, 2015 at 10:07 PM, dendle notifications@github.com wrote:

It appears as though all calls to the EventLog class below cause the AppPool crash and therefore the role to recycle.

If I comment the below method, and its calls, Swashbuckle works fine.

I can see no correlation between this code and the Swashbuckle source. Its worth noting that I am not using WebActivatorEx, and I am not configuring or enabling Swashbuckle - im just adding the reference to Swashbuckle.Core

• that is enough to crash it.

private void LogEvent(string message) { const string sSource = "MyApiSource"; const string sLog = "Application"; string sEvent = message;

if (!EventLog.SourceExists(sSource))
{
    EventLog.CreateEventSource(sSource, sLog);
}
EventLog.WriteEntry(sSource, sEvent, EventLogEntryType.Information);

}

Not sure where to go from here, so for now, I'll just leave it commented, and hope this ticket helps others with similar problems.

Cheers, Matt

— Reply to this email directly or view it on GitHub https://github.com/domaindrivendev/Swashbuckle/issues/380#issuecomment-110721276 .

[ptjhuang]

I had the same issue. Here's my solution. Make a copy of Web.config, and call it YourWebApp.dll.config, then redeploy the app.

Here's my take on what's happened (and many other role recycling issues that's related to dependencies).

• Your web app depends on some version 2.0 of library X.
• However, you've another dependency library Y which depends on version 1.0 of library X.
• WaIISHost tries to load up your web project's assembly to call into RoleEntryPoint
• But it can't resolve the version incompatibility without any assembly redirect instructions

Normally this is dealt with in web.config, however WaIISHost.exe - the Azure web application host process, doesn't read it. But it will read YourWebApp.dll.config for redirect instructions.

You could get away without redirects when the startup code can successfully bind without running into conflicting dependencies, but as soon as your app gets complex and feature rich, chances of running into startup code that has conflicting depdencies are pretty good.

I found this issue following these steps:

1. Checked event viewer, under Azure Application, and found the startup TypeLoadException
2. Enabled fusion log to see the dependency chain - found WaIISHost -> swashbuckle.core -> system.web.http 4
3. Checked github source and see that it does depend on system.web.http 4

[dendle]

Thank you angrybug, for posting your solution - It seems like a good explanation of the root cause.

I will try your solution and see if it works.

[amanzoor]

Any update on this? I am facing similar issues, I am not doing an custom configuration on the role entry point to IIS or event logs also tried adding YourWebApp.dll.config on the web role VM instance to no avail.

[dendle]

Im trying the suggestion from @angrybug, and will report back my findings.

My suggestion doesn't actually work anymore - I have no idea why.

[dendle]

No luck either I'm afraid.

I am scheduled to have some time to spend on tracking this down in my project, but not till next month.

Anyone else able to shed some light on this problem?

Matt

[lakeba]

I am getting this error in IIS 8.0, is probably related to your problem?

[VerificationException: Method System.Net.Http.CloneableExtensions.Clone: type argument 'System.Net.Http.Headers.MediaTypeHeaderValue' violates the constraint of type parameter 'T'.] System.Net.Http.Formatting.MediaTypeConstants.get_ApplicationJsonMediaType() +0 System.Net.Http.Formatting.JsonMediaTypeFormatter..ctor() +79 System.Net.Http.Formatting.MediaTypeFormatterCollection.CreateDefaultFormatters() +49 System.Web.Http.HttpConfiguration..ctor(HttpRouteCollection routes) +255 System.Web.Http.GlobalConfiguration.b__0() +93 System.Lazy1.CreateValue() +14337348 System.Lazy1.LazyInitValue() +524 Easy.Docs.API.SwaggerConfig.Register() +73

[TargetInvocationException: Exception has been thrown by the target of an invocation.] System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0 System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +229 System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +211 System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) +35 WebActivatorEx.ActivationManager.RunActivationMethods(Boolean designerMode) +445 WebActivatorEx.ActivationManager.Run() +105

[InvalidOperationException: The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..] System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection1 methods, Func1 setHostingEnvironmentCultures) +12619651 System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection`1 methods) +12619372 System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +280 System.Web.Compilation.BuildManager.ExecutePreAppStart() +172 System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +1151

[HttpException (0x80004005): The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..] System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +12618692 System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +159 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +12458309

[mahmoud-samy]

@angrybug finding is confirmed using AsmSpy:

Reference: System.Web.Http
   5.2.2.0 by MyWebApp
   4.0.0.0 by MyWebApp
   4.0.0.0 by Swashbuckle.Core
   5.2.2.0 by System.Web.Http.Owin
   5.2.2.0 by System.Web.Http.WebHost

Swashbuckle.Core reference System.Web.Http v4.0

[javitoHertfy]

Hi guys,

any update with this? I had the same issue last week, we were deploying our cloud services just fine and then we updated one package and now we cannot deploy anymore with swashbucle, any ideas?

[javitoHertfy]

Why as soon as I add Swashbuckle appears twice FXStreet.Post.I3.WebApi same as @mahmoud-samy-symbyo Reference: System.Web.Http 5.2.0.0 by Autofac.Integration.WebApi 5.2.3.0 by FXStreet.Post.I3.WebApi 5.2.2.0 by FXStreet.Post.I3.WebApi 5.2.3.0 by FXStreet.SwaggerExtensions 5.2.3.0 by FXStreet.Web 5.2.2.0 by Swashbuckle.Core 5.2.3.0 by System.Web.Http.Cors 5.2.3.0 by System.Web.Http.WebHost

[jesuseib]

@javitoHertfy, I had the exact same problem, and I made it work by installing a Nuget Package that is a fork from this project. Nuget Package: https://www.nuget.org/packages/Swashbuckle.Net45/ Fork: https://github.com/SEEK-Jobs/Swashbuckle

[withinboredom]

This appears to also happens if you update to webapi/mvc 5, which uses a later version of system.web.http ...

[javitoHertfy]

@jesuseib thanks for the info, I installed the fork and still having the same issue, also I have installed the 5.2.2 and the bug is still there, any plans of fixing it?

[JohnnyMaxK]

@jesuseib Why you do not merge to the master this change ? Thanks for your help !

[jesuseib]

@Johnny-Bee, I just wanted to make it work in Azure as fast as I could, and I found this branch was targeted to .NET framework 4.5, it solved my issue and I haven't payed much attention since then. However, as of Oct 18th, it didn't work for @javitoHertfy, so I can't ensure that it'd solve your problem.

One thing you can try, instead of using the nuget package, is to download the code from master, include it in your solution, try to make it compile (probably at this step you'll find out the dll reference problem by running it locally) and push it to Azure. I can't guarantee it's going to work, but at least I'd give it a try.

[JohnnyMaxK]

@jesuseib @javitoHertfy Guys,

For your information. We have this error on the event viewer on Azure Web Role, after activating the assembly detection log in the windows registry:

The description for Event ID 1007 from source Windows Azure Runtime 2.7.0.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

868
WaIISHost
Role entrypoint could not be created: System.TypeLoadException: Unable to load the role entry point due to the following exceptions:
-- System.IO.FileLoadException: Could not load file or assembly 'System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

=== Pre-bind state information ===
LOG: DisplayName = System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///E:/approot/bin
LOG: Initial PrivatePath = E:\approot\bin
Calling assembly : Swashbuckle.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cd1bb07a5ac7c7bc.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: E:\base\x64\WaIISHost.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: Attempting download of new URL file:///E:/approot/bin/System.Web.Http.DLL.
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

-- System.IO.FileLoadException: Could not load file or assembly 'System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

=== Pre-bind state information ===
LOG: DisplayName = System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///E:/approot/bin
LOG: Initial PrivatePath = E:\approot\bin
Calling assembly : Swashbuckle.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cd1bb07a5ac7c7bc.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: E:\base\x64\WaIISHost.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: System.Web.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
LOG: Attempting download of new URL file:///E:/approot/bin/System.Web.Http.DLL.
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

 ---> System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly)
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CreateRoleEntryPoint(RoleType roleTypeEnum)
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum)

the message resource is present but the message is not found in the string/message table

Locally we do not have this issue because it's only on Azure the issue. We have redirected the assembly to the latest version in the web.config.

      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http.Formatting" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.2.3.0" newVersion="5.2.3.0" />
      </dependentAssembly>

The problem is when the WaIISHost.exe who loads the role entry point asembly in the approot folder without loading the web.config.

If your web application is called for example: toto, then WaIISHost.exe will make Assembly.LoadFrom("approot/toto.dll"), so we have to make sure that the assembly redirection of the web.config is correctly taken in account by the WaIISHost.exe using toto.dll.config.

http://blogs.msdn.com/b/friis/archive/2014/05/15/webrole-entry-point-and-config-file.aspx

For that you just have to create in your web project, the file toto.dll.config with copy always activated, this file can be empty it will contains the content of your web.config automatically and will be taken in account by the azure package.

So now, we don't need to upgrade the swashbuckle dependencies... :-)

[jesuseib]

Great explanation @Johnny-Bee! thanks for sharing

[javitoHertfy]

@Johnny-Bee thank you very much!! it worked like a charm! You made my week :dancer:

[dendle]

Thanks @Johnny-Bee!

Closing....

[maf1024]

I created the empty [projectname].dll.config file as suggested. (With the actual web.config containing the redirects) But when I debug, the Azure compute emulator encounters a System.Xml.XmlException "Root element is missing", which seems to cause the web role to get stuck in a cycle of "Role state Unknown" "Role state Destroyed".

[TylerAngell]

Add the below bit of code to your Build Events for your WebRole. The file isn't auto-populated otherwise with the existing web.config.

copy "$(ProjectDir)Web.config" "$(TargetDir)../$(TargetFileName).config"

PS - Dont forget to set [projectname].dll.config to Copy Always or Copy If Newer

[auonhaidar]

@Johnny-Bee How did you get that log? I cannot login in to my VM cause of my WebRole not starting with the same ReflectionTypeLoadException. After this error happens, VM keeps getting restarted so I cannot remote login. How did you get inside VM?