app lifecycle memory leak endpoint has mutex issue

madamkiwi commented 6 years ago

While running wats the following test failed:

• Failure [38.023 seconds]
Application Lifecycle
/tmp/build/2b769089/wats/src/wats/limits_test.go:9
  An app staged on Diego and running on Diego
  /tmp/build/2b769089/wats/src/wats/limits_test.go:10
    exercises the app through its lifecycle [It]
    /tmp/build/2b769089/wats/src/wats/limits_test.go:11

    Expected
        <string>: <!DOCTYPE html>
        <html>
            <head>
                <title>Mutex could not be created.</title>
                <meta name="viewport" content="width=device-width" />
                <style>
                 body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
                 p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
                 b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
                 H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
                 H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
                 pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
                 .marker {font-weight: bold; color: black;text-decoration: none;}
                 .version {color: gray;}
                 .error {margin-bottom: 10px;}
                 .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
                 @media screen and (max-width: 639px) {
                  pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
                 }
                 @media screen and (max-width: 479px) {
                  pre { width: 280px; }
                 }
                </style>
            </head>

            <body bgcolor="white">

                    <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>

                    <h2> <i>Mutex could not be created.</i> </h2></span>

                    <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

                    <b> Description: </b>An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

                    <br><br>

                    <b> Exception Details: </b>System.InvalidOperationException: Mutex could not be created.<br><br>

                    <b>Source Error:</b> <br><br>

                    <table width=100% bgcolor="#ffffcc">
                       <tr>
                          <td>
                              <code>

        An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.</code>

                          </td>
                       </tr>
                    </table>

                    <br>

                    <b>Stack Trace:</b> <br><br>

                    <table width=100% bgcolor="#ffffcc">
                       <tr>
                          <td>
                              <code><pre>

        [InvalidOperationException: Mutex could not be created.]
           System.Web.Compilation.CompilationMutex..ctor(String name, String comment) +12254195
           System.Web.Compilation.CompilationLock..cctor() +256

        [TypeInitializationException: The type initializer for &#39;System.Web.Compilation.CompilationLock&#39; threw an exception.]
           System.Web.Compilation.CompilationLock.GetLock(Boolean&amp; gotLock) +39
           System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate(Int64 cachedHash) +46
           System.Web.Compilation.BuildManager.ExecutePreAppStart() +148
           System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +878

        [HttpException (0x80004005): The type initializer for &#39;System.Web.Compilation.CompilationLock&#39; threw an exception.]
           System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +525
           System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +124
           System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +700
        </pre></code>

                          </td>
                       </tr>
                    </table>

                    <br>

                    <hr width=100% size=1 color=silver>

                    <b>Version Information:</b>&nbsp;Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.7.2106.0

                    </font>

            </body>
        </html>
        <!-- 
        [InvalidOperationException]: Mutex could not be created.
           at System.Web.Compilation.CompilationMutex..ctor(String name, String comment)
           at System.Web.Compilation.CompilationLock..cctor()
        [TypeInitializationException]: The type initializer for &#39;System.Web.Compilation.CompilationLock&#39; threw an exception.
           at System.Web.Compilation.CompilationLock.GetLock(Boolean& gotLock)
           at System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate(Int64 cachedHash)
           at System.Web.Compilation.BuildManager.ExecutePreAppStart()
           at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
        [HttpException]: The type initializer for &#39;System.Web.Compilation.CompilationLock&#39; threw an exception.
           at System.Web.HttpRuntime.FirstRequestInit(HttpContext context)
           at System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context)
           at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
        --><!-- 
        This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using &lt;customErrors mode="Off"/&gt;. Consider using &lt;customErrors mode="On"/&gt; or &lt;customErrors mode="RemoteOnly"/&gt; in production environments.-->
    to contain substring
        <string>: Insufficient memory

    /tmp/build/2b769089/wats/src/wats/limits_test.go:22

Our cf was running diego 1.7.1, garden windows 0.7.0, wats f7b4ea2715c15e479eaf45331d90081af9145a06.

This was a flake because the wats passed afterwards. Just wanted to let you know so this flake can be resolved.

cc @kkallday

cf-gitbot commented 6 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/153647845

The labels on this github issue will be updated when the story is started.

jvshahid commented 6 years ago

created a gist with the output from wats in our CI where we saw the same error https://gist.github.com/jvshahid/5c551119a123b02a8cbf4ca3af3de32c

sunjayBhatia commented 6 years ago

We have tried to reproduce this by continuously/concurrently pushing the nora test app and curling it but have not been successful in seeing this issue again. We can open this up again if it resurfaces/becomes a blocker for y'all.

kkallday commented 6 years ago

Gotcha. If it happens again, I'll keep the env around and let you guys know.

On Wed, Jan 3, 2018, 12:11 Sunjay Bhatia notifications@github.com wrote:

Closed #21 https://github.com/cloudfoundry/wats/issues/21.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cloudfoundry/wats/issues/21#event-1408456232, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4kkddaiFRbK8l_lBMMD7ALD3-WNzHuks5tG97YgaJpZM4RBHhi .

ljfranklin commented 6 years ago

@sunjayBhatia we've seen this error in two more environments today. Would you like to take a look at one? Is there any other log output that would be helpful?

sunjayBhatia commented 6 years ago

@ljfranklin we saw this recently too in our CI and reopened the story so I will reopen this issue as well

I don’t think you need to keep the environment around for us. The cell BOSH logs and Windows Event Log have not told us much as we have an environment up with the same issue but there are some things we can do with a modified buildpack to get some more .NET debug tracing out

It looks like the issue is coming from an exception in the actual .NET Framework in initializing a lock, not in the app so that could be fortunate/unfortunate depending on your point of view

cf-gitbot commented 6 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/154384516

The labels on this github issue will be updated when the story is started.

sunjayBhatia commented 6 years ago

This issue will be sorted out once the next release of hwc reaches the hwc-buildpack

(this commit: https://github.com/cloudfoundry-incubator/hwc/commit/19ed675a69b616fbef03b47f66f70b65a762db13 ensures that the compilation mutex that is failing to be acquired/created is unique per app)

cloudfoundry-attic / wats

app lifecycle memory leak endpoint has mutex issue #21