Test failure: HelloWorld_Nginx

[aspnet-hello]

This test fails occasionally with the following error:

System.Exception : Failed to start nginx
   at Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer.SetupNginx(String redirectUri, Uri originalUri)
   at Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer.DeployAsync()
   at ServerComparison.FunctionalTests.HelloWorldTests.HelloWorld(ServerType serverType, RuntimeFlavor runtimeFlavor, RuntimeArchitecture architecture, ApplicationType applicationType, String testName, HostingModel hostingModel, String additionalPublishParameters) in /_/test/ServerComparison.FunctionalTests/HelloWorldTest.cs:line 95
--- End of stack trace from previous location where exception was thrown ---
------- Stdout: -------
| [2018-08-30T23:04:56] TestLifetime Information: Starting test HelloWorld_Nginx-CoreClr-Standalone
| [2018-08-30T23:04:56] TestLifetime Information: Starting test HelloWorld_Nginx_Nginx_CoreClr_x64_Standalone
| [2018-08-30T23:04:56] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Executing dotnet run --no-build -c Release --framework netcoreapp2.1 -- --server.urls http://localhost:46852/  --server Microsoft.AspNetCore.Server.Kestrel
| [2018-08-30T23:04:56] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: SET ASPNETCORE_ENVIRONMENT=HelloWorld
| [2018-08-30T23:04:56] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: SET ASPNETCORE_DETAILEDERRORS=true
| [2018-08-30T23:04:56] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Started dotnet. Process Id : 10948
| [2018-08-30T23:04:56] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: host process ID 10893 shut down
| [2018-08-30T23:04:56] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Host process shutting down.
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet stdout: Using launch settings from /mnt/work/33bdfc1cae7b2a38/modules/ServerTests/test/ServerComparison.TestSites/Properties/launchSettings.json...
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: dotnet stderr: The launch profile "(Default)" could not be applied.
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: dotnet stderr: A usable launch profile could not be located.
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet stdout: Hosting environment: HelloWorld
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet stdout: Content root path: /mnt/work/33bdfc1cae7b2a38/modules/ServerTests/test/ServerComparison.TestSites/bin/Release/netcoreapp2.1/
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet stdout: Now listening on: http://localhost:46852
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet stdout: Application started. Press Ctrl+C to shut down.
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
| [2018-08-30T23:04:57] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:40912 failed (98: Address already in use)
| [2018-08-30T23:04:58] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:40912 failed (98: Address already in use)
| [2018-08-30T23:04:58] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:40912 failed (98: Address already in use)
| [2018-08-30T23:04:59] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:40912 failed (98: Address already in use)
| [2018-08-30T23:04:59] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:40912 failed (98: Address already in use)
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] still could not bind()
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx stop stderr: nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx stop stderr: 2018/08/30 23:05:00 [notice] 10989#10989: signal process started
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx stop stderr: 2018/08/30 23:05:00 [error] 10989#10989: open() "/mnt/work/33bdfc1cae7b2a38/modules/ServerTests/test/ServerComparison.TestSites/52297a1f-84ac-478b-8db6-77f275ade5bf.nginx.pid" failed (2: No such file or directory)
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: nginx stop command issued
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Attempting to cancel process 10948
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet stdout: Application is shutting down...
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: Unable to terminate the host process with process Id '10948
| [2018-08-30T23:05:00] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: [Time]: Total time taken for this test variation '0' seconds
| [2018-08-30T23:05:00] TestLifetime Information: Finished test HelloWorld_Nginx_Nginx_CoreClr_x64_Standalone in 3.9340713s

Other tests within that build may have failed with a similar message, but they are not listed here. Check the link above for more info.

This test failed on 2.1.

CC @Eilon (because the bot doesn't know who else to pick)

This issue was made automatically. If there is a problem contact ryanbrandenburg.

[aspnet-hello]

Please use this workflow to address this flaky test issue, including checking applicable checkboxes and filling in the applicable "TODO" entries:

• Is this actually a flaky test?

  • No, this is a regular test failure, fix the test/product (TODO: Link to commit/PR)
  • Yes, proceed below...

• Is this test failure caused by product code flakiness? (Either this product, or another product this test depends on.)

  • [ ] File a bug against the product (TODO: Link to other bug)
  • Is it possible to change the test to avoid the flakiness?
  • Yes? Go to the "Change the test!" section.
  • No?
    • [ ] Disable the test (TODO: Link to PR/commit)
    • [ ] Wait for other bug to be resolved
    • [ ] Wait for us to get build that has the fix
    • [ ] Re-enable our test (TODO: Link to PR/commit)
    • [ ] Close this bug

• Is it that the test itself is flaky? This includes external transient problems (e.g. remote server problems, file system race condition, etc.)

  • Is there is a way to change our test to avoid this flakiness?
  • Yes? Change the test!
    • [ ] Change the test to avoid the flakiness, for example by using a different test strategy, or by adding retries w/ timeouts (TODO: Link to PR/commit)
    • [ ] Run the test 100 times locally as a sanity check.
    • [ ] Close this bug
  • No?
    • Is there any logging or extra information that we could add to make this more diagnosable when it happens again?
    • Yes?
      • [ ] Add the logging (TODO: Link to PR/commit)
    • No?
      • [ ] Delete the test because flaky tests are not useful (TODO: Link to PR/commit)

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[aspnet-hello]

ResponseCompression_AppAndHostCompression failed with about the same error on 2.2.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[Eilon]

@muratg can you assign?

[muratg]

@Tratcher could you take an initial look please?

[Tratcher]

Hmm, it's still having port conflicts bind() to 0.0.0.0:40912 failed (98: Address already in use)

[aspnet-hello]

HelloWorld failed with about the same error on master.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[Tratcher]

This most recent failure is unrelated, there's a bad corefx in play. https://github.com/aspnet/Home/issues/3544

[aspnet-hello]

There were 1 failures with about the same error on 2.2:

• ResponseFormats_ManuallyChunk

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[aspnet-hello]

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 10:55:30 AM:

• HelloWorld

There were 1 failures with about the same error on master at 1:21:35 PM:

• ResponseCompression_HostCompression

[aspnet-hello]

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on 2.1 at 4:33:46 PM:

• ResponseFormats_Nginx_ManuallyChunk

There were 1 failures with about the same error on 2.1 at 1:36:28 AM:

• ResponseCompression_Nginx_HostCompression

[aspnet-hello]

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on 2.1 at 4:47:15 PM:

• ResponseCompression_Nginx_NoCompression

There were 1 failures with about the same error on master at 6:40:36 PM:

• ResponseFormats_Chunked

[aspnet-hello]

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on 2.2 at 12:10:10 PM:

• ResponseCompression_AppAndHostCompression

[aspnet-hello]

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 7:03:30 PM:

• ResponseFormats_Chunked

[Tratcher]

More [emerg] bind() to 0.0.0.0:37971 failed (98: Address already in use) failures, all branches.

[Tratcher]

Closing this. We've improved the test logging so we'll see what we get if this happens again.