Flaky test: HelloWorld

[ryanbrandenburg]

This test fails occasionally with the following error:

System.InvalidOperationException : Failed to start nginx
   at Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer.SetupNginx(String redirectUri, Uri originalUri)
   at Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer.DeployAsync()
   at ServerComparison.FunctionalTests.HelloWorldTests.HelloWorld(TestVariant variant) in /_/test/ServerComparison.FunctionalTests/HelloWorldTest.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
------- Stdout: -------
| [0.001s] TestLifetime Information: Starting test HelloWorld-Server: Nginx, TFM: netcoreapp2.2, Type: Standalone, Arch: x64 at 2018-07-28T02:41:17
| [0.001s] TestLifetime Information: Starting test HelloWorld_Nginx_netcoreapp2.2_x64_Standalone at 2018-07-28T02:41:17
| [0.019s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Removing environment variable ASPNETCORE_ENVIRONMENT
| [0.020s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Executing command dotnet publish  --output "/mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1" --framework netcoreapp2.2 --configuration Release --no-restore -p:VerifyMatchingImplicitPackageVersion=false --runtime linux-x64 
| [1.031s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: host process ID 100765 shut down
| [1.033s] Microsoft.AspNetCore.Server.IntegrationTesting.SelfHostDeployer Information: Host process shutting down.
| [0.368s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet-publish stdout: Microsoft (R) Build Engine version 15.7.179.6572 for .NET Core
| [0.369s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet-publish stdout: Copyright (C) Microsoft Corporation. All rights reserved.
| [2.423s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet-publish stdout:   ServerComparison.TestSites -> /mnt/work/33bdfc1cae7b2a38/modules/ServerTests/test/ServerComparison.TestSites/bin/Release/netcoreapp2.2/linux-x64/ServerComparison.TestSites.dll
| [3.698s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet-publish stdout:   ServerComparison.TestSites -> /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/
| [3.758s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: dotnet publish finished with exit code : 0
| [3.758s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Executing /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/ServerComparison.TestSites  --urls http://localhost:37444/ --server Microsoft.AspNetCore.Server.Kestrel
| [3.758s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Removing environment variable ASPNETCORE_ENVIRONMENT
| [3.759s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: SET ASPNETCORE_DETAILEDERRORS=true
| [3.762s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: Started /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/ServerComparison.TestSites. Process Id : 100845
| [4.243s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/ServerComparison.TestSites stdout: Hosting environment: Production
| [4.244s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/ServerComparison.TestSites stdout: Content root path: /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/
| [4.245s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/ServerComparison.TestSites stdout: Now listening on: http://localhost:37444
| [4.245s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Information: /mnt/temp/buildTmp/ccec2cc6-c1c0-46f0-a0f6-a8f37c078bd1/ServerComparison.TestSites stdout: Application started. Press Ctrl+C to shut down.
| [4.265s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
| [4.270s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:36400 failed (98: Address already in use)
| [4.771s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:36400 failed (98: Address already in use)
| [5.271s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:36400 failed (98: Address already in use)
| [5.771s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:36400 failed (98: Address already in use)
| [6.271s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:36400 failed (98: Address already in use)
| [6.771s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx start stderr: nginx: [emerg] still could not bind()
| [6.785s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx stop stderr: nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
| [6.788s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx stop stderr: 2018/07/28 02:41:24 [notice] 100885#100885: signal process started
| [6.788s] Microsoft.AspNetCore.Server.IntegrationTesting.NginxDeployer Warning: nginx stop stderr: 2018/07/28 02:41:24 [error] 100885#100885: open() "/mnt/work/33bdfc1cae7b2a38/modules/ServerTests/test/ServerComparison.TestSites/bfba1321-6197-4b16-82e8-bc9b03524bb4.nginx.pid" failed (2: No such file or directory)
| [6.788s] 
...

Other tests within that build may have failed with a similar message, but they are not listed here. Check the link above for more info.

CC @Eilon,@muratg,@mkArtakMSFT

This issue was made automatically. If there is a problem contact @ryanbrandenburg.

[ryanbrandenburg]

Please use this workflow to address this flaky test issue, including checking applicable checkboxes and filling in the applicable "TODO" entries:

• Is this actually a flaky test?

  • No, this is a regular test failure, fix the test/product (TODO: Link to commit/PR)
  • Yes, proceed below...

• Is this test failure caused by product code flakiness? (Either this product, or another product this test depends on.)

  • [ ] File a bug against the product (TODO: Link to other bug)
  • Is it possible to change the test to avoid the flakiness?
  • Yes? Go to the "Change the test!" section.
  • No?
    • [ ] Disable the test (TODO: Link to PR/commit)
    • [ ] Wait for other bug to be resolved
    • [ ] Wait for us to get build that has the fix
    • [ ] Re-enable our test (TODO: Link to PR/commit)
    • [ ] Close this bug

• Is it that the test itself is flaky? This includes external transient problems (e.g. remote server problems, file system race condition, etc.)

  • Is there is a way to change our test to avoid this flakiness?
  • Yes? Change the test!
    • [ ] Change the test to avoid the flakiness, for example by using a different test strategy, or by adding retries w/ timeouts (TODO: Link to PR/commit)
    • [ ] Run the test 100 times locally as a sanity check.
    • [ ] Close this bug
  • No?
    • [ ] Delete the test because flaky tests are not useful (TODO: Link to PR/commit)

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[ryanbrandenburg]

HelloWorld failed again.

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[Eilon]

@Tratcher - please take a look at this high priority flaky test issue.

[ryanbrandenburg]

ResponseCompression_HostCompression failed with about the same error. ResponseCompression_Nginx_NoCompression failed again.

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[Tratcher]

HelloWorldTests.HelloWorld(variant: Server: Nginx, TFM: netcoreapp2.2, Type: Standalone, Arch: x64) HelloWorldTests.HelloWorld(variant: Server: Nginx, TFM: netcoreapp2.2, Type: Portable, Arch: x64)

• 1 failure each in recorded test history, in the same run.

ResponseCompressionTests.ResponseCompression_Nginx_NoCompression(serverType: Nginx, runtimeFlavor: CoreClr, architecture: x64, applicationType: Standalone)

• only 1 failure since an infrastructure problem was resolved in May.

All of the above have two interesting errors: 1) [warn] 30990#30990: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /mnt/temp/buildTmp/tmp2zylcu.tmp:2 @natemcmaster didn't you just make changes to nginx user profiles? Is this warning concerning? https://github.com/aspnet/Hosting/commit/fed2eb1cdfef598b5315da045f741403fad89e05 2) nginx start stderr: nginx: [emerg] bind() to 0.0.0.0:33636 failed (98: Address already in use) - Transient port conflict. This happened once per test, all on the same agent. This port was selected by TestPortHelper.GetNextPort() which binds to 0 and lets the OS choose. This may be a temporary resource issue on this agent. Recommend waiting for additional repros.

[natemcmaster]

the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /mnt/temp/buildTmp/tmp2zylcu.tmp

I didn't change userprofiles, just made the logic for figuring out the current username stronger. It was a fix for VSTS tests which failed to boot because the LOGNAME env var was empty.

[ryanbrandenburg]

ResponseCompression_HostCompression failed with about the same error. ResponseFormats_Nginx_ContentLength failed again.

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[ryanbrandenburg]

HelloWorld failed with about the same error. HelloWorld failed with about the same error.

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[Tratcher]

Hmm, these new failures are all the same port conflict on different tests for the same OS (Ubuntu). There's no history of flakiness here, it's become flaky in the last two weeks.

[ryanbrandenburg]

ResponseFormats_ManuallyChunk failed with about the same error.

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[Tratcher]

Resolved by https://github.com/aspnet/Hosting/pull/1515