Automated tests failed!

[neonforgedevbot]

Outcome:	TESTS FAILED
Branch:	master
Config:	debug
Filter:	Category!=buggy&Category!=investigate
Commit:	dce41c601a194cb33f355da4ad66976e63c36433
Runner:	WRT-00-GITHUB-RUNNER-01
Workflow Run:	link
Workflow:	link

[neonforgedevbot]

Outcome:	TESTS FAILED
Branch:	master
Config:	debug
Filter:	Category!=buggy&Category!=investigate
Commit:	-na-
Runner:	WRT-00-GITHUB-RUNNER-01
Workflow Run:	link
Workflow:	link
Test.Neon.Cassandra.csproj:	✔ details - 00:02:25 pass: 16 fail: 0 skipped: 0
Test.Neon.Common.csproj:	❌ details - 00:03:21 pass: 601 fail: 3 skipped: 23
Test.Neon.Cryptography.csproj:	❌ details - 00:00:08 pass: 45 fail: 1 skipped: 2
Test.Neon.Deployment.csproj:	⚠ details - 00:00:28 pass: 122 fail: 0 skipped: 2
Test.Neon.Identity.csproj:	✔ details - 00:01:23 pass: 1 fail: 0 skipped: 0
Test.Neon.Kube.csproj:	✔ details - 00:00:09 pass: 16 fail: 0 skipped: 0
Test.Neon.Postgres.csproj:	✔ details - 00:06:37 pass: 20 fail: 0 skipped: 0
Test.Neon.Service.csproj:	❌ details - 00:01:40 pass: 26 fail: 1 skipped: 0
Test.Neon.YugaByte.csproj:	✔ details - 00:03:22 pass: 8 fail: 0 skipped: 0
Test.NeonCli.csproj:	✔ details - 00:00:17 pass: 24 fail: 0 skipped: 0

[jefflill]

Test.Neon.Cryptography:

The unit test had a bug where it was comparing local date/time rather than UTC times against certificate expiration range. This failed on the runners because they're configured for UTC and seems like it should fail for machines configured for any other timezone than pacific and it seems like this should also have failed for changes in daylight savings time.

Test.Neon.Service:

Test_QueueService.Success: is failing intermittently on my workstation.

I replaced a hardcoded 5 second delay with NeonHelper.WaitFor() with a 30 second timeout. It's possible that the service wasn't always able to do its thing within the original 5 seconds. I'm not sure if this was actually the problem here, but the WaitFor() call is the best practice for tests like this.

After playing with this, I also saw failures at:

        [Fact]
        public void Success()
        {
            // Restart the service with with valid environment variables,
            // let it run for a vew seconds and verify that it actually
            // sent and received some queue messages.

            var service = CreateQueueService();

            queueServiceFixture.Restart(() => service);
            Assert.True(queueServiceFixture.IsRunning);       <-- FAILED HERE!

            // Give the service some time to process some messages.

            NeonHelper.WaitFor(() => service.SentCount > 0 && service.ReceiveCount > 0, timeout: TimeSpan.FromSeconds(60));

            Assert.True(service.SentCount > 0);
            Assert.True(service.ReceiveCount > 0);

            // Signal the service to stop and verify that it returned [exitcode=0].

            service.Stop();
            Assert.Equal(0, service.ExitCode);
        }

This was ultimately caused by the TestHelper.ResetDocker(null); call in the test constructor which killed the NAT fixture which then causes test cases after the first to fail due to the NAT server not running.

I changed that call to TestHelper.ResetDocker(this.GetType()); which only resets Docker when the first test method in the class is executed.

I also noticed that many of the test classes aren't calling Terminator.ReadyToExit(); after await Terminator.StopEvent.WaitAsync(); returns, which is causing unit tests to stop executing somewhat randomly. I've added this calls and all service tests are running again.