Investigate flaky test sequential/test-net-connect-econnrefused

[mhdawson]

Version

amster

Platform

s390x (but unlikely to be platform specific)

Subsystem

net

What steps will reproduce the bug?

run test sequential/test-net-econnect-econnrefused - https://ci.nodejs.org/job/node-test-commit-linuxone/nodes=rhel7-s390x/29181/testReport/junit/(root)/test/sequential_test_net_connect_econnrefused/

How often does it reproduce? Is there a required condition?

Intermittent, seen in 1 job and does not seem to be any open issue tracking failures

What is the expected behavior?

Test passes

What do you see instead?

Test fails

Additional information

No response

[Trott]

I think you're using the Bug Report template but there's a Flaky Test template that asks for more useful information on these. See fourth item from the top below:

[mhdawson]

@Trott thanks, I'd missed that.

[richardlau]

Also it looks like the reliability reports in https://github.com/nodejs/reliability/ started working again around the time of the main branch rename so you can look there for how often a failure occurs and if it has been seen elsewhere.

[lpinca]

It also failed today on the same host (s390x)

https://ci.nodejs.org/job/node-test-commit-linuxone/29684/nodes=rhel7-s390x/testReport/junit/(root)/test/sequential_test_net_connect_econnrefused/

12:11:43 not ok 3502 sequential/test-net-connect-econnrefused
12:13:43   ---
12:13:43   duration_ms: 120.24
12:13:43   severity: fail
12:13:43   exitcode: -15
12:13:43   stack: |-
12:13:43     timeout
12:13:43     pending 50 rounds 1
12:13:43     pending 49 rounds 1
12:13:43     pending 48 rounds 1
12:13:43     pending 47 rounds 1
12:13:43     pending 46 rounds 1
12:13:43     pending 45 rounds 1
12:13:43     pending 44 rounds 1
12:13:43     pending 43 rounds 1
12:13:43     pending 42 rounds 1
12:13:43     pending 41 rounds 1
12:13:43     pending 40 rounds 1
12:13:43     pending 39 rounds 1
12:13:43     pending 38 rounds 1
12:13:43     pending 37 rounds 1
12:13:43     pending 36 rounds 1
12:13:43     pending 35 rounds 1
12:13:43     pending 34 rounds 1
12:13:43     pending 33 rounds 1
12:13:43     pending 32 rounds 1
12:13:43     pending 31 rounds 1
12:13:43     pending 30 rounds 1
12:13:43     pending 29 rounds 1
12:13:43     pending 28 rounds 1
12:13:43     pending 27 rounds 1
12:13:43     pending 26 rounds 1
12:13:43     pending 25 rounds 1
12:13:43     pending 24 rounds 1
12:13:43     pending 23 rounds 1
12:13:43     pending 22 rounds 1
12:13:43     pending 21 rounds 1
12:13:43     pending 20 rounds 1
12:13:43     pending 19 rounds 1
12:13:43     pending 18 rounds 1
12:13:43     pending 17 rounds 1
12:13:43     pending 16 rounds 1
12:13:43     pending 15 rounds 1
12:13:43     pending 14 rounds 1
12:13:43     pending 13 rounds 1
12:13:43     pending 12 rounds 1
12:13:43     pending 11 rounds 1
12:13:43     pending 10 rounds 1
12:13:43     pending 9 rounds 1
12:13:43     pending 8 rounds 1
12:13:43     pending 7 rounds 1
12:13:43     pending 6 rounds 1
12:13:43     pending 5 rounds 1
12:13:43     pending 4 rounds 1
12:13:43     pending 3 rounds 1
12:13:43     pending 2 rounds 1
12:13:43     pending 1 rounds 1
12:13:43     pending 50 rounds 2
12:13:43     pending 49 rounds 2
12:13:43     pending 48 rounds 2
12:13:43     pending 47 rounds 2
12:13:43     pending 46 rounds 2
12:13:43     pending 45 rounds 2
12:13:43     pending 44 rounds 2
12:13:43     pending 43 rounds 2
12:13:43     pending 42 rounds 2
12:13:43     pending 41 rounds 2
12:13:43     pending 40 rounds 2
12:13:43     pending 39 rounds 2
12:13:43     pending 38 rounds 2
12:13:43     pending 37 rounds 2
12:13:43     pending 36 rounds 2
12:13:43     pending 35 rounds 2
12:13:43     pending 34 rounds 2
12:13:43     pending 33 rounds 2
12:13:43     pending 32 rounds 2
12:13:43     pending 31 rounds 2
12:13:43     pending 30 rounds 2
12:13:43     pending 29 rounds 2
12:13:43     pending 28 rounds 2
12:13:43     pending 27 rounds 2
12:13:43     pending 26 rounds 2
12:13:43     pending 25 rounds 2
12:13:43     pending 24 rounds 2
12:13:43     pending 23 rounds 2
12:13:43     pending 22 rounds 2
12:13:43     pending 21 rounds 2
12:13:43     pending 20 rounds 2
12:13:43     pending 19 rounds 2
12:13:43     pending 18 rounds 2
12:13:43     pending 17 rounds 2
12:13:43     pending 16 rounds 2
12:13:43     pending 15 rounds 2
12:13:43     pending 14 rounds 2
12:13:43     pending 13 rounds 2
12:13:43     pending 12 rounds 2
12:13:43     pending 11 rounds 2
12:13:43     pending 10 rounds 2
12:13:43     pending 9 rounds 2
12:13:43     pending 8 rounds 2
12:13:43     pending 7 rounds 2
12:13:43     pending 6 rounds 2
12:13:43     pending 5 rounds 2
12:13:43     pending 4 rounds 2
12:13:43     pending 3 rounds 2
12:13:43     pending 2 rounds 2
12:13:43     pending 1 rounds 2
12:13:43     pending 50 rounds 3
12:13:43     pending 49 rounds 3
12:13:43     pending 48 rounds 3
12:13:43     pending 47 rounds 3
12:13:43     pending 46 rounds 3
12:13:43     pending 45 rounds 3
12:13:43     pending 44 rounds 3
12:13:43     pending 43 rounds 3
12:13:43     pending 42 rounds 3
12:13:43     pending 41 rounds 3
12:13:43     pending 40 rounds 3
12:13:43     pending 39 rounds 3
12:13:43     pending 38 rounds 3
12:13:43     pending 37 rounds 3
12:13:43     pending 36 rounds 3
12:13:43     pending 35 rounds 3
12:13:43     pending 34 rounds 3
12:13:43     pending 33 rounds 3
12:13:43     pending 32 rounds 3
12:13:43     pending 31 rounds 3
12:13:43     pending 30 rounds 3
12:13:43     pending 29 rounds 3
12:13:43     pending 28 rounds 3
12:13:43     pending 27 rounds 3
12:13:43     pending 26 rounds 3
12:13:43     pending 25 rounds 3
12:13:43     pending 24 rounds 3
12:13:43     pending 23 rounds 3
12:13:43     pending 22 rounds 3
12:13:43     pending 21 rounds 3
12:13:43     pending 20 rounds 3
12:13:43     pending 19 rounds 3
12:13:43     pending 18 rounds 3
12:13:43     pending 17 rounds 3
12:13:43     pending 16 rounds 3
12:13:43     pending 15 rounds 3
12:13:43     pending 14 rounds 3
12:13:43     pending 13 rounds 3
12:13:43     pending 12 rounds 3
12:13:43     pending 11 rounds 3
12:13:43     pending 10 rounds 3
12:13:43     pending 9 rounds 3
12:13:43     pending 8 rounds 3
12:13:43     pending 7 rounds 3
12:13:43     pending 6 rounds 3
12:13:43     pending 5 rounds 3
12:13:43     pending 4 rounds 3
12:13:43     pending 3 rounds 3
12:13:43     pending 2 rounds 3

[lpinca]

By the way, for reasons unknown to me, it seems to be just slow. The test is made of 5 rounds and it tests always the same thing in all rounds. In this specific case, it completes almost 3 rounds and then it times out.

I wonder if it is really necessary to run 5 rounds.

[lpinca]

The test was moved from pummel to sequential in https://github.com/nodejs/node/commit/df3465a9e7c6d6bb9554226b7648cf4d041d7d69.

[Trott]

I'm OK with cutting it down to three iterations and/or moving it back to pummel.

[lpinca]

It would be nice to understand what actually happens on the host machine. Why this only happens on s390x? It seems that all callbacks are called within the same second and then the test hangs.

[Trott]

It would be nice to understand what actually happens on the host machine. Why this only happens on s390x? It seems that all callbacks are called within the same second and then the test hangs.

Hmmm...yeah, in that case, I'm a little less excited about reducing iterations and/or moving to pummel. :-D

[mhdawson]

Failed again today https://ci.nodejs.org/job/node-test-commit-linuxone/30016/

[Trott]

Failed again today https://ci.nodejs.org/job/node-test-commit-linuxone/30016/

The CI links expire so it's good to include the relevant output.

12:20:42 not ok 3519 sequential/test-net-connect-econnrefused
12:22:42   ---
12:22:42   duration_ms: 120.99
12:22:42   severity: fail
12:22:42   exitcode: -15
12:22:42   stack: |-
12:22:42     timeout
12:22:42     pending 50 rounds 1
12:22:42     pending 49 rounds 1
12:22:42     pending 48 rounds 1
12:22:42     pending 47 rounds 1
12:22:42     pending 46 rounds 1
12:22:42     pending 45 rounds 1
12:22:42     pending 44 rounds 1
12:22:42     pending 43 rounds 1
12:22:42     pending 42 rounds 1
12:22:42     pending 41 rounds 1
12:22:42     pending 40 rounds 1
12:22:42     pending 39 rounds 1
12:22:42     pending 38 rounds 1
12:22:42     pending 37 rounds 1
12:22:42     pending 36 rounds 1
12:22:42     pending 35 rounds 1
12:22:42     pending 34 rounds 1
12:22:42     pending 33 rounds 1
12:22:42     pending 32 rounds 1
12:22:42     pending 31 rounds 1
12:22:42     pending 30 rounds 1
12:22:42     pending 29 rounds 1
12:22:42     pending 28 rounds 1
12:22:42     pending 27 rounds 1
12:22:42     pending 26 rounds 1
12:22:42     pending 25 rounds 1
12:22:42     pending 24 rounds 1
12:22:42     pending 23 rounds 1
12:22:42     pending 22 rounds 1
12:22:42     pending 21 rounds 1
12:22:42     pending 20 rounds 1
12:22:42     pending 19 rounds 1
12:22:42     pending 18 rounds 1
12:22:42     pending 17 rounds 1
12:22:42     pending 16 rounds 1
12:22:42     pending 15 rounds 1
12:22:42     pending 14 rounds 1
12:22:42     pending 13 rounds 1
12:22:42     pending 12 rounds 1
12:22:42     pending 11 rounds 1
12:22:42     pending 10 rounds 1
12:22:42     pending 9 rounds 1
12:22:42     pending 8 rounds 1
12:22:42     pending 7 rounds 1
12:22:42     pending 6 rounds 1
12:22:42     pending 5 rounds 1
12:22:42     pending 4 rounds 1
12:22:42     pending 3 rounds 1
12:22:42     pending 2 rounds 1
12:22:42   ...