sequential.test-watch-mode-inspect seems flaky

[mhdawson]

Test

sequential.test-watch-mode-inspect

Platform

Linux ARMv7

Console output

Error Message
fail (-15)
Stacktrace
---
duration_ms: 360029.938
exitcode: -15
severity: fail
...

Build links

• https://ci.nodejs.org/job/node-test-binary-armv7l/7719/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/
• https://ci.nodejs.org/job/node-test-binary-armv7l/7715/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/

Additional information

No response

[mhdawson]

Failed on same machine, two builds in a row test-equinix-ubuntu2004_container-armv7l-1

[mhdawson]

Passed when it ran on test-equinix-ubuntu2204-x64-2 so could be specific to that machine.

Need to track to see if it fails across the machines or not.

[mhdawson]

and again - https://ci.nodejs.org/job/node-test-binary-armv7l/7720/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console

[mhdawson]

Interestingly I see this in the output

not ok 3844 sequential/test-watch-mode-inspect # TODO : Fix flaky test

which seems to say it's a known issue even though I could not find a tracking issue.

[richardlau]

FWIW the test is already marked flaky https://github.com/nodejs/node/blob/1b96975f273ca92e6ded4d7f4280158f441c8da6/test/sequential/sequential.status#L10-L12

https://ci.nodejs.org/job/node-test-binary-armv7l/7720/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console has failed because of leftover Node.js processes (which are probably related to the test being timed out):

17:02:09 All tests passed.
17:02:09 ps awwx | grep Release/node | grep -v grep | cat
17:02:09 2212632 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 --watch /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
17:02:09 2212638 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
17:02:09 make: *** [Makefile:538: test-ci-js] Error 1

[anonrig]

It failed on again on ubuntu 22.04 https://ci.nodejs.org/job/node-test-commit-osx-arm/13900/

[anonrig]

Seen on node-test-commit-arm-fanned https://ci.nodejs.org/job/node-test-binary-armv7l/8168/

[mhdawson]

I don't see if failing in https://ci.nodejs.org/job/node-test-commit-osx-arm/13900/, that seemed to be a different test test-watch-mode without the inspect part.

[mhdawson]

again - https://ci.nodejs.org/job/node-test-binary-armv7l/8829/

[mhdawson]

again - https://ci.nodejs.org/job/node-test-binary-armv7l/9064/

[mhdawson]

Not sure why I keep seeing these even though the test is marked as flaky. Looked at the CI jobs and I don't see anything obviously wrong.

@richardlau do you know of any reason off the top of your head a job might not be treating flaky tests as ok?

[richardlau]

@mhdawson https://ci.nodejs.org/job/node-test-binary-armv7l/9064/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console failed because:

18:27:41 All tests passed.
18:27:41 ps awwx | grep Release/node | grep -v grep | cat
18:27:41  601756 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 --watch /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
18:27:41  601762 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
18:27:41 make: *** [Makefile:538: test-ci-js] Error 1

"All tests passed" means that the failure was treated as flaky. The problem is that it left behind child processes and the build will fail if any node process are still running after the test runner exits: https://github.com/nodejs/node/blob/342ddb0e3d232e1262391580cf77d9b1341e8f59/Makefile#L554-L559

[mhdawson]

@richardlau, if I get what you are saying is that even though the job out reports this test, that was not the problem, instead it was because child processes were left behind that may/likely have nothing to do with that flaky test right?

[richardlau]

@mhdawson Yes, although I suspect the leftover processes are from the flaky test which I believe has been timed out.

[mhdawson]

@richardlau ok, if we believe they are related to the test then SKIP might be more appropriate right? If so I'll submit a PR to do that.

[mhdawson]

Failed again - https://ci.nodejs.org/job/node-test-binary-armv7l/9322/

[mhdawson]

Faillures on ubuntu shared libs - https://ci.nodejs.org/job/node-test-commit-linux-containered/40987/nodes=ubuntu2204_sharedlibs_shared_x64/testReport/junit/(root)/sequential/test_watch_mode_inspect_/

[mhdawson]

Seems to fail on ubuntu as well - https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu2204_sharedlibs_shared_x64/41479/

[jakecastelli]

Again - https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos20-64/55713/

logs

``` 22:09:53 not ok 4228 sequential/test-watch-mode-inspect # TODO : Fix flaky test 22:09:53 --- 22:09:53 duration_ms: 300129.29800 22:09:53 severity: fail 22:09:53 exitcode: -15 22:09:53 stack: |- 22:09:53 timeout 22:09:53 [test] Connecting to a child Node process 22:09:53 [test] Testing /json/list 22:09:53 TAP version 13 22:09:53 [err] Debugger listening on ws://127.0.0.1:61622/bec85c05-bcf7-4407-86de-720d30c10727 22:09:53 [err] For help, see: https://nodejs.org/en/docs/inspector 22:09:53 [err] 22:09:53 [err] Debugger attached. 22:09:53 [err] 22:09:53 [test] Connecting to a child Node process 22:09:53 [test] Testing /json/list 22:09:53 [err] Debugger ending on ws://127.0.0.1:61622/bec85c05-bcf7-4407-86de-720d30c10727 22:09:53 [err] For help, see: https://nodejs.org/en/docs/inspector 22:09:53 [err] 22:09:53 [out] safe to debug now 22:09:53 [out] 22:09:53 ... ```

This test seems to fail quite often lately

[targos]

https://ci.nodejs.org/job/node-test-commit-smartos/55913/nodes=smartos20-64/consoleFull

[aduh95]

Again https://ci.nodejs.org/job/node-test-commit-linux-containered/46199/