CITGM is in deep trouble

[targos]

We got a regression in the latest v22.5.0 that affects at least yarn and npm: https://github.com/nodejs/node/issues/53902

It seems that citgm would have caught it, but for jobs end up green even when they have lots of failures.

From https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/

22:06:57 warn: @nearform/bubbleprof npm-install:| npm error Exit handler never called!
22:06:57 warn: @nearform/bubbleprof npm-install:| npm error This is an error with npm itself. Please report this error at:                                                                                                
22:06:57 warn:                                  | npm error   <https://github.com/npm/cli/issues>                                                                                                                         
22:06:57 warn:                                  | npm error A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/bf0d267e-49e6-467a-bf5b-dcf938d40bcc/home/.npm/_logs/2024-07-16T20_05_57_634Z-debug-0.log
22:06:58 error: failure             | The canary is dead: 
22:07:03 error: @nearform/bubbleprof done| done - the test suite for @nearform/bubbleprof version 7.0.1 failed
22:07:03 (node:783693) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.
22:07:03 (node:783693) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.
22:08:09 warn: @nearform/doctor npm-install:| npm error Exit handler never called!
22:08:09 warn: @nearform/doctor npm-install:| npm error This is an error with npm itself. Please report this error at:                                                                                                
22:08:09 warn:                              | npm error   <https://github.com/npm/cli/issues>                                                                                                                         
22:08:09 warn:                              | npm error A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/50a9288d-b4f4-4ba6-88c9-df6e8ba9bca3/home/.npm/_logs/2024-07-16T20_07_07_903Z-debug-0.log
22:08:10 error: failure             | The canary is dead: 
22:08:16 error: @nearform/doctor done| done - the test suite for @nearform/doctor version 8.1.0 failed

[ljharb]

This is also filed as https://github.com/npm/cli/issues/7657, and https://github.com/nodejs/node/pull/53462 was pointed to me as a possible cause.

[targos]

I think the cause is https://github.com/nodejs/node/pull/53627, but this issue is not about the regression itself. It's about the fact that CITGM somehow reported successful runs instead of failures.

[targos]

@nodejs/build-infra I believe this is an issue with the Jenkins job. If I run node bin/citgm-all.js --includeTags binary-split, I can reproduce the regression and it correctly exits with code 1.

[richardlau]

Possibilities:

• The job runs citgm via eval:

  # Using eval so we can use a complicated CITGM_COMMAND such as 'NODE_OPTIONS=--xzy citgm-all'
  eval $CITGM_COMMAND "--nodedir=$npm_config_nodedir -v $CITGM_LOGLEVEL -x $PWD/report.xml -q $NPM_LOGLEVEL --tmpDir $temp"

• We have Do not fail the build on empty test results turned on

I'll turn off Do not fail the build on empty test results.

[richardlau]

Rerunning https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/ with Do not fail the build on empty test results disabled: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console

[richardlau]

Rerunning https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/ with Do not fail the build on empty test results disabled: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console

With Do not fail the build on empty test results disabled, the CITGM run for 22.5.0 now fails due to having no test results.

[targos]

That's better, thanks ! Now do we need to investigate why it "thinks" there are no results ?

[richardlau]

https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console

13:55:12 Recording test results
13:55:12 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?

which likely means smoker/report.xml wasn't generated:

we use -x to specify outputting to report.xml.

# Using eval so we can use a complicated CITGM_COMMAND such as 'NODE_OPTIONS=--xzy citgm-all'
eval $CITGM_COMMAND "--nodedir=$npm_config_nodedir -v $CITGM_LOGLEVEL -x $PWD/report.xml -q $NPM_LOGLEVEL --tmpDir $temp"

[targos]

Here's the report.xml I get with Node.js 22.5.0 using the following command: node bin/citgm-all.js --includeTags binary-split -x $PWD/report.xml

``` binary-split@1.0.5 test > standard --verbose && node test.js npm error Exit handler never called! npm error This is an error with npm itself. Please report this error at: npm error npm error A complete log of this run can be found in: /var/folders/9n/05y6s4ls78515vmnbd86hktm0000gn/T/9987debd-ab8e-402c-a690-08daa9c9d7de/home/.npm/_logs/2024-07-20T09_32_11_815Z-debug-0.log sh: standard: command not found]]> Error: The canary is dead: ```

And with Node.js 22.5.1:

``` binary-split@1.0.5 test > standard --verbose && node test.js TAP version 13 # ldjson file ok 1 should be equal # custom matcher ok 2 should be equal ok 3 should be equal # long matcher ok 4 should be equal ok 5 should be equal ok 6 should be equal # matcher at index 0 check ok 7 should be equal ok 8 should be equal ok 9 should be equal # chunked input ok 10 should be equal # chunked input with long matcher ok 11 should be equal ok 12 should be equal # lookbehind in multi character matcher ok 13 should be equal ok 14 should be equal ok 15 should be equal 1..15 # tests 15 # pass 15 # ok]]> ```

[RedYetiDev]

FWIW another fast api regression made it through CITGM: https://github.com/nodejs/node/issues/54518