v20.12.2 failed for `x64-debug` which seems to have stopped other targets being built

[SamFarrington]

https://unofficial-builds.nodejs.org/logs/202404101633-v20.12.2/

Failed Build: The attempt to build the x64-debug variant fails.

The script explicitly notes the failure and attempts to move the non-existent build files, resulting in an error because the specified files do not exist.

It seems to me that subsequent targets like loong64, riscv64, x64 with glibc-217 are then not attempted.

Comparing it with the previous version's build.log, it seems like the introduction of x64-debug is new? https://unofficial-builds.nodejs.org/logs/202404082133-v20.12.1/

I'm afraid I don't know enough to suggest a fix to the issue, just to highlight there are some missing builds.

[scosol]

Confirmed that the debug target did not build:

../deps/v8/src/runtime/runtime-strings.cc:532:1: fatal error: error writing to /tmp/ccphwsYn.s: No space left on device

That requires some admin-level setting of the instance size that is beyond the ability of any code-level commits to fix.

I am not sure why the rest of the targets didn't build- but the logic is broken for local builds so that may have something to do with it. (opening a new issue for that)

[rvagg]

"no space left on device" is because they're running in docker using temporary container space which is on the filesystem without enough space for these newer builds, that needs to get fixed for them to proceed

we've always had the issue of the pipeline being unforgiving, but I think we're iterating toward making it more robust so that one failure doesn't mean all failures, I think we need some more changes for that though

this requires manual intervention still, unfortunately, someone else with access could queue them up, or it'll just have to wait till I have time and access to do it

[scosol]

@rvagg who else has the needed access?

[zacthompson]

All the subsequent builds are missed because the mv command is failing due to having no built files, and the script is being run with bash -xe. -e causes exit on error, but that's probably the right thing for build scripts! And even if -e were turned off, there are commands inside the for loop which would exit the whole script anyway as soon as the first failure happened.

I suggest adding a check just before the distribution actions. Something like:

if ! ls  "${stagingoutdir}"/node-v* >/dev/null; then continue; fi

[rvagg]

@zacthompson you're welcome to do a PR for this, it was on the proposed list of things to do, it just hasn't been got to

wrt the out of space errors, I've just freed up a ton of journal space in /var/log and moved the nginx logs to a new volume. Hopefully the build will run now, it's in the queue for 20.12.2 so we'll see later today but you can watch the logs if you like