Downloading node.js can fail resulting in build failures

[elasticmachine]

:broken_heart: Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

#### Build stats * Start Time: 2023-06-26T16:28:34.999+0000 * Duration: 67 min 55 sec #### Test stats :test_tube: | Test | Results | | ------------ | :-----------------------------: | | Failed | 0 | | Passed | 29806 | | Skipped | 2357 | | Total | 32163 |

Steps errors

Expand to view the steps failures

##### `Installing Node.js 14.17.5`

• Took 0 min 6 sec . View more details here
• Description: set -e export NVM_DIR="${HOME}/.nvm" [ -s "${NVM_DIR}/nvm.sh" ] && . "${NVM_DIR}/nvm.sh" nvm install 14.17.5 nvm version | head -n1 > "/var/lib/jenkins/workspace/8.9-6-adc984c3-b771-48f5-8d84-c3e84c5dabe4/src/github.com/elastic/beats@tmp/.nvm-node-version"

##### `Installing Node.js 14.17.5`

• Took 0 min 8 sec . View more details here
• Description: set -e export NVM_DIR="${HOME}/.nvm" [ -s "${NVM_DIR}/nvm.sh" ] && . "${NVM_DIR}/nvm.sh" nvm install 14.17.5 nvm version | head -n1 > "/var/lib/jenkins/workspace/8.9-6-adc984c3-b771-48f5-8d84-c3e84c5dabe4/src/github.com/elastic/beats@tmp/.nvm-node-version"

##### `Installing Node.js 14.17.5`

• Took 0 min 27 sec . View more details here
• Description: set -e export NVM_DIR="${HOME}/.nvm" [ -s "${NVM_DIR}/nvm.sh" ] && . "${NVM_DIR}/nvm.sh" nvm install 14.17.5 nvm version | head -n1 > "/var/lib/jenkins/workspace/8.9-6-adc984c3-b771-48f5-8d84-c3e84c5dabe4/src/github.com/elastic/beats@tmp/.nvm-node-version"

##### `Installing Node.js 14.17.5`

• Took 0 min 6 sec . View more details here
• Description: set -e export NVM_DIR="${HOME}/.nvm" [ -s "${NVM_DIR}/nvm.sh" ] && . "${NVM_DIR}/nvm.sh" nvm install 14.17.5 nvm version | head -n1 > "/var/lib/jenkins/workspace/8.9-6-c39a3bb1-25e9-432a-ba8f-6d52d4dcef06/src/github.com/elastic/beats@tmp/.nvm-node-version"

##### `Installing Node.js 14.17.5`

• Took 0 min 17 sec . View more details here
• Description: set -e export NVM_DIR="${HOME}/.nvm" [ -s "${NVM_DIR}/nvm.sh" ] && . "${NVM_DIR}/nvm.sh" nvm install 14.17.5 nvm version | head -n1 > "/var/lib/jenkins/workspace/8.9-6-c39a3bb1-25e9-432a-ba8f-6d52d4dcef06/src/github.com/elastic/beats@tmp/.nvm-node-version"

##### `Installing Node.js 14.17.5`

• Took 0 min 4 sec . View more details here
• Description: set -e export NVM_DIR="${HOME}/.nvm" [ -s "${NVM_DIR}/nvm.sh" ] && . "${NVM_DIR}/nvm.sh" nvm install 14.17.5 nvm version | head -n1 > "/var/lib/jenkins/workspace/8.9-6-c39a3bb1-25e9-432a-ba8f-6d52d4dcef06/src/github.com/elastic/beats@tmp/.nvm-node-version"

##### `Error signal`

• Took 0 min 0 sec . View more details here
• Description: Error "hudson.AbortException: script returned exit code 1"

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

@v1v did we change something lately here?

[cmacknz]

[2023-06-26T16:54:11.228Z] + CURL_COMPRESSED_FLAG=--compressed

[2023-06-26T16:54:11.228Z] + curl --fail --compressed -q -L -C - --progress-bar https://nodejs.org/dist/v14.17.5/node-v14.17.5-linux-x64.tar.xz -o /var/lib/jenkins/workspace/8.9-6-c39a3bb1-25e9-432a-ba8f-6d52d4dcef06/.nvm/.cache/bin/node-v14.17.5-linux-x64/node-v14.17.5-linux-x64.tar.xz

[2023-06-26T16:54:14.943Z] #=#=#                                                                         
##O#-#                                                                        
##O=#  #                                                                      
#=#=-#  #                                                                     
-#O#- #   #                                                                   
curl: (22) The requested URL returned error: 500

This is a server error downloading node.js, probably it was transient. We could solve this by mirroring the node.js packages ourselves, or perhaps simpler adding some retries during the download attempt.

[jlind23]

@cmacknz shall we ask the uptime team to work on this?

[cmacknz]

Yes, node.js is a dependency of synthetics.

[cmacknz]

This also appears to affect the unified release process, here's a sample of a recent failure.

10:12:29 #21 552.8 curl: (18) transfer closed with 15708476 bytes remaining to read
10:12:29 #21 552.8 xz: (stdin): Unexpected end of input
10:12:29 #21 552.8 tar: Unexpected EOF in archive
10:12:29 #21 552.8 tar: Unexpected EOF in archive
10:12:29 #21 552.8 tar: Error is not recoverable: exiting now
10:12:29 #21 ERROR: process "/bin/sh -c cd /usr/share/elastic-agent/.node   && NODE_DOWNLOAD_URL=\"\"   && case \"$(arch)\" in         arm64)            NODE_DOWNLOAD_URL=[https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz](https://nodejs.org/dist/v$%7BNODE_VERSION%7D/node-v$%7BNODE_VERSION%7D-linux-x64.tar.xz)            ;;        x86_64)            NODE_DOWNLOAD_URL=[https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz](https://nodejs.org/dist/v$%7BNODE_VERSION%7D/node-v$%7BNODE_VERSION%7D-linux-x64.tar.xz)            ;;        aarch64)            NODE_DOWNLOAD_URL=[https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-arm64.tar.xz](https://nodejs.org/dist/v$%7BNODE_VERSION%7D/node-v$%7BNODE_VERSION%7D-linux-arm64.tar.xz)            ;;        *)            echo >&2 ; echo >&2 \"Unsupported architecture \\$(arch)\" ; echo >&2 ; exit 1 ;            ;;      esac   && mkdir -p node   && curl ${NODE_DOWNLOAD_URL} | tar -xJ --strip 1 -C node   && chmod ug+rwX -R $NODE_PATH" did not complete successfully: exit code: 2

[jlind23]

@paulb-elastic Can we please have someone from your team taking a look at this? This is impacting Beats-CI and the overall unified release process as stated by @cmacknz

[andrewvc]

@jlind23 I think there are multiple conflated failures here, one of which may be us, and the other almost certainly not. The common denominator is that the service hosting node JS is unreliable apparently.

It appears that something is using NVM.nvmrc, which is not the synthetics team. It's also using an old v14 version of node.

We do use node but only in building the docker containers in the dockerfile.

Either way I suspect the fix is that we host node binaries ourself or use some sort of more reliable caching proxy.

I'm going to investigate further, but I wanted to provide a partial update now.

[andrewvc]

I'll have to partially retract the bit about NVM, it is only used in our tests, but via a helper used at various other places within Elastic for builds (and maintained by the robots it looks like). https://github.com/elastic/apm-pipeline-library/blob/a1c0d088ebeaac0741592d77e2fc60157557f854/vars/withNodeJSEnvUnix.groovy#L21 .

[cmacknz]

This is still happening and has affected our BC builds for 8.11

[v1v]

This is still happening and has affected our BC builds for 8.11

I cannot see any failures in the packaging for 8.11 -> https://beats-ci.elastic.co/job/Beats/job/packaging/job/8.11/ similarly with the CI pipeline -> https://beats-ci.elastic.co/job/Beats/job/beats/job/8.11/

are BC builds for 8.11 running outside of the DRA process?

[cmacknz]

Looking again this was actually the node.js version bundled into the elastic-agent-complete Docker image for Heartbeat/Synthetics. So same problem but in a different part of the build than what this issue originally reported.