checkout@v3 using node20 instead of node16

jcooke-impinj commented 3 months ago

Running on rhel7 with self-hosted runners and starting today we're running into issues where @v3 is using node20 instead of node16. Happening on runner versions 2.316.1 and 2.317.0. Oddly some of our Repos don't run into this issue when executing checkout@v3 on the same runners.

Run actions/checkout@v3 /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: versionGLIBCXX_3.4.20' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version CXXABI_1.3.9' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: versionGLIBCXX_3.4.21' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version GLIBC_2.28' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: versionGLIBC_2.25' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)

jcooke-impinj commented 3 months ago

Our team found that setting these before starting the runners get's things going again. Still unsure what's causing this.

export ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16
export ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION=node16

adpaco-aws commented 3 months ago

This doesn't seem to be specific to actions/checkout@v3 since we tried to upgrade to actions/checkout@v4 (here) and got the same failure (here).

biswalc commented 3 months ago

I am having the same issue while running below versions, on ubuntu 20.04 VM:

actions/checkout@v3.6.0
actions/checkout@v3
actions/checkout@v4
actions/checkout@v4.1.7

Error:

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

Issue started today, i.e. 3 Jul 10am PDT.

junghyun-apr commented 3 months ago

Hello! Our development team is also experiencing the same issue. We are using AWS instances as self-hosted. We temporarily resolved the issue by simply changing the node16 directory to node20 in the same path. 🥲

AWS AMI :

amzn2-ami-kernel-5.10-hvm-2.0.20240223.0-x86_64-gp2

Git Action Error :

/home/ec2-user/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/actions-runner/externals/node20/bin/node)
/home/ec2-user/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /home/ec2-user/actions-runner/externals/node20/bin/node)

GuyAv46 commented 3 months ago

Following this blog post: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ Adding this to the envs that require node16 fixed the issue

ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true

Flamefire commented 3 months ago

Our team found that setting these before starting the runners get's things going again. Still unsure what's causing this.

export ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16 export ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION=node16

I tried setting this inside a workflow running on GH hosted runners and it didn't work. So it looks like this only works when the runner itself was started with those set, i.e. you cannot set them for a running runner which includes the GH hosted ones.

This doesn't seem to be specific to actions/checkout@v3 since we tried to upgrade to actions/checkout@v4 (here) and got the same failure (here).

v4 always used Node 20

I am having the same issue while running below versions, on ubuntu 20.04 VM:
actions/checkout@v3.6.0
actions/checkout@v3
actions/checkout@v4
actions/checkout@v4.1.7

See above

david-batranu commented 3 months ago

Fixed by renaming the node20 folder and symlinking node16 to node20:

cd $RUNNER_HOME/actions-runner/externals
mv node20 node20.bak
ln -s node16 node20

eshogo commented 3 months ago

I tried modifying /home/runner/actions-runner/.env and env.sh and it works fine.

% cat .env
ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16
ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION=node16

and adding their environment variables into 'varCheckList' array in env.sh. (not needed?)

varCheckList=(
    'LANG'
    'JAVA_HOME'
    'ANT_HOME'
    'M2_HOME'
    'ANDROID_HOME'
    'ANDROID_SDK_ROOT'
    'GRADLE_HOME'
    'NVM_BIN'
    'NVM_PATH'
    'LD_LIBRARY_PATH'
    'PERL5LIB'
    'ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION'
    'ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION'
    )

Then restart runner service

% systemctl restart github-actions-runner.service

david-batranu commented 3 months ago

Looks like this is happening because of: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

n1ngu commented 3 months ago

Following this blog post: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ Adding this to the envs that require node16 fixed the issue
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true

Quoting the post paragraph about that workaround: "This will only work until we upgrade the runner removing Node16 later in the spring"

We are walking a tightrope.

Just wanted to raise some awareness about that. Plan to actually upgrade your github actions in the short term or move away from them. E.g. I think I'll quit my job. :smile:

biswalc commented 3 months ago

I am having the same issue while running below versions, on ubuntu 20.04 VM:
actions/checkout@v3.6.0
actions/checkout@v3
actions/checkout@v4
actions/checkout@v4.1.7
Error:
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
Issue started today, i.e. 3 Jul 10am PDT.

My glibc version:

ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.16) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

So why do I get this error? I am using checkout@v4.1.7. I do not understand what I have to do to upgrade.

ayizhi commented 3 months ago

Here's a critical issue: Firstly, you shouldn't upgrade such fundamental environments arbitrarily. Even if you do upgrade, you must ensure the environment is correct. Upgrading to Node 20 and causing a glib error is absolutely unacceptable, especially in a product from a company like Microsoft.

FelixSchwarz commented 3 months ago

I wish there was an alternative version of the checkout action, maybe written in rust so we could just deploy a static binary. Then we would just compile that against a really old version of glibc. This would also simplify running GitHub Actions with nektos/act which does not map node into the container.

Update: I found checkout-action in GitHub's market place and added support for private repos. Not sure if that is a permanent solution but at least our CI jobs are passing again...

hasufell commented 3 months ago

This means you now can't run any docker image that has a glibc older than 2.27. That includes:

debian:9
fedora:27
ubuntu:18.04

What a huge f*ckup. I'm baffled by the amount of incompetence. What's wrong with providing a statically linked node binary? Do you need help with redistribution github?

imeugeneco commented 3 months ago

After many failed attempts trying to update glibc and related dependencies on amazonlinux2, i've found using export ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true seems to be the best workaround for now.

Updating a core library like glibc is very challenging and the workaround methods many are suggesting here & in similar discussions to use an insecure node version seems like a temporary fix rather than a proper solution.

Please provide a fallback for systems with older glibc versions 😭😭

hasufell commented 3 months ago

@imeugeneco for your own self-hosted runners, you can build a fully statically linked node binary (against musl... e.g. build in an alpine linux container). Then use https://github.com/ChristopherHX/github-act-runner instead of github's agent and make sure your static node binary comes first in PATH.

samkearney commented 3 months ago

I understand the need to drop support for EOL stacks, but I would expect such a change to be done with a version increment rather than under the same version, breaking workflows which haven't made any changes.

eli-schwartz commented 3 months ago

Upgrading the version of node can be done independently of upgrading the version of glibc. Building the node binary redistributables used by actions should ideally be done using either one of:

building it using an old glibc at compile time, so that it works well on both old and new containers
building node statically linked, so that it doesn't care which glibc version is available in the container

The former has very little to do with dropping support for EOL stacks, and the latter has nothing to do with dropping support for EOL stacks.

Therefore IMHO this breaking change was totally unnecessary.

eli-schwartz commented 3 months ago

(Incidentally there's no real point in us trying to upgrade the containers we are testing on, since we have dozens of jobs and exactly one of them fails: the one that tests whether the software still works on really old versions of Ubuntu. We could, of course, stop testing whether the software works on really old versions of Ubuntu, but that would be defeating the purpose of having such a test. It does work quite well except for the Actions Runner parts.)

eli-schwartz commented 3 months ago

The underlying issue here is probably https://github.com/actions/runner/issues/3373

hasufell commented 3 months ago

The reason is this:

github runner images provisioning uses https://github.com/tj/n to install node: https://github.com/actions/runner-images/blob/e81c4806fe17299618a98e49a857bf989f6ac833/images/ubuntu/scripts/build/install-nodejs.sh#L12
tj/n project fetches "official binaries": https://github.com/tj/n/blob/76d6b9870c619d9393ba5c2e3e35fdda171e9773/bin/n#L75
official binaries require at least glibc 2.28: https://nodejs.org/dist/v22.4.0/node-v22.4.0-linux-x64.tar.xz
- unpack and then run objdump -T bin/node | grep --color=never GLIBC_ | sed 's/.*GLIBC_$[.0-9]*$.*/\1/g' | sort -Vu
- the platform/toolchain that is used to build official binaries is documented here: https://github.com/nodejs/node/blob/main/BUILDING.md#official-binary-platforms-and-toolchains
- so for linux it's RHEL8
the reason there is no static binary is likely that it complicates/breaks loading native node addons (because dlopen doesn't work in static binary afaiu)

What is not clear to me is whether native addons are relevant for github actions. If not, then a fully static node build against musl might be possible/feasible.

So I think to have any chance of the situation improving is to ask node upstream to provide a fully static official build of node.

daiyam commented 2 months ago

You can find an working example here: https://github.com/actions/checkout/issues/1590#issuecomment-2207052044

But until when? Since MS plans to:

This will only work until we upgrade the runner removing Node16 later in the spring. (https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/)

So we will have to move to 3rd party actions like the mentioned by @FelixSchwarz:

Update: I found checkout-action in GitHub's market place and added support for private repos. Not sure if that is a permanent solution but at least our CI jobs are passing again...

I hope that MS allows legacy actions to run with node16

evitiello commented 2 months ago

I'm hitting this error using GH actions on Ubuntu-24.04. As this is the most recent Linux image available on GH Actions, is there no Linux that checkout@v3 or @v4 is compatible with now?

clarkmcc commented 2 months ago

Also getting this issue on ubuntu-22.04, 20240630.1.0 with actions/checkout@v3

JesusPaz commented 2 months ago

You can fix this using in your yaml:

env:
  ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

It works with actions/checkout@v3

clarkmcc commented 2 months ago

@JesusPaz okay that works, but according to this and this make it sound like actions/checkout@v4 should use the updated NodeJS version, but I still get this error on v4.

christianbpedersen commented 2 months ago

Neither checkout@v3 or checkout@v4 works with amazon/aws-cli:latest

vmoens commented 2 months ago

This workflow still fails.

Code: https://github.com/pytorch/tensordict/blob/f8bae96c983a0916312f95d9de9fd95548a456db/.github/workflows/nightly_build.yml#L45-L50

edit: solved, just had to move it up as pointed by https://github.com/actions/checkout/issues/1809#issuecomment-2217391660

rolnico commented 2 months ago

This workflow still fails.

Code: https://github.com/pytorch/tensordict/blob/f8bae96c983a0916312f95d9de9fd95548a456db/.github/workflows/nightly_build.yml#L45-L50

Maybe try to move

        env:
          ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true  # https://github.com/actions/checkout/issues/1809

before the jobs (line 29)?

Kobzol commented 2 months ago

I had to add the three ENV vars to the job level, otherwise it didn't work. See https://github.com/It4innovations/hyperqueue/pull/721.

matthinea commented 2 months ago

@Kobzol thank you for that link, your setup is the only thing that worked for me.

daiyam commented 2 months ago

in my workflow: https://github.com/mmomtchev/hadron/blob/main/.github/workflows/os_comp.yml#L32-L33

It's working fine (only ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION needed)

friedrichroell commented 2 months ago

has anyone tried to run the runner inside a docker container (which is easier to run node20 with)?

Kobzol commented 2 months ago

I'm (and I suspect others are) running the checkout action inside a Docker container that is specifically using an old glibc version. The problem is not how to use Node 20, but rather how not to use it.

friedrichroell commented 2 months ago

its the other way around for me, i have an old system that i cant update, but my actions are running directly on the host. could you point me to a guide or something for running the actions on a docker container? 🙏 Cheers!

Kobzol commented 2 months ago

https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container

friedrichroell commented 2 months ago

i meant running the self hosted runner itself in a container

sawwn23 commented 2 months ago

Fixed by renaming the node20 folder and symlinking node16 to node20:
cd $RUNNER_HOME/actions-runner/externals
mv node20 node20.bak
ln -s node16 node20

Thanks for your workaround solution

eli-schwartz commented 2 months ago

Upgrading the version of node can be done independently of upgrading the version of glibc. Building the node binary redistributables used by actions should ideally be done using either one of:

building it using an old glibc at compile time, so that it works well on both old and new containers

building node statically linked, so that it doesn't care which glibc version is available in the container

The former has very little to do with dropping support for EOL stacks, and the latter has nothing to do with dropping support for EOL stacks.

Therefore IMHO this breaking change was totally unnecessary.

Oh also, by the way: it appears this has actually been proposed in https://github.com/actions/runner/pull/3128 back in February.

actions / checkout

checkout@v3 using node20 instead of node16 #1809