search

actions / runner-images

GitHub Actions runner images
MIT License
10.17k stars 3.06k forks source link

Update/Add nvm #4

Closed scottohara closed 4 years ago

scottohara commented 5 years ago

Tool information

Virtual environments affected

(Already installed one macOS virtual env)

Can this tool be installed during the build?

Yes, but not easily. See https://github.community/t5/GitHub-Actions/How-to-share-shell-profile-between-steps-or-how-to-use-nvm-rvm/td-p/33185

Are you willing to submit a PR?

kaylangan commented 5 years ago

👋 Thanks for the suggestion! Is there something that nvm does that isn't possible with the actions/setup-node action?

scottohara commented 5 years ago

setup-node requires an explicit version of nodejs needed for the job to be specified in the with: property.

Many nodejs projects already have a file in their repo that specifies a desired node version (e.g..nvmrc), so having to repeat the desired node version in .github/workflows/*.yml would seem unnecessary duplication.

If setup-node could detect and use the version specified in an existing .nvmrc then arguably installing nvm would not be necessary.

Additional points:

  1. The macOS virtual env already has nvm preinstalled (but the Linux ones don’t)
  2. CI services such as TravisCI have nvm preinstalled for this very purpose
github-actions[bot] commented 4 years ago

This issue has not had any activity for 45 days and will be closed in 45 days if there continues to be no activity.

thejoebourneidentity commented 4 years ago

I'm working with the setup-node owners to determine if https://github.com/actions/setup-node/issues/32 is something they plan on fixing. Leaving this issue open for now to track.

thejoebourneidentity commented 4 years ago

Thanks for the suggestion @scottohara, we're going to add nvm as requested.

@maxim-lobanov , let's do it, let me know if you need any additional assistance.

maxim-lobanov commented 4 years ago

@scottohara , before we start adding it to the Linux, could you please confirm that it works as expected on macOS images? I am asking because as I know, nvm doesn't work properly on macOS due to NVM initialization way and you have to write initialization trick in each pipeline:

# Delete prefix from system node config
npm config delete prefix
# Need for using `nvm` inside script
. "$HOME/.nvm/nvm.sh"
nvm use $version
nvm alias default $(nvm current)

echo "Node version: $(node -v)"
echo "npm version: $(npm -v)"

# Update `PATH` for the next steps (AzDO specific)
echo '##vso[task.setvariable variable=PATH;]'$PATH
# Update `PATH` for the next steps (GitHub specific)
echo "::set-env name=PATH::$PATH"

After quick look, the same issue will happen on Ubuntu. This issue comes from the way how to NVM works, it initializes per-session only and we have to use tricks to pass it to the next steps. So I am very careful about the idea to pre-install NVM to Ubuntu images.

@thejoebourneidentity , . nvmrc and .node_version are very popular approaches to specify node. Is it something that we would like to add to setup-node task? We can improve setup-node task to look for these files and use them if they exist

scottohara commented 4 years ago

@maxim-lobanov the preinstalled nvm on the macOS image certainly works.

Yes, you need to delete the npm prefix, and to have it pass through to each step you need to specify a login shell (shell: bash --login {0}).

Other than that, it works fine.

runs-on: macOS-latest 
steps:
  - name: Checkout repository
    uses: actions/checkout@v1
    
  - name: Delete npm prefix
    run: npm config delete prefix
    
  - name: Install dependencies
    shell: bash --login {0}
    run: nvm install && nvm use && npm ci
   
  - name: Lint and test
    shell: bash --login {0}
    run: nvm use && npm test
Kerizer commented 4 years ago

@maxim-lobanov the preinstalled nvm on the macOS image certainly works.

Yes, you need to delete the npm prefix, and to have it pass through to each step you need to specify a login shell (shell: bash --login {0}).

Other than that, it works fine.

runs-on: macOS-latest 
steps:
  - name: Checkout repository
    uses: actions/checkout@v1
    
  - name: Delete npm prefix
    run: npm config delete prefix
    
  - name: Install dependencies
    shell: bash --login {0}
    run: nvm install && nvm use && npm ci
   
  - name: Lint and test
    shell: bash --login {0}
    run: nvm use && npm test

This stopped working few hours ago. Now it uses some generic shell image

maxim-lobanov commented 4 years ago

@Kerizer , no changes from our side for macOS images for the last 2-3 days...

maxim-lobanov commented 4 years ago

@Kerizer , your screenshot is from ubuntu image. NVM is not installed there. macOS still works as expected.

Kerizer commented 4 years ago

@Kerizer , your screenshot is from ubuntu image. NVM is not installed there. macOS still works as expected.

Still, we used nvm on Ubuntu as following:

      - name: Install nvm
        shell: bash -l {0}
        # In case if we need to use node or nvm - we should only use it from `bash -l {0}` shell.
        run: |
          cd ..
          echo $HOME
          curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
          export NVM_DIR="$HOME/.nvm"
          echo "source $NVM_DIR/nvm.sh --install" >> "$HOME/.bash_profile"
          cd -
          source $HOME/.bash_profile
          nvm install

And it worked for us for month. Starting from today it fails and it looks like macOS and Ubuntu now work in a different ways and are inconsistent.

maxim-lobanov commented 4 years ago

Thank you for the provided example, We will take care about installation NVM on Ubuntu. As for the issue above, no changes related to shell from our side, but we will take a look

dentuzhik commented 4 years ago

@maxim-lobanov we (me and @Kerizer) have been rather internally confused a bit, are you a Github staff memeber or just a core contributor/maintainer of this package?

Our workflows are a bit clusterfucked now, since we relied quite heavily on NVM setup this way (you know, because no other way) and now it would be nice if we could at least point them to some commit where it's not yet broken 🙇

Thanks for quick responses anyway!

maxim-lobanov commented 4 years ago

Maintainer of virtual-environments repository.

I have found the root cause of your issue. And unfortunately, it caused by the latest Ubuntu image update. In scope of this update, we have added XDG_CONFIG_HOME variable to image. And it looks like NPM installation depends on this variable. So as a result, npm started to be installed to “$HOME/.config/nvm” instead of “$HOME/.nvm”.

Could you please modify script a bit to fix your build? (the single difference is that we define NVM_DIR before NVM installation and invoke mkdir

cd ..
echo $HOME
export NVM_DIR="$HOME/.nvm"
mkdir $NVM_DIR
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
echo "source $NVM_DIR/nvm.sh --install" >> "$HOME/.bash_profile"
cd -
source $HOME/.bash_profile
nvm install

This script will unblock your build.

Sorry for inconvenience and hope adding NVM to the image will improve your experience a bit :)

dentuzhik commented 4 years ago

it caused by the latest Ubuntu image update

Just curious, was it a silent automatic update, or does this package fix the version some place and some commit updated it? I am not sure how to parse this json file in my head, so if you feel like sharing some insides, much appreciated 🙌 https://github.com/actions/virtual-environments/blob/master/images/linux/ubuntu1804.json

If the updates of images are not frozen, would it make sense to freeze them maybe to at least release them alongside updates of action itself?

I remember already voicing a concern somewhere how all updates to all actions by design are released to upstreams like actions/checkout@v2 what can cause overnight failures like this one.

\cc @bryanmacfarlane

maxim-lobanov commented 4 years ago

Unfortunately, it is not possible for images. All hosted VMs have the same image version. That json file is just packer template to generate image. We generate image based on this template on weekly basis and update all VMs automatically from our side. Image updates can be tracked by updating README file or Releases in this repository

dentuzhik commented 4 years ago

So was it a broken release then or a silent update with a nasty side effect? If it is a release, can you point to the one which broke it, if you have any idea? 🙇

miketimofeev commented 4 years ago

@dentuzhik Let me explain it a bit. We have a user request to add this variable - https://github.com/actions/virtual-environments/issues/491 but unfortunately, that can be done only during the VM initialization process in the scope of hosted pool creation, so that we've made changes in the internal scripts to add the variable that's why you couldn't find it at https://github.com/actions/virtual-environments/blob/master/images/linux/ubuntu1804.json. We're going to roll back the changes asap, sorry for the inconvenience.

dentuzhik commented 4 years ago

Thanks for the explanation, much appreciated 🙌

We are not against a breaking migration, if it's really necessary, since we sort of relied on not that much documented feature, but I guess we could make somewhat a "major" release of images where it would be possible to upgrade?

Unfortunately, it is not possible for images. All hosted VMs have the same image version.

I know it was mentioned before, but it would rather make sense to have a way to update images in the backwards incompatible way with providing a path to migrate to them, no?

I mean if we look at version of alpine-linux or ubuntu images in Docker, there's a way to freeze previous version and release a new one which can be broken, without affecting existing CI machines around the globe: https://hub.docker.com/_/ubuntu/?tab=tags https://hub.docker.com/_/alpine?tab=tags

If we look at example from CircleCI, they have similar stuff: https://circleci.com/docs/2.0/executor-intro/#machine image

Now after writing all this, I think the problem here which has surfaced, that versioning of images is too tightly coupled to versioning of distributions, which has backfired when you started to need to have two different versions of the same distro for fixes like https://github.com/actions/virtual-environments/issues/491

I understand that this is oftopic of this issues, and I can help creating a new one, if you guys agree this is a problem

maxim-lobanov commented 4 years ago

@dentuzhik, Yes, please create the issue for your proposal. As I know, we don't have any plans in this direction at the moment but it would be great to have the issue where we can track popularity of this proposal.

bryanmacfarlane commented 4 years ago

Also note that there is currently a way to freeze the toolsets and image. Self-hosted runners (on cloud VMs as well). Yes, much less convenient but at the other end of the spectrum on the control vs convenience spectrum. Not saying that having a way to pin or do n-1 on images, isn't very useful. Just highlighting as an option.

Off topic but actions also have a way to pin to a specific version (ref or sha).

So between self-hosted and what you choose to pin to, it is possible to lock down. One thing you can't lockdown (until actions is in GHES) is the service itself and that moves forward.

dentuzhik commented 4 years ago

Yes, much less convenient but at the other end of the spectrum on the control vs convenience spectrum.

This is indeed a valid point.

However I would assume that the default option, which is provided to consumers of this package, should be safe enough to avoid accidental breaking.

Therefore there should likely be both a way to push to upstream and let others update (docker actions/checkout@v2:latest tag / master branch) and provide a way to reference to specific LTS version (actions/checkout@v2 in v2 branch) AND have a way to create backwards incompatible changes (actions/checkout@v2:20200319 or actions/checkout@v2.3).

Versioning in a way that everyone is happy is hard. I have put into my lists to come up with a proposal, at least for this specific case for virtual-environments, let's hope that I get there some time soon, we are quite under crossfires because of COVID-19

dmitry-shibanov commented 4 years ago

Hello, pull request was merged and changes would be added in the next images rollout.

dmitry-shibanov commented 4 years ago

Hello, changes were added. If you have any concerns please feel free to reopen the issue.

sakulstra commented 4 years ago

shouldn't sth like this:

name: nvm
on:
  push:
jobs:
  nvm:
    runs-on: ubuntu-latest
    steps:
      - name: try out nvm
        run: nvm install

work now? Getting:

Run nvm install
/home/runner/work/_temp/6f648dd8-690e-4082-9988-421256974eb6.sh: line 1: nvm: command not found
##[error]Process completed with exit code 127.
Kerizer commented 4 years ago

shouldn't sth like this:

name: nvm
on:
  push:
jobs:
  nvm:
    runs-on: ubuntu-latest
    steps:
      - name: try out nvm
        run: nvm install

work now? Getting:

Run nvm install
/home/runner/work/_temp/6f648dd8-690e-4082-9988-421256974eb6.sh: line 1: nvm: command not found
##[error]Process completed with exit code 127.

nvm is a shell function, not an executable. You need to use global shell, like this:

name: nvm
on:
  push:
jobs:
  nvm:
    runs-on: ubuntu-latest
    steps:
      - name: try out nvm
        shell: bash -l {0}
        run: nvm install
privatenumber commented 3 years ago

Note, for each step that creates a new shell, you must re-initialize nvm use.

williamlfish commented 2 years ago

A little off topic, but wanted to document somewhere just incase someone else was struggling with getting nvm to work with self hosted runners and very little set up each step. If you're building your own runners images I found this works pretty well.

RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
RUN echo -n 'nvm "$@" && echo $NVM_BIN >> $GITHUB_PATH' >>  ~/.nvm/nvm.sh
RUN sudo cp ~/.nvm/nvm.sh /usr/local/bin/nvm
RUN sudo chmod +x /usr/local/bin/nvm

From here nvm install <version> && nvm use <version> persists and is very speedy.

ljharb commented 1 year ago

The version of nvm that seems to be present on the runners atm is v0.39.2 - I published v0.39.3 earlier today. How long should it take before that's updated? I'd have hoped it's largely in realtime.

karlhorky commented 3 months ago

Since actions/setup-node does not resolve version aliases like lts/* in a timely manner, we're switching our workflows off actions/setup-node and to the preinstalled nvm, adding the new path to node to the start of the $PATH:

# Use nvm because actions/setup-node does not install latest versions
# https://github.com/actions/setup-node/issues/940
- name: Install latest LTS with nvm
  run: |
    nvm install 'lts/*'
    echo "$(dirname $(which node))" >> $GITHUB_PATH
  shell: bash -l {0}

This allows all further workflow steps to use the new node binary without nvm use:

- run: node --version # v20.16.0 (latest LTS just installed by nvm)
- run: which node # /home/runner/.nvm/versions/node/v20.16.0/bin/node