search

uraimo / run-on-arch-action

A Github Action that executes jobs/commands on non-x86 cpu architectures (ARMv6, ARMv7, aarch64, s390x, ppc64le, riscv64) via QEMU
BSD 3-Clause "New" or "Revised" License
665 stars 146 forks source link

Multiple steps in a job #123

Open Krakinou opened 1 year ago

Krakinou commented 1 year ago

Hello, I was wondering if it was possible to use the same containers across multiple steps in the same jobs? Something along the line of:

    - name : Build system (qemu)
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: ${{ matrix.qemu_arch }}
        distro: ubuntu20.04
        install: |
          echo "Install packages"
          apt-get update && apt-get install -y pkg1 pkg2
        run: |
          echo "System successfully built"

    - name : Build project (qemu)
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: ${{ matrix.qemu_arch }}
        distro: ubuntu20.04
        run: |
          ... build project...

    - name: functional-tests (qemu)
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: ${{ matrix.qemu_arch }}
        distro: ubuntu20.04
        run: |
          ...do some tests
      continue-on-error: false
...

The idea is to split the building chain into multiple steps so it's easier to monitor. It seems that at the moment, each steps is recreating the container at each run...

felfert commented 1 year ago

This should be possible right now, provided the steps exist in the same workflow. If it does not happen and the images get rebuilt on every run, perhaps your github.token does not have the necessary permission to push the image. After running your workflow, simply look at your project's published "packages". There should be at least one containing the string run-on-arch-action like here. If there are none, then your token does not have these permissions. In that case, in the github GUI, go the Project settings, select Actions->General in the left pane, then scroll down to Workflow permissions and select Read and write permissions there, and hit the Save button.

For possible future releases:

Oh, and looking at your example, you also should have the install: parameter present exactly the same in all invocations. After all, the install step gets saved to the image and if the 2nd and 3rd invocation do not have exactly the same install parameters, this would result in different images.

Krakinou commented 1 year ago

I do have the package in my repo, so this is not the issue

Oh, and looking at your example, you also should have the install: parameter present exactly the same in all invocations. After all, the install step gets saved to the image and if the 2nd and 3rd invocation do not have exactly the same install parameters, this would result in different images.

I will try to add the exact same install parameter and check if it works, thanks for the tip!

uraimo commented 1 year ago

Hi, thanks @felfert, duplicating the install is not necessary and until the workflow, arch and distro are still the same the next steps should pickup the original image created with the install step, if you set and configured correctly the token.

Krakinou commented 1 year ago

Hi, So I ran a bunch of test and I confirm what @felfert said : install parameter have to be the same accross multiple steps so that the docker image is not reinstalled again. If I skip the install parameter, the image is rebuilt at each step (which obviously does not work).

felfert commented 1 year ago

Well, I guess that @uraimo is right. Problem is, that you did just post an example and not the real script that you actually use. I believe you use this one. In there, ALL invocations include the parameter githubToken which enables the docker push in every step. This is wrong. If you use that parameter in the Build system step ONLY, then it should work without the duplicated install as well.

Krakinou commented 1 year ago

Yep, you're correct this is the action I'm working on. Sorry for not giving the full script (as it was quite long with a lot of stuff, I did not want to put too many unrelated info). However, I tried removing the github token parameter and it does not seems to work: I have 2 versions of the same script, one with token+install parameter, and the second one without: The first script is running correctly, with each step re-using the container as installed (. The second one is failling on the first step using run-on-arch without install parameter as it does not find git (which was installed in the container).

Or am I missing something obvious?

Krakinou commented 1 year ago

So, I made a lighter script to test. Both github token and install part are required so that the steps reused the cached package:

Here is the script:

name: Check run on arch

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-22.04

    steps:

    - name : Build system
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: armv7
        distro: ubuntu22.04
        githubToken: ${{ github.token }}
        install: |
            echo "Install packages"
            apt-get update && apt-get -y install python3
        run: |
          echo "system installed"

    - name : Check system 1 - Token & install
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: armv7
        distro: ubuntu22.04
        githubToken: ${{ github.token }}
        install: |
            echo "Install packages"
            apt-get update && apt-get -y install python3
        run: |
          echo "this step will run"
          python3 --version

    - name : Check system 2 - Install only
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: armv7
        distro: ubuntu22.04
        install: |
            echo "Install packages"
            apt-get update && apt-get -y install python3
        run: |
          echo "this step will also run but will reinstall everything"
          python3 --version

    - name : Check system 3 - Token only
      uses: uraimo/run-on-arch-action@v2.5.0
      with:
        arch: armv7
        distro: ubuntu22.04
        githubToken: ${{ github.token }}
        run: |
          echo "this step will fail"
          python3 --version
Here are the logs for step 1 ```` Run uraimo/run-on-arch-action@v2.5.0 Configuring Docker for multi-architecture support /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/Dockerfiles/Dockerfile.armv7.ubuntu22.04 run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 Build container GitHub token provided, caching to ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded latest: Pulling from krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 Digest: sha256:4ef06c3de1eac1fa18976a946d985e793bdd4bc95e7d6332754b0f218b17ac18 Status: Image is up to date for ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest Sending build context to Docker daemon 69.63kB Step 1/3 : FROM arm32v7/ubuntu:22.04 ---> eb792700b748 Step 2/3 : COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh ---> Using cache ---> 1aad52f28564 Step 3/3 : RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh ---> Using cache ---> c094c8af338e Successfully built c094c8af338e Successfully tagged run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest Using default tag: latest The push refers to repository [ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04] 479eec1abe5f: Preparing f71ce63d2bea: Preparing 8d5d171f01dd: Preparing f71ce63d2bea: Layer already exists 8d5d171f01dd: Layer already exists 479eec1abe5f: Layer already exists latest: digest: sha256:4ef06c3de1eac1fa18976a946d985e793bdd4bc95e7d6332754b0f218b17ac18 size: 948 Run container WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested this step will run Python 3.10.6 ````
Here are the logs for step 2 ```` Run uraimo/run-on-arch-action@v2.5.0 Configuring Docker for multi-architecture support /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/Dockerfiles/Dockerfile.armv7.ubuntu22.04 run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 Build container Sending build context to Docker daemon 69.63kB Step 1/3 : FROM arm32v7/ubuntu:22.04 ---> eb792700b748 Step 2/3 : COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh ---> 8d7c50bc4535 Step 3/3 : RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh Warning: rning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested ---> Running in 9c1df93cbda8 Install packages Get:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB] Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [119 kB] Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease [108 kB] Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [110 kB] Get:5 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse armhf Packages [199 kB] Get:6 http://ports.ubuntu.com/ubuntu-ports jammy/universe armhf Packages [16.8 MB] Get:7 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf Packages [1701 kB] Get:8 http://ports.ubuntu.com/ubuntu-ports jammy/restricted armhf Packages [13.3 kB] Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse armhf Packages [3270 B] Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf Packages [633 kB] Get:11 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted armhf Packages [11.2 kB] Get:12 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe armhf Packages [838 kB] Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main armhf Packages [48.9 kB] Get:14 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe armhf Packages [23.6 kB] Get:15 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted armhf Packages [10.3 kB] Get:16 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe armhf Packages [612 kB] Get:17 http://ports.ubuntu.com/ubuntu-ports jammy-security/main armhf Packages [359 kB] Get:18 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse armhf Packages [594 B] Fetched 21.8 MB in 12s (1768 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: libexpat1 libmpdec3 libpython3-stdlib libpython3.10-minimal libpython3.10-stdlib libreadline8 libsqlite3-0 media-types python3-minimal python3.10 python3.10-minimal readline-common Suggested packages: python3-doc python3-tk python3-venv python3.10-venv python3.10-doc binutils binfmt-support readline-doc The following NEW packages will be installed: libexpat1 libmpdec3 libpython3-stdlib libpython3.10-minimal libpython3.10-stdlib libreadline8 libsqlite3-0 media-types python3 python3-minimal python3.10 python3.10-minimal readline-common 0 upgraded, 13 newly installed, 0 to remove and 4 not upgraded. Need to get 5933 kB of archives. After this operation, 19.6 MB of additional disk space will be used. Get:1 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libpython3.10-minimal armhf 3.10.6-1~22.04.2ubuntu1 [794 kB] Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libexpat1 armhf 2.4.7-1ubuntu0.2 [66.9 kB] Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3.10-minimal armhf 3.10.6-1~22.04.2ubuntu1 [1941 kB] Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3-minimal armhf 3.10.6-1~22.04 [24.3 kB] Get:5 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf media-types all 7.0.0 [25.5 kB] Get:6 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf libmpdec3 armhf 2.5.1-2build2 [76.5 kB] Get:7 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf readline-common all 8.1.2-1 [53.5 kB] Get:8 http://ports.ubuntu.com/ubuntu-ports jammy/main armhf libreadline8 armhf 8.1.2-1 [128 kB] Get:9 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libsqlite3-0 armhf 3.37.2-2ubuntu0.1 [548 kB] Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libpython3.10-stdlib armhf 3.10.6-1~22.04.2ubuntu1 [1750 kB] Get:11 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3.10 armhf 3.10.6-1~22.04.2ubuntu1 [497 kB] Get:12 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf libpython3-stdlib armhf 3.10.6-1~22.04 [6910 B] Get:13 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main armhf python3 armhf 3.10.6-1~22.04 [22.8 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 5933 kB in 3s (2323 kB/s) Selecting previously unselected package libpython3.10-minimal:armhf. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 4390 files and directories currently installed.) Preparing to unpack .../libpython3.10-minimal_3.10.6-1~22.04.2ubuntu1_armhf.deb ... Unpacking libpython3.10-minimal:armhf (3.10.6-1~22.04.2ubuntu1) ... Selecting previously unselected package libexpat1:armhf. Preparing to unpack .../libexpat1_2.4.7-1ubuntu0.2_armhf.deb ... Unpacking libexpat1:armhf (2.4.7-1ubuntu0.2) ... Selecting previously unselected package python3.10-minimal. Preparing to unpack .../python3.10-minimal_3.10.6-1~22.04.2ubuntu1_armhf.deb ... Unpacking python3.10-minimal (3.10.6-1~22.04.2ubuntu1) ... Setting up libpython3.10-minimal:armhf (3.10.6-1~22.04.2ubuntu1) ... Setting up libexpat1:armhf (2.4.7-1ubuntu0.2) ... Setting up python3.10-minimal (3.10.6-1~22.04.2ubuntu1) ... Selecting previously unselected package python3-minimal. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 4692 files and directories currently installed.) Preparing to unpack .../0-python3-minimal_3.10.6-1~22.04_armhf.deb ... Unpacking python3-minimal (3.10.6-1~22.04) ... Selecting previously unselected package media-types. Preparing to unpack .../1-media-types_7.0.0_all.deb ... Unpacking media-types (7.0.0) ... Selecting previously unselected package libmpdec3:armhf. Preparing to unpack .../2-libmpdec3_2.5.1-2build2_armhf.deb ... Unpacking libmpdec3:armhf (2.5.1-2build2) ... Selecting previously unselected package readline-common. Preparing to unpack .../3-readline-common_8.1.2-1_all.deb ... Unpacking readline-common (8.1.2-1) ... Selecting previously unselected package libreadline8:armhf. Preparing to unpack .../4-libreadline8_8.1.2-1_armhf.deb ... Unpacking libreadline8:armhf (8.1.2-1) ... Selecting previously unselected package libsqlite3-0:armhf. Preparing to unpack .../5-libsqlite3-0_3.37.2-2ubuntu0.1_armhf.deb ... Unpacking libsqlite3-0:armhf (3.37.2-2ubuntu0.1) ... Selecting previously unselected package libpython3.10-stdlib:armhf. Preparing to unpack .../6-libpython3.10-stdlib_3.10.6-1~22.04.2ubuntu1_armhf.deb ... Unpacking libpython3.10-stdlib:armhf (3.10.6-1~22.04.2ubuntu1) ... Selecting previously unselected package python3.10. Preparing to unpack .../7-python3.10_3.10.6-1~22.04.2ubuntu1_armhf.deb ... Unpacking python3.10 (3.10.6-1~22.04.2ubuntu1) ... Selecting previously unselected package libpython3-stdlib:armhf. Preparing to unpack .../8-libpython3-stdlib_3.10.6-1~22.04_armhf.deb ... Unpacking libpython3-stdlib:armhf (3.10.6-1~22.04) ... Setting up python3-minimal (3.10.6-1~22.04) ... Selecting previously unselected package python3. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 5121 files and directories currently installed.) Preparing to unpack .../python3_3.10.6-1~22.04_armhf.deb ... Unpacking python3 (3.10.6-1~22.04) ... Setting up media-types (7.0.0) ... Setting up libsqlite3-0:armhf (3.37.2-2ubuntu0.1) ... Setting up libmpdec3:armhf (2.5.1-2build2) ... Setting up readline-common (8.1.2-1) ... Setting up libreadline8:armhf (8.1.2-1) ... Setting up libpython3.10-stdlib:armhf (3.10.6-1~22.04.2ubuntu1) ... Setting up libpython3-stdlib:armhf (3.10.6-1~22.04) ... Setting up python3.10 (3.10.6-1~22.04.2ubuntu1) ... Setting up python3 (3.10.6-1~22.04) ... Processing triggers for libc-bin (2.35-0ubuntu3.1) ... Removing intermediate container 9c1df93cbda8 ---> b976105ef36a Successfully built b976105ef36a Successfully tagged run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest Run container WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested this step will also run but will reinstall everything Python 3.10.6 ````
Here are the logs for step 3 ```` Run uraimo/run-on-arch-action@v2.5.0 Configuring Docker for multi-architecture support /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/Dockerfiles/Dockerfile.armv7.ubuntu22.04 run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 Build container GitHub token provided, caching to ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 WARNING! Your password will be stored unencrypted in /home/runner/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded latest: Pulling from krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04 Digest: sha256:4ef06c3de1eac1fa18976a946d985e793bdd4bc95e7d6332754b0f218b17ac18 Status: Image is up to date for ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest Sending build context to Docker daemon 69.63kB Step 1/3 : FROM arm32v7/ubuntu:22.04 ---> eb792700b748 Step 2/3 : COPY ./run-on-arch-install.sh /root/run-on-arch-install.sh ---> 7ff62b887b06 Step 3/3 : RUN chmod +x /root/run-on-arch-install.sh && /root/run-on-arch-install.sh Warning: rning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested ---> Running in 68cc33632e91 Removing intermediate container 68cc33632e91 ---> a4521b503491 Successfully built a4521b503491 Successfully tagged run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04:latest Using default tag: latest The push refers to repository [ghcr.io/krakinou/runonarch/run-on-arch-krakinou-runonarch-check-run-on-arch-armv7-ubuntu22-04] 9d76813aa08e: Preparing 2bd3b0e1ef93: Preparing 8d5d171f01dd: Preparing 8d5d171f01dd: Layer already exists 2bd3b0e1ef93: Pushed 9d76813aa08e: Pushed latest: digest: sha256:af074a21d4696411f029dcaca470ddcfc9fb124f1dfcdd31f37814b1e303fd0b size: 943 Run container WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested this step will fail /home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch-commands.sh: line 4: python3: command not found Error: The process '/home/runner/work/_actions/uraimo/run-on-arch-action/v2.5.0/src/run-on-arch.sh' failed with exit code 127````