Using microk8s ctr in workflow

[erikgb]

I am trying out this GHA to see if microk8s can work better than k3d/k3s on GH hosted runners with limited resources. So I have created a draft PR for the migration: https://github.com/statnett/image-scanner-operator/pull/231/files

Our current workflow is based on building the operator image to file, and importing it into the cluster. Browsing https://microk8s.io/docs/registry-images, I thought this would work without issues, but it seems like it's not: https://github.com/statnett/image-scanner-operator/actions/runs/4251577056/jobs/7394121697

$ microk8s ctr images ls
Arguments file /var/snap/microk8s/[4](https://github.com/statnett/image-scanner-operator/actions/runs/4251577056/jobs/7394121697#step:10:4)[5](https://github.com/statnett/image-scanner-operator/actions/runs/4251577056/jobs/7394121697#step:10:5)[9](https://github.com/statnett/image-scanner-operator/actions/runs/4251577056/jobs/7394121697#step:10:9)5/args/ctr is missing.

Any idea how to fix this? Am I doing something wrong, or is there a limitation in the GHA?

[balchua]

Thanks for reporting the issue.

Its pretty standard from what i can tell. Its just strange that the file /var/snap/microk8s/args/ctr doesn't exist. It should be part of the default microk8s installation. Let me check with the microk8s folks

[erikgb]

@balchua Thanks!

[balchua]

Hi @erikgb Will it be possible to try sudo microk8s ctr ls? In github action we don't have the chance to switch users session to be a part of microk8s group. Many of the microk8s commands will require the use of sudo or sg commands.

Let me know if it works.

[erikgb]

@balchua Running the commands with sudo seems to work. Tested here: https://github.com/statnett/image-scanner-operator/actions/runs/4253842764/jobs/7399288010

Maybe you can add a GHA input parameter to import image(s) from file? Could that make sense?

[balchua]

Hi @erikgb Instead of building the image as a tar, why don't you just use the built-in image registry for which you already enabled as an addon.

https://microk8s.io/docs/registry-built-in You can start microk8s first in your pipeline, then you build your image and push to the internal registry as explained in the link i shared above. This way you don't need extra step to load image into containerd.

[erikgb]

Instead of building the image as a tar, why don't you just use the built-in image registry for which you already enabled as an addon.

@balchua Thanks for the suggestion! I will try that! 😍

[erikgb]

@balchua I seem unable to make the docker push action push directly to the built-in image registry. I've tried both localhost and the runner public IP with the aide from https://github.com/haythem/public-ip.

Localhost: https://github.com/statnett/image-scanner-operator/actions/runs/4260746563/jobs/7414277544 Public IP: https://github.com/statnett/image-scanner-operator/actions/runs/4261081167/jobs/7414999400

Any other suggestion?

[balchua]

I am guessing the registry pod isn't ready yet. You can try waiting for the deployment to be available before proceeding.

kubectl wait deployment -n container-registry registry --for condition=Available=True --timeout=90s

[erikgb]

@balchua Thanks for the suggestion, but no luck.

deployment.apps/registry condition met

I tried with both localhost and runner public IP. Same as before. It would be really nice to be able to import an image not pushed to a public registry. We run this workflow on forked PRs also, so using our GHCR registry for PRs is something we try to avoid....

[balchua]

I am not against adding a feature to load image tar balls to containerd.

Do you think the usage will look like this?

jobs:
  test:
    runs-on: ubuntu-latest
    name: A job to install MicroK8s with strict confinement
    steps:
    - uses: balchua/microk8s-actions@v0.3.0
      with:
        channel: '1.26-strict/stable'
        devMode: 'true'
        addons: '["rbac", "dns", "hostpath-storage", "registry", "metrics-server"]'
        image-tar: '["/path/to/tarball-1.tar", "/path/to/tarball-2.tar"]'

[balchua]

I think i know what is happening here. The hostpath-provisioner isn't ready and that the registry is not yet mounted to the host path volume.

I've released a new version v0.3.2 which allows you to push to the local registry.

Here's how I test it.

https://github.com/balchua/microk8s-actions/blob/7cab1049ff858184e1524e46da64d090308769e5/.github/workflows/e2e.yaml#L19-L26

Do you mind giving it a try?

[erikgb]

@balchua I tested the v0.3.2 tag now, and while I can observe that the microk8s action waits longer for things to be ready, the docker-push action is still not satisfied: https://github.com/statnett/image-scanner-operator/actions/runs/4270513132/jobs/7434359918.

Are you able to test with the docker-push action to see what's going on? The docker GH actions are really good, and I would prefer to avoid using docker CLI directly in my workflows.

[balchua]

@erikgb i will give docker buildx GH action a try. And thanks for your patience on this issue. 🙏

[balchua]

Hi @erikgb I think i have it working on my end. The key is to allow buildx to use the host network, so that localhost will refer to the runner's host instead of the buildx container.

so here's what i did:

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2
      with:
        driver-opts: |
          network=host

The result of the build is over here.

The source is in the fix/test-with-buildx branch

Run docker/build-push-action@v4
  with:
    context: ./test-docker-build/
    push: true
    tags: localhost:32000/myrepo/app:latest
    load: false
    no-cache: false
    pull: false
    github-token: ***
GitHub Actions runtime token access controls
Docker info
Buildx version
/usr/bin/docker buildx build --iidfile /tmp/docker-build-push-gfxlur/iidfile --provenance mode=max,builder-id=https://github.com/balchua/microk8s-actions/actions/runs/4272798625 --tag localhost::32000/myrepo/app:latest --metadata-file /tmp/docker-build-push-gfxlur/metadata-file --push ./test-docker-build/
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 9[4](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:4)B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/busybox:1
#3 ...

#4 [auth] library/busybox:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/busybox:1
#3 DONE 1.2s

#[5](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:5) [internal] load build context
#5 transferring context: [6](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:6)1B done
#5 DONE 0.0s

#6 [1/3] FROM docker.io/library/busybox:1@sha256:[7](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:7)b3ccabffc97de[8](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:8)72a30dfd234fd[9](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:9)72a66d247c8cfc69b0550f276481852627c
#6 resolve docker.io/library/busybox:1@sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c done
#6 sha256:205dae5015e78dd8c4d302e3db4eb31576fac715b46d099fe09680ba28093a7a 2.59MB / 2.59MB 0.1s done
#6 extracting sha256:205dae5015e78dd8c4d302e3db4eb31576fac715b46d099fe09680ba28093a7a 0.1s done
#6 DONE 0.3s

#7 [2/3] COPY run.sh /run.sh
#7 DONE 0.0s

#8 [3/3] RUN chmod +x /run.sh
#8 DONE 0.1s

#9 exporting to image
#9 exporting layers 0.0s done
#9 exporting manifest sha256:d3c8e2d0bc74[10](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:11)eb7d87156ecedc253c0058bea77ce7fb65f714d97d37070897 done
#9 exporting config sha256:aa9aa5190f8951ede409bf75c1e5b2b95f98d52e9fc3e1f0f8[21](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:23)e878cbc9f7d8 done
#9 exporting attestation manifest sha256:b43a88fc02a0fe5bc7fd3e6cab0b4857822df822006455fb7dbfc5d0[113](https://github.com/balchua/microk8s-actions/actions/runs/4272798625/jobs/7438254327#step:6:117)a5ced 0.0s done
#9 exporting manifest list sha256:4220f4d75696cf46b5659d8d6706c3fef3bb75e7f1863d2c8f1881c895da38b9
#9 exporting manifest list sha256:4220f4d75696cf46b5659d8d6706c3fef3bb75e7f1863d2c8f1881c895da38b9 done
#9 pushing layers 0.1s done
#9 pushing manifest for localhost:32000/myrepo/app:latest@sha256:4220f4d75696cf46b5659d8d6706c3fef3bb75e7f1863d2c8f1881c895da38b9 0.0s done
#9 DONE 0.2s

[erikgb]

Eureka, it works! Well done, @balchua! Thanks! 👏

[balchua]

@erikgb finally 😁. If you are all good, let me know so i can close the issue.

[erikgb]

@balchua, I'm all good. I would have preferred if the startup-time were a bit faster, but that could be handled as another improvement. Do you think this tip should be documented? I would imagine using the docker GH actions is not uncommon to use for importing images into the cluster.

[balchua]

@erikgb you're right, documenting it will definitely a big help to those facing the same problem.

I will try to add an FAQ section. Thanks for the feedback.

In order to make it start up faster you can always resort to using

sudo microk8s ctr image import <image tar>

Then you can remove the following addons:

• hostpath-storage
• registry

[balchua]

@erikgb I recently added this new feature from MicroK8s, called Launch Configuration. The MicroK8s v1.27+ also allows side loading of images, which is probably what you need.

If you are interested, v0.4.0 of the microk8s-actions include these 2 features.

[erikgb]

@balchua, thanks for the ping! I will try it out. 👍

[erikgb]

@balchua This works better than the last time I tried, thanks: https://github.com/statnett/image-scanner-operator/pull/549 👍

[balchua]

Thanks for the feedback!!