Fatal Error When Using Remote Directories in kustomization.yaml with microk8s on Ubuntu 24.04 LTS

[advanceboy]

Summary

When installing microk8s via snap, kubectl encounters a fatal error when specifying a remote directory as a resource in kustomization.yaml.

Channel: latest/stable
Version: v1.29.0
OS: Ubuntu 24.04 LTS (minimized)

What Should Happen Instead?

No errors should occur, and kubectl and helm commands should succeed.

Reproduction Steps

$ sudo snap install microk8s --classic --channel=latest/stable
$ cat << EOF > ./kustomization.yaml
resources:
- https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?timeout=120&ref=v3.3.1
EOF
$ microk8s kubectl apply -k .

Expected Result:

configmap/the-map created
service/the-service created
deployment.apps/the-deployment created

Actual Result:

error: accumulating resources: accumulation err='accumulating resources from 'https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?timeout=120&ref=v3.3.1': URL is a git repository': failed to run '/snap/microk8s/6364/usr/bin/git fetch --depth=1 https://github.com/kubernetes-sigs/kustomize v3.3.1': fatal: unable to find remote helper for 'https'
: exit status 128

Introspection Report

inspection-report-20240618_154427.tar.gz

Can you suggest a fix?

The issue seems to be caused by the absence of git-remote-https $SNAP/usr/lib/git-core/git-remote-https not being referenced in the git package included with snap.

$ snap run --shell microk8s
$ $SNAP/usr/bin/git --version
git version 2.25.1
$ $SNAP/usr/bin/git clone https://github.com/octocat/Hello-World.git
Cloning into 'Hello-World'...
warning: templates not found in /usr/share/git-core/templates
fatal: unable to find remote helper for 'https'
$ exit

To resolve this, it seems possible to define the environment variable in the snapcraft.yaml file.

https://github.com/canonical/microk8s/blob/b25957fe365f9382ff5626c381a1f8093beee868/snap/snapcraft.yaml#L15-L20

--- current/snap/snapcraft.yaml
+++ suggestion/snap/snapcraft.yaml
@@ -17,6 +17,7 @@
   REAL_LD_LIBRARY_PATH: $LD_LIBRARY_PATH
   REAL_PYTHONPATH: $PYTHONPATH
   SNAPCRAFT_ARCH_TRIPLET: $SNAPCRAFT_ARCH_TRIPLET
+  GIT_EXEC_PATH: $SNAP/usr/lib/git-core/

 parts:
   build-deps:

Let's try this.

$ mkdir ~/snap-microk8s
$ cd ~/snap-microk8s
$ sudo apt install git
$ git clone https://github.com/canonical/microk8s.git -b v1.30 .
$ patch -u snap/snapcraft.yaml --no-backup-if-mismatch << 'EOF'
--- current/snap/snapcraft.yaml
+++ suggestion/snap/snapcraft.yaml
@@ -17,6 +17,7 @@
   REAL_LD_LIBRARY_PATH: $LD_LIBRARY_PATH
   REAL_PYTHONPATH: $PYTHONPATH
   SNAPCRAFT_ARCH_TRIPLET: $SNAPCRAFT_ARCH_TRIPLET
+  GIT_EXEC_PATH: $SNAP/usr/lib/git-core/

 parts:
   build-deps:
EOF
$ snapcraft --use-lxd --debug
Created snap package Snapped microk8s_v1.30.2_amd64.snap
$ sudo snap install ./microk8s_v1.30.2_amd64.snap --dangerous --classic
$ sudo apt remove git --auto-remove
$ mkdir ~/k8s-kust
$ cd ~/k8s-kust
$ cat << EOF > ./kustomization.yaml
resources:
- https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?timeout=120&ref=v3.3.1
EOF
$ microk8s kubectl apply -k .
configmap/the-map created
service/the-service created
deployment.apps/the-deployment created

I'm not familiar with Snapcraft, so I'm unsure if this is the most appropriate fix.

note

If git is installed on the host OS, the error message changes, suggesting it references the host OS's git-remote-https.

$ microk8s kubectl apply -k .
error: accumulating resources: accumulation err='accumulating resources from 'https://github.com/kubernetes-sigs/kustomize//examples/multibases?timeout=120&ref=v3.3.1': URL is a git repository': failed to run '/snap/microk8s/6364/usr/bin/git fetch --depth=1 https://github.com/kubernetes-sigs/kustomize v3.3.1': /usr/lib/git-core/git-remote-https: symbol lookup error: /snap/core20/current/lib/x86_64-linux-gnu/libpthread.so.0: undefined symbol: __libc_pthread_init, version GLIBC_PRIVATE
: exit status 128

Using a remote file instead of a remote directory works as expected.

$ cat << EOF > ./kustomization.yaml
resources:
- https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/examples/helloWorld/configMap.yaml
EOF
$ microk8s kubectl apply -k .
configmap/the-map created

[advanceboy]

Here is a workaround that has been identified:

$ sudo ln -s /snap/microk8s/current/usr/lib/git-core /usr/lib/git-core
$ microk8s kubectl apply -k .
configmap/the-map created
service/the-service created
deployment.apps/the-deployment created

However, it relies on the absence of git on the host OS, making it an impractical solution.

[advanceboy]

This workaround is better because it has less impact on the host OS:

$ cat << EOF > ./kustomization.yaml
resources:
- https://github.com/kubernetes-sigs/kustomize//examples/helloWorld/?timeout=120&ref=v3.3.1
EOF
$ GIT_EXEC_PATH=/snap/microk8s/current/usr/lib/git-core/ microk8s kubectl apply -k .
configmap/the-map created
service/the-service created
deployment.apps/the-deployment created

[advanceboy]

I have added a proposed fix to the issue description.

Additionally, similar issues have been reported in #1260 and #4453.