kustomize feature of kubectl does not function

[darrenstarr]

Summary

microk8s kubectl apply -k

Has issues resolving paths.

What Should Happen Instead?

I discovered this problem while trying to deploy awx on microk8s. See the installation instructions here : https://github.com/ansible/awx-operator#basic-install

The correct behavior is that when I execute

kubectl apply -k .

in a directory containing a properly formatted kustomization.yaml file, the awx deployment should take effect.

What is seen however is the following

root@microk8s:~# microk8s kubectl apply -k .
error: accumulating resources: accumulation err='accumulating resources from 'github.com/ansible/awx-operator/config/default?ref=2.2.1': evalsymlink failure on '/root/github.com/ansible/awx-operator/config/default?ref=2.2.1' : lstat /root/github.com: no such file or directory': git cmd = '/snap/microk8s/5219/usr/bin/git fetch --depth=1 origin 2.2.1': exit status 128

Reproduction Steps

1. Install microk8s on a fresh Ubuntu 22.04 LTS system using snap
2. Create a file named kustomization.yaml containing the following text

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
   - github.com/ansible/awx-operator/config/default?ref=2.2.1
   images:
   - name: quay.io/ansible/awx-operator
   newTag: 2.2.1
   namespace: awx

3. execute microk8s kubectl apply -k .

Introspection Report

I don't have a fresh installation, the reproduction steps are consistent on every machine I've tried.

Can you suggest a fix?

I'm not sure about a fix, but this is a temporary work around.

Install kustomize on its own following the instructions on their website.

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  |  bash

Which installs kustomize in the current directory. Then use kustomize in "legacy form"

./kustomize build | microk8s kubectl apply -f -

Are you interested in contributing with a fix?

no, thought I am willing to test the fix if one is provided.

[neoaggelos]

Hi @darrenstarr thank you for raising this.

I am not able to reproduce on 1.26 or 1.27:

sudo snap install microk8s --classic
mkdir dir
cd dir
echo '
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.2.1
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.2.1
namespace: awx
' | tee kustomization.yaml
microk8s kubectl apply -k .

This works without issues:

namespace/awx created
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Which leads me to think that this is probably an environment issue. What OS is this on, and which MicroK8s version are you running?

[k1n6b0b]

I'm also having this issue, i believe its the git version with microk8s 1.27.1

# microk8s version
MicroK8s v1.27.1 revision 5250

This fails

# /snap/microk8s/5250/usr/bin/git --version
git version 2.25.1

This works

# /usr/bin/git --version
git version 2.34.1

If i try with a remote repository, git fails

# microk8s kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main'             
error: failed to run '/snap/microk8s/5250/usr/bin/git fetch --depth=1 https://github.com/intel/intel-device-plugins-for-kubernetes main': fatal: couldn't find remote ref main
: exit status 128

If i use git from my local os bin (Ubuntu 22.04) and clone the repository, it reads the customization file but failes on the next remote repo it trys to grab

# microk8s kubectl apply -k .
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
error: accumulating resources: accumulation err='accumulating resources from 'base': '/home/[redacted]/intel-device-plugins-for-kubernetes/deployments/nfd/base' must resolve to a file': recursed accumulation of path '/home/[redacted]/intel-device-plugins-for-kubernetes/deployments/nfd/base': accumulating resources: accumulation err='accumulating resources from 'https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref=v0.13.1': URL is a git repository': failed to run '/snap/microk8s/5250/usr/bin/git fetch --depth=1 https://github.com/kubernetes-sigs/node-feature-discovery v0.13.1': fatal: couldn't find remote ref v0.13.1
: exit status 128

Is there any way to change the version or location for git used by microk8s? I tried but the common folder is read only? (I'm pretty novice with microk8s)

@darrenstarr do you think this is what you're seeing?

[k1n6b0b]

This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way

# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin
# /snap/microk8s/5250/usr/bin/git --version
git version 2.34.1

then

# microk8s kubectl apply -k "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main"
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/node-feature-discovery created
customresourcedefinition.apiextensions.k8s.io/nodefeaturerules.nfd.k8s-sigs.io created
customresourcedefinition.apiextensions.k8s.io/nodefeatures.nfd.k8s-sigs.io created
serviceaccount/nfd-master created
serviceaccount/nfd-worker created
role.rbac.authorization.k8s.io/nfd-worker created
clusterrole.rbac.authorization.k8s.io/nfd-master created
rolebinding.rbac.authorization.k8s.io/nfd-worker created
clusterrolebinding.rbac.authorization.k8s.io/nfd-master created
configmap/nfd-master-conf created
configmap/nfd-worker-conf created
service/nfd-master created
deployment.apps/nfd-master created
daemonset.apps/nfd-worker created

For me at least, it was a git version issue

[akzov]

Brand new install, only thing I did differently from @neoaggelos was to follow the official install guide and run microk8s enable dashboard dns registry istio

[neoaggelos]

@k1n6b0b OK, I was able to reproduce this, the git version difference was enlightening.

I have not dived deep into this, but it appears that the kustomization package is using some git features that are not available in 2.34.1 but not 2.25.1. I am not sure what the easiest way would be to bump this version, since microk8s is simply shipping the latest git from the ubuntu 20.04 archives.

One workaround for this would be to independently install kubectl on your host system and not rely on microk8s kubectl.

[kriegster108]

bumping, this is a problem and should be fixed if possible.

[smarthusker]

This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way

# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin
# /snap/microk8s/5250/usr/bin/git --version
git version 2.34.1

then

# microk8s kubectl apply -k "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main"
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/node-feature-discovery created
customresourcedefinition.apiextensions.k8s.io/nodefeaturerules.nfd.k8s-sigs.io created
customresourcedefinition.apiextensions.k8s.io/nodefeatures.nfd.k8s-sigs.io created
serviceaccount/nfd-master created
serviceaccount/nfd-worker created
role.rbac.authorization.k8s.io/nfd-worker created
clusterrole.rbac.authorization.k8s.io/nfd-master created
rolebinding.rbac.authorization.k8s.io/nfd-worker created
clusterrolebinding.rbac.authorization.k8s.io/nfd-master created
configmap/nfd-master-conf created
configmap/nfd-worker-conf created
service/nfd-master created
deployment.apps/nfd-master created
daemonset.apps/nfd-worker created

For me at least, it was a git version issue

Worked for me as well. Runnig microk8s v1.28.4 Rev 6227.