Open darrenstarr opened 1 year ago
Hi @darrenstarr thank you for raising this.
I am not able to reproduce on 1.26 or 1.27:
sudo snap install microk8s --classic
mkdir dir
cd dir
echo '
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/ansible/awx-operator/config/default?ref=2.2.1
images:
- name: quay.io/ansible/awx-operator
newTag: 2.2.1
namespace: awx
' | tee kustomization.yaml
microk8s kubectl apply -k .
This works without issues:
namespace/awx created
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created
Which leads me to think that this is probably an environment issue. What OS is this on, and which MicroK8s version are you running?
I'm also having this issue, i believe its the git version with microk8s 1.27.1
# microk8s version
MicroK8s v1.27.1 revision 5250
This fails
# /snap/microk8s/5250/usr/bin/git --version
git version 2.25.1
This works
# /usr/bin/git --version
git version 2.34.1
If i try with a remote repository, git fails
# microk8s kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main'
error: failed to run '/snap/microk8s/5250/usr/bin/git fetch --depth=1 https://github.com/intel/intel-device-plugins-for-kubernetes main': fatal: couldn't find remote ref main
: exit status 128
If i use git from my local os bin (Ubuntu 22.04) and clone the repository, it reads the customization file but failes on the next remote repo it trys to grab
# microk8s kubectl apply -k .
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
error: accumulating resources: accumulation err='accumulating resources from 'base': '/home/[redacted]/intel-device-plugins-for-kubernetes/deployments/nfd/base' must resolve to a file': recursed accumulation of path '/home/[redacted]/intel-device-plugins-for-kubernetes/deployments/nfd/base': accumulating resources: accumulation err='accumulating resources from 'https://github.com/kubernetes-sigs/node-feature-discovery/deployment/overlays/default?ref=v0.13.1': URL is a git repository': failed to run '/snap/microk8s/5250/usr/bin/git fetch --depth=1 https://github.com/kubernetes-sigs/node-feature-discovery v0.13.1': fatal: couldn't find remote ref v0.13.1
: exit status 128
Is there any way to change the version or location for git used by microk8s? I tried but the common folder is read only? (I'm pretty novice with microk8s)
@darrenstarr do you think this is what you're seeing?
This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way
# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin
# /snap/microk8s/5250/usr/bin/git --version
git version 2.34.1
then
# microk8s kubectl apply -k "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main"
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
namespace/node-feature-discovery created
customresourcedefinition.apiextensions.k8s.io/nodefeaturerules.nfd.k8s-sigs.io created
customresourcedefinition.apiextensions.k8s.io/nodefeatures.nfd.k8s-sigs.io created
serviceaccount/nfd-master created
serviceaccount/nfd-worker created
role.rbac.authorization.k8s.io/nfd-worker created
clusterrole.rbac.authorization.k8s.io/nfd-master created
rolebinding.rbac.authorization.k8s.io/nfd-worker created
clusterrolebinding.rbac.authorization.k8s.io/nfd-master created
configmap/nfd-master-conf created
configmap/nfd-worker-conf created
service/nfd-master created
deployment.apps/nfd-master created
daemonset.apps/nfd-worker created
For me at least, it was a git version issue
Brand new install, only thing I did differently from @neoaggelos was to follow the official install guide and run microk8s enable dashboard dns registry istio
@k1n6b0b OK, I was able to reproduce this, the git version difference was enlightening.
I have not dived deep into this, but it appears that the kustomization package is using some git
features that are not available in 2.34.1
but not 2.25.1
. I am not sure what the easiest way would be to bump this version, since microk8s is simply shipping the latest git from the ubuntu 20.04 archives.
One workaround for this would be to independently install kubectl
on your host system and not rely on microk8s kubectl
.
bumping, this is a problem and should be fixed if possible.
This workaround for snap allowed me to deploy via kustomize: https://askubuntu.com/questions/919091/why-cant-snap-files-be-modified-in-any-way
# sudo mount --bind -o nodev,ro /usr/bin /snap/microk8s/5250/usr/bin # /snap/microk8s/5250/usr/bin/git --version git version 2.34.1
then
# microk8s kubectl apply -k "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=main" # Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically. # Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically. namespace/node-feature-discovery created customresourcedefinition.apiextensions.k8s.io/nodefeaturerules.nfd.k8s-sigs.io created customresourcedefinition.apiextensions.k8s.io/nodefeatures.nfd.k8s-sigs.io created serviceaccount/nfd-master created serviceaccount/nfd-worker created role.rbac.authorization.k8s.io/nfd-worker created clusterrole.rbac.authorization.k8s.io/nfd-master created rolebinding.rbac.authorization.k8s.io/nfd-worker created clusterrolebinding.rbac.authorization.k8s.io/nfd-master created configmap/nfd-master-conf created configmap/nfd-worker-conf created service/nfd-master created deployment.apps/nfd-master created daemonset.apps/nfd-worker created
For me at least, it was a git version issue
Worked for me as well. Runnig microk8s v1.28.4 Rev 6227.
Summary
Has issues resolving paths.
What Should Happen Instead?
I discovered this problem while trying to deploy awx on microk8s. See the installation instructions here : https://github.com/ansible/awx-operator#basic-install
The correct behavior is that when I execute
in a directory containing a properly formatted kustomization.yaml file, the awx deployment should take effect.
What is seen however is the following
Reproduction Steps
microk8s kubectl apply -k .
Introspection Report
I don't have a fresh installation, the reproduction steps are consistent on every machine I've tried.
Can you suggest a fix?
I'm not sure about a fix, but this is a temporary work around.
Install kustomize on its own following the instructions on their website.
Which installs kustomize in the current directory. Then use kustomize in "legacy form"
Are you interested in contributing with a fix?
no, thought I am willing to test the fix if one is provided.