bug(arm64) Cannot run the operator

[EugeneTorap]

Hi, do you know how to resolve this issue?

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl port-forward deploy/jupyternotebook-elastic 8888:8888
Error from server (NotFound): deployments.apps "jupyternotebook-elastic" not found

All steps

eugene@Eugenes-Mac-mini elastic-jupyter-operator % minikube start 
😄  minikube v1.24.0 on Darwin 12.1 (arm64)
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl apply -f ./hack/enterprise_gateway/prepare.yaml
namespace/enterprise-gateway created
serviceaccount/enterprise-gateway-sa created
clusterrole.rbac.authorization.k8s.io/enterprise-gateway-controller created
clusterrole.rbac.authorization.k8s.io/kernel-controller created
clusterrolebinding.rbac.authorization.k8s.io/enterprise-gateway-controller created

eugene@Eugenes-Mac-mini elastic-jupyter-operator % make deploy

```bash go: creating new go.mod: module tmp go get: installing executables with 'go get' in module mode is deprecated. To adjust and download dependencies of the current module, use 'go get -d'. To install using requirements of the current module, use 'go install'. To install ignoring the current module, use 'go install' with a version, like 'go install example.com/cmd@latest'. For more information, see https://golang.org/doc/go-get-install-deprecation or run 'go help get' or 'go help install'. go get: added github.com/fatih/color v1.9.0 go get: added github.com/go-logr/logr v0.2.0 go get: added github.com/gobuffalo/flect v0.2.2 go get: added github.com/gogo/protobuf v1.3.1 go get: added github.com/google/gofuzz v1.1.0 go get: added github.com/inconshreveable/mousetrap v1.0.0 go get: added github.com/json-iterator/go v1.1.10 go get: added github.com/mattn/go-colorable v0.1.8 go get: added github.com/mattn/go-isatty v0.0.12 go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go get: added github.com/modern-go/reflect2 v1.0.1 go get: added github.com/spf13/cobra v1.1.1 go get: added github.com/spf13/pflag v1.0.5 go get: added golang.org/x/mod v0.3.0 go get: added golang.org/x/net v0.0.0-20201110031124-69a78807bb2b go get: added golang.org/x/sys v0.0.0-20201112073958-5cba982894dd go get: added golang.org/x/text v0.3.4 go get: added golang.org/x/tools v0.0.0-20200616133436-c1934b75d054 go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go get: added gopkg.in/inf.v0 v0.9.1 go get: added gopkg.in/yaml.v2 v2.3.0 go get: added gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 go get: added k8s.io/api v0.20.2 go get: added k8s.io/apiextensions-apiserver v0.20.2 go get: added k8s.io/apimachinery v0.20.2 go get: added k8s.io/klog/v2 v2.4.0 go get: added k8s.io/utils v0.0.0-20201110183641-67b214c5f920 go get: added sigs.k8s.io/controller-tools v0.5.0 go get: added sigs.k8s.io/structured-merge-diff/v4 v4.0.2 go get: added sigs.k8s.io/yaml v1.2.0 /Users/eugene/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases go: creating new go.mod: module tmp go get: installing executables with 'go get' in module mode is deprecated. To adjust and download dependencies of the current module, use 'go get -d'. To install using requirements of the current module, use 'go install'. To install ignoring the current module, use 'go install' with a version, like 'go install example.com/cmd@latest'. For more information, see https://golang.org/doc/go-get-install-deprecation or run 'go help get' or 'go help install'. go get: added cloud.google.com/go v0.38.0 go get: added github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 go get: added github.com/Azure/go-autorest/autorest v0.9.0 go get: added github.com/Azure/go-autorest/autorest/adal v0.5.0 go get: added github.com/Azure/go-autorest/autorest/date v0.1.0 go get: added github.com/Azure/go-autorest/logger v0.1.0 go get: added github.com/Azure/go-autorest/tracing v0.5.0 go get: added github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd go get: added github.com/PuerkitoBio/purell v1.1.1 go get: added github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 go get: added github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5 go get: added github.com/davecgh/go-spew v1.1.1 go get: added github.com/dgrijalva/jwt-go v3.2.0+incompatible go get: added github.com/docker/docker v0.7.3-0.20190327010347-be7ac8be2ae0 go get: added github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96 go get: added github.com/emicklei/go-restful v2.9.5+incompatible go get: added github.com/evanphx/json-patch v4.5.0+incompatible go get: added github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d go get: added github.com/ghodss/yaml v1.0.0 go get: added github.com/go-errors/errors v1.0.1 go get: added github.com/go-openapi/jsonpointer v0.19.3 go get: added github.com/go-openapi/jsonreference v0.19.3 go get: added github.com/go-openapi/spec v0.19.5 go get: added github.com/go-openapi/swag v0.19.5 go get: added github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d go get: added github.com/golang/protobuf v1.3.2 go get: added github.com/google/btree v1.0.0 go get: added github.com/google/go-cmp v0.3.0 go get: added github.com/google/gofuzz v1.0.0 go get: added github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go get: added github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d go get: added github.com/gophercloud/gophercloud v0.1.0 go get: added github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 go get: added github.com/hashicorp/errwrap v1.0.0 go get: added github.com/hashicorp/go-multierror v1.0.0 go get: added github.com/hashicorp/golang-lru v0.5.1 go get: added github.com/imdario/mergo v0.3.5 go get: added github.com/inconshreveable/mousetrap v1.0.0 go get: added github.com/jonboulle/clockwork v0.1.0 go get: added github.com/json-iterator/go v1.1.8 go get: added github.com/konsorten/go-windows-terminal-sequences v1.0.1 go get: added github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de go get: added github.com/mailru/easyjson v0.7.0 go get: added github.com/mattn/go-runewidth v0.0.7 go get: added github.com/mitchellh/go-wordwrap v1.0.0 go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go get: added github.com/modern-go/reflect2 v1.0.1 go get: added github.com/olekukonko/tablewriter v0.0.4 go get: added github.com/peterbourgon/diskv v2.0.1+incompatible go get: added github.com/pkg/errors v0.8.1 go get: added github.com/posener/complete/v2 v2.0.1-alpha.12 go get: added github.com/posener/script v1.0.4 go get: added github.com/russross/blackfriday v1.5.2 go get: added github.com/sirupsen/logrus v1.4.2 go get: added github.com/spf13/cobra v0.0.5 go get: added github.com/spf13/pflag v1.0.5 go get: added github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca go get: added golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392 go get: added golang.org/x/net v0.0.0-20191004110552-13f9640d40b9 go get: added golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 go get: added golang.org/x/sys v0.0.0-20190922100055-0a153f010e69 go get: added golang.org/x/text v0.3.2 go get: added golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 go get: added google.golang.org/appengine v1.5.0 go get: added gopkg.in/inf.v0 v0.9.1 go get: added gopkg.in/yaml.v2 v2.2.4 go get: added gopkg.in/yaml.v3 v3.0.0-20191026110619-0b21df46bc1d go get: added k8s.io/api v0.17.0 go get: added k8s.io/apimachinery v0.17.0 go get: added k8s.io/cli-runtime v0.17.0 go get: added k8s.io/client-go v0.17.0 go get: added k8s.io/component-base v0.17.0 go get: added k8s.io/klog v1.0.0 go get: added k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a go get: added k8s.io/kubectl v0.0.0-20191219154910-1528d4eea6dd go get: added k8s.io/utils v0.0.0-20191114184206-e782cd3c129f go get: added sigs.k8s.io/kustomize v2.0.3+incompatible go get: added sigs.k8s.io/kustomize/api v0.3.2 go get: added sigs.k8s.io/kustomize/cmd/config v0.0.5 go get: added sigs.k8s.io/kustomize/cmd/kubectl v0.0.3 go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.5.4 go get: added sigs.k8s.io/kustomize/kyaml v0.0.6 go get: added sigs.k8s.io/yaml v1.1.0 cd config/manager && /Users/eugene/go/bin/kustomize edit set image controller=ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest /Users/eugene/go/bin/kustomize build config/default | kubectl apply -f - namespace/elastic-jupyter-operator-system created customresourcedefinition.apiextensions.k8s.io/jupytergateways.kubeflow.tkestack.io created customresourcedefinition.apiextensions.k8s.io/jupyterkernels.kubeflow.tkestack.io created customresourcedefinition.apiextensions.k8s.io/jupyterkernelspecs.kubeflow.tkestack.io created customresourcedefinition.apiextensions.k8s.io/jupyterkerneltemplates.kubeflow.tkestack.io created customresourcedefinition.apiextensions.k8s.io/jupyternotebooks.kubeflow.tkestack.io created role.rbac.authorization.k8s.io/elastic-jupyter-operator-leader-election-role created clusterrole.rbac.authorization.k8s.io/elastic-jupyter-operator-manager-role created rolebinding.rbac.authorization.k8s.io/elastic-jupyter-operator-leader-election-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/elastic-jupyter-operator-manager-rolebinding created deployment.apps/elastic-jupyter-operator-controller-manager created ```

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupyternotebook.yaml
jupyternotebook.kubeflow.tkestack.io/jupyternotebook-elastic created

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl apply -f ./examples/elastic/kubeflow.tkestack.io_v1alpha1_jupytergateway.yaml
jupytergateway.kubeflow.tkestack.io/jupytergateway-elastic created

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl port-forward deploy/jupyternotebook-elastic 8888:8888
Error from server (NotFound): deployments.apps "jupyternotebook-elastic" not found

[gaocegege]

Hi, could you please show the logs of elastic-jupyter-operator-controller-manager?

I think the operator is not running.

[EugeneTorap]

@gaocegege I am working on Apple M1. It's ARM64 chip. You support only amd64 arch. I've tested it on linux/intel server and all is ok. When I use M1 arm64 I have a issue:

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                           READY   STATUS              RESTARTS   AGE
elastic-jupyter-operator-system   elastic-jupyter-operator-controller-manager-64fffc97d7-cg4np   0/1     ContainerCreating   0          18s
kube-system                       coredns-78fcd69978-nzpk7                                       1/1     Running             0          35s
kube-system                       etcd-minikube                                                  1/1     Running             0          48s
kube-system                       kube-apiserver-minikube                                        1/1     Running             0          50s
kube-system                       kube-controller-manager-minikube                               1/1     Running             0          48s
kube-system                       kube-proxy-b92g7                                               1/1     Running             0          35s
kube-system                       kube-scheduler-minikube                                        1/1     Running             0          48s
kube-system                       storage-provisioner                                            1/1     Running             0          47s

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                           READY   STATUS    RESTARTS      AGE
elastic-jupyter-operator-system   elastic-jupyter-operator-controller-manager-64fffc97d7-cg4np   0/1     Error     1 (14s ago)   54s
kube-system                       coredns-78fcd69978-nzpk7                                       1/1     Running   0             71s
kube-system                       etcd-minikube                                                  1/1     Running   0             84s
kube-system                       kube-apiserver-minikube                                        1/1     Running   0             86s
kube-system                       kube-controller-manager-minikube                               1/1     Running   0             84s
kube-system                       kube-proxy-b92g7                                               1/1     Running   0             71s
kube-system                       kube-scheduler-minikube                                        1/1     Running   0             84s
kube-system                       storage-provisioner                                            1/1     Running   0             83s

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                           READY   STATUS             RESTARTS      AGE
elastic-jupyter-operator-system   elastic-jupyter-operator-controller-manager-64fffc97d7-cg4np   0/1     CrashLoopBackOff   1 (12s ago)   60s
kube-system                       coredns-78fcd69978-nzpk7                                       1/1     Running            0             77s
kube-system                       etcd-minikube                                                  1/1     Running            0             90s
kube-system                       kube-apiserver-minikube                                        1/1     Running            0             92s
kube-system                       kube-controller-manager-minikube                               1/1     Running            0             90s
kube-system                       kube-proxy-b92g7                                               1/1     Running            0             77s
kube-system                       kube-scheduler-minikube                                        1/1     Running            0             90s
kube-system                       storage-provisioner                                            1/1     Running            0             89s

I will create a new issue related to Support ARM64 architecture.

[gaocegege]

Interesting. I do not have a ARM64 machine here. Could you please show the logs of the elastic-jupyter-operator-controller-manager-64fffc97d7-cg4np?

Maybe this command kubectl logs -n elastic-jupyter-operator-system logs -p elastic-jupyter-operator-controller-manager-64fffc97d7-cg4np

[EugeneTorap]

@gaocegege Back-off restarting failed container

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl get event -n elastic-jupyter-operator-system
LAST SEEN   TYPE      REASON              OBJECT                                                              MESSAGE
2m44s       Normal    Scheduled           pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Successfully assigned elastic-jupyter-operator-system/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr to minikube
29s         Normal    Pulling             pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Pulling image "ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest"
95s         Normal    Pulled              pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Successfully pulled image "ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest" in 1m7.989985864s
24s         Normal    Created             pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Created container manager
24s         Normal    Started             pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Started container manager
83s         Normal    Pulled              pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Successfully pulled image "ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest" in 7.805613796s
5s          Warning   BackOff             pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Back-off restarting failed container
61s         Normal    Pulled              pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Successfully pulled image "ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest" in 5.947165502s
24s         Normal    Pulled              pod/elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr    Successfully pulled image "ccr.ccs.tencentyun.com/kubeflow-oteam/elastic-jupyter-operator:latest" in 4.720454502s
2m44s       Normal    SuccessfulCreate    replicaset/elastic-jupyter-operator-controller-manager-64fffc97d7   Created pod: elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr
2m44s       Normal    ScalingReplicaSet   deployment/elastic-jupyter-operator-controller-manager              Scaled up replica set elastic-jupyter-operator-controller-manager-64fffc97d7 to 1

[gaocegege]

Could you please show the log? kubectl logs -n elastic-jupyter-operator-system logs -p elastic-jupyter-operator-controller-manager-64fffc97d7-cg4np

[EugeneTorap]

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl get pods --all-namespaces                                                                                   
NAMESPACE                         NAME                                                           READY   STATUS             RESTARTS      AGE
elastic-jupyter-operator-system   elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr   0/1     CrashLoopBackOff   7 (62s ago)   14m
kube-system                       coredns-78fcd69978-crw4p                                       1/1     Running            0             15m
kube-system                       etcd-minikube                                                  1/1     Running            0             15m
kube-system                       kube-apiserver-minikube                                        1/1     Running            0             15m
kube-system                       kube-controller-manager-minikube                               1/1     Running            0             15m
kube-system                       kube-proxy-brv2b                                               1/1     Running            0             15m
kube-system                       kube-scheduler-minikube                                        1/1     Running            0             15m
kube-system                       storage-provisioner                                            1/1     Running            0             15m

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl logs -n elastic-jupyter-operator-system logs -p elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr
Error from server (NotFound): pods "logs" not found

[gaocegege]

Sorry, the command should be kubectl -n elastic-jupyter-operator-system logs -p elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr

[EugeneTorap]

eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl -n elastic-jupyter-operator-system logs -p elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr
eugene@Eugenes-Mac-mini elastic-jupyter-operator % kubectl -n elastic-jupyter-operator-system logs -p elastic-jupyter-operator-controller-manager-64fffc97d7-s7kdr
eugene@Eugenes-Mac-mini elastic-jupyter-operator % 

[EugeneTorap]

Maybe there are no logs for this pod. I will investigate it.

[gaocegege]

Sure, thanks. Can you please show the exit code of the pod?

[EugeneTorap]

https://www.alibabacloud.com/press-room/alibaba-cloud-unveils-new-server-chips-to-optimize-cloud-computing-services

Built upon advanced 5nm process technology, Yitian 710 is powered by 128 Arm cores with 3.2GHz top clock speed to deliver exceptional performance and excellent energy efficiency. Each processor chip has 60 billion integrated transistors. Yitian 710 is the first server processor that is compatible with the latest Armv9 architecture and includes 8 DDR5 channels and 96-lane PCIe 5.0, providing high memory and I/O bandwidth.

@gaocegege Can we create a small arm64 server with minikube and test it?

[gaocegege]

@EugeneTorap I am not sure if minikube supports it.

[EugeneTorap]

@gaocegege I wrote about it in discord

[gaocegege]

Gotcha, I replied there.