Closed Baojizhong closed 2 years ago
houshengbo
commented
3 years ago @Baojizhong Would you mind sharing the content of your KnativeServing CR? Which platform do you use as the Kubernetes service?
Are you using https://github.com/knative/operator/releases/download/v0.24.0/operator.yaml as your operator.yaml?
Baojizhong
commented
3 years ago @houshengbo 我看你也是个中国人，我就打汉字了。 knative CR ：
apiVersion: operator.knative.dev/v1alpha1 kind: KnativeServing metadata: name: knative-service namespace: knative-service `
使用的是十六台centos物理机搭建的v1.19.13 k8s集群 operator是将这个https://github.com/knative/operator/releases/download/v0.24.0/operator.yaml 中的operator镜像地址替换成了私有化仓库地址，没有做其他改动。
houshengbo
commented
3 years ago Did you rebuild the image or still use the same image as in the release.
This is a public discussion. We better use the language everyone can read and join the discussion.
Baojizhong
commented
3 years ago @houshengbo same image. ` docker pull gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:21c3426aa7fb67566193b9d33aadceb167e347bbad876d3bf5e63c8270f6ff20
docker tag gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:21c3426aa7fb67566193b9d33aadceb167e347bbad876d3bf5e63c8270f6ff20 inner-hub.com/knative/operator:v0.24.0
docker push inner-hub.com/knative/operator:v0.24.0 ` then use inner-hub.com/knative/operator:v0.24.0
Baojizhong
commented
3 years ago 在这个故障集群中，尝试部署了v0.23.0，获得了同样的错误。使用另一个集群部署0.23.0没有遇到错误，怀疑是集群间有什么参数差异导致，有可供参考的可能导致此问题的参数建议吗？
houshengbo
commented
3 years ago In this situation, the question becomes what is the difference between the cluster that is working and the cluster that is not working, in terms of opening the ko path.
What I suggest is to try the image built with the current source code. You can change the source code, checking the permission of the directory of /var/run/ko/knative-serving. Compare the difference between the clusters you use.
You can use os.Chmod(dir_path, mode_num) to change the mode, and you can use os.Stat(dir_path) to check the mode. You can change along the line: https://github.com/knative/operator/blob/95b21edd6b019d48c33c27ed851c222f20c8ab05/pkg/reconciler/common/releases.go#L355.
github-actions[bot]
commented
2 years ago This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.
gshaibi
commented
2 years ago I'm having this issue too
knative-prow-robot
commented
2 years ago @gshaibi: You can't reopen an issue/PR unless you authored it or you are a collaborator.
xunfeng1980
commented
2 years ago I'm having this issue too
knative-prow[bot]
commented
2 years ago @xunfeng1980: You can't reopen an issue/PR unless you authored it or you are a collaborator.
sysedwinistrator
commented
2 years ago I'm experiencing the issue, too. However only on my RKE arm64 homelab cluster, on an amd64 GKE cluster the operator works fine.
With the help of undocker and dive I took a look at the image to see whether there are any differences between the amd64 and arm64 one. As it should be, the only difference is the ko-app/operator binary. But I noticed that in both images some of the directories under /var/run/ko were missing permissions in dive or didn't have a separate listing when showing the tar contents via undocker:
It's possible that different container runtimes handle these apparently non-existing directories entries differently, or that they differ in the root user's ability to open a directory with no permissions.
I couldn't find a Dockerfile or any other file that shows how the manifests are copied into the docker file. So I fixed it with a multi-stage docker build:
FROM gcr.io/knative-releases/knative.dev/operator/cmd/operator:v1.7.0 as get_manifests
FROM alpine:latest as chmodder
COPY --from=get_manifests /var/run/ko /var/run/ko
RUN chmod -R 755 /var/run/ko
FROM gcr.io/knative-releases/knative.dev/operator/cmd/operator:v1.7.0
COPY --from=chmodder /var/run/ko /var/run/koI pushed the image to edwinmowen/knative-operator:v1.7.0 on dockerhub for linux/arm64 and linux/amd64. After switching to the image the operator doesn't crash anymore and creates the resources for the knative serving.
Describe the bug knative-operator pod panic when apply a knativeServing
Expected behavior no panic
To Reproduce apply operator.yaml apply KnativeServing
Knative release version 0.24.0 k8s 1.19.13
Additional context {"severity":"INFO","timestamp":"2021-08-06T11:30:44.35666209Z","logger":"knative-operator","caller":"knativeserving/knativeserving.go:99","message":"Reconciling KnativeServing","knative.dev/pod":"knative-operator-8695967fcb-prfm8","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"3900070f-b460-4118-b2a8-4ecf91a6058d","knative.dev/key":"knative-service/knative-service","status":{"observedGeneration":1,"conditions":[{"type":"DependenciesInstalled","status":"Unknown","lastTransitionTime":"2021-08-06T11:30:44Z"},{"type":"DeploymentsAvailable","status":"Unknown","lastTransitionTime":"2021-08-06T11:30:44Z"},{"type":"InstallSucceeded","status":"Unknown","lastTransitionTime":"2021-08-06T11:30:44Z"},{"type":"Ready","status":"Unknown","lastTransitionTime":"2021-08-06T11:30:44Z"},{"type":"VersionMigrationEligible","status":"Unknown","lastTransitionTime":"2021-08-06T11:30:44Z"}]}} panic: open /var/run/ko/knative-serving: permission denied
goroutine 607 [running]: knative.dev/operator/pkg/reconciler/common.getLatestRelease(0x20a2cd0, 0xc0012e9180, 0x0, 0x0, 0x0, 0x0) knative.dev/operator/pkg/reconciler/common/releases.go:406 +0xba knative.dev/operator/pkg/reconciler/common.latestRelease(...) knative.dev/operator/pkg/reconciler/common/releases.go:386 knative.dev/operator/pkg/reconciler/common.TargetVersion(0x20a2cd0, 0xc0012e9180, 0xc000b43570, 0x537aa5) knative.dev/operator/pkg/reconciler/common/releases.go:58 +0x2a5 knative.dev/operator/pkg/reconciler/common.IsVersionValidMigrationEligible(0x20a2cd0, 0xc0012e9180, 0x1d7a1b8, 0x1a) knative.dev/operator/pkg/reconciler/common/releases.go:123 +0x50 knative.dev/operator/pkg/reconciler/knativeserving.(Reconciler).ReconcileKind(0xc0007360e0, 0x2079b38, 0xc000923ec0, 0xc0012e9180, 0x0, 0x0) knative.dev/operator/pkg/reconciler/knativeserving/knativeserving.go:101 +0x309 knative.dev/operator/pkg/client/injection/reconciler/operator/v1alpha1/knativeserving.(reconcilerImpl).Reconcile(0xc000cb66e0, 0x2079b38, 0xc000923ec0, 0xc0009e8c60, 0x1f, 0xc000455858, 0x2079b38) knative.dev/operator/pkg/client/injection/reconciler/operator/v1alpha1/knativeserving/reconciler.go:246 +0x1272 knative.dev/pkg/controller.(Impl).processNextWorkItem(0xc000db0180, 0xc0008aa300) knative.dev/pkg@v0.0.0-20210622173328-dd0db4b05c80/controller/controller.go:531 +0x5e4 knative.dev/pkg/controller.(Impl).RunContext.func3(0xc000e6a010, 0xc000db0180) knative.dev/pkg@v0.0.0-20210622173328-dd0db4b05c80/controller/controller.go:468 +0x53 created by knative.dev/pkg/controller.(*Impl).RunContext knative.dev/pkg@v0.0.0-20210622173328-dd0db4b05c80/controller/controller.go:466 +0x1a5