Closed dependabot[bot] closed 1 month ago
/test ci/centos/mini-e2e/k8s-1.28
/test ci/centos/mini-e2e/k8s-1.28
/test ci/centos/mini-e2e/k8s-1.28
/test ci/centos/mini-e2e/k8s-1.28
/test ci/centos/mini-e2e/k8s-1.27
verifying that ceph-fuse recovery works for new pods
test is failing,
tested locally,
pod describe
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 21m default-scheduler Successfully assigned test/csi-cephfs-demo-depl-7d55fdc599-szfb5 to c1
Normal Pulled 21m kubelet Successfully pulled image "docker.io/library/nginx:latest" in 2.292636931s (2.292676293s including waiting)
Normal Pulled 21m kubelet Successfully pulled image "docker.io/library/nginx:latest" in 2.187885572s (2.187891578s including waiting)
Normal Pulled 20m kubelet Successfully pulled image "docker.io/library/nginx:latest" in 2.190849647s (2.190855193s including waiting)
Warning Failed 20m (x4 over 21m) kubelet Error: failed to start container "web-server": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/8c66d061-189a-4b20-bd45-e14d5b3ab2e3/volumes/kubernetes.io~csi/pvc-ad9c8917-8c50-456f-911b-37a6deb03f91/mount': mkdir /var/lib/kubelet/pods/8c66d061-189a-4b20-bd45-e14d5b3ab2e3/volumes/kubernetes.io~csi/pvc-ad9c8917-8c50-456f-911b-37a6deb03f91/mount: file exists
Normal Pulled 20m kubelet Successfully pulled image "docker.io/library/nginx:latest" in 2.387547365s (2.387572149s including waiting)
Normal Pulling 19m (x5 over 21m) kubelet Pulling image "docker.io/library/nginx:latest"
Normal Created 19m (x5 over 21m) kubelet Created container web-server
Normal Pulled 19m kubelet Successfully pulled image "docker.io/library/nginx:latest" in 2.334123775s (2.334129399s including waiting)
Warning BackOff 59s (x88 over 20m) kubelet Back-off restarting failed container web-server in pod csi-cephfs-demo-depl-7d55fdc599-szfb5_test(8c66d061-189a-4b20-bd45-e14d5b3ab2e3)
after killall -9 ceph-fuse
both staging and target path seems to be corrupted?
$ sudo ls -l /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/c57e487f74b05bade7ab81d4ed500cededd686dbcfab02b9cea5600d371b9c6f
ls: cannot access '/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/c57e487f74b05bade7ab81d4ed500cededd686dbcfab02b9cea5600d371b9c6f/globalmount': Transport endpoint is not connected
total 4
d????????? ? ? ? ? ? globalmount
-rw-r--r-- 1 root root 138 May 22 08:48 vol_data.json
$ sudo ls -l /var/lib/kubelet/pods/8c66d061-189a-4b20-bd45-e14d5b3ab2e3/volumes/kubernetes.io~csi/pvc-ad9c8917-8c50-456f-911b-37a6deb03f91
ls: cannot access '/var/lib/kubelet/pods/8c66d061-189a-4b20-bd45-e14d5b3ab2e3/volumes/kubernetes.io~csi/pvc-ad9c8917-8c50-456f-911b-37a6deb03f91/mount': Transport endpoint is not connected
total 4
d????????? ? ? ? ? ? mount
-rw-r--r-- 1 root root 330 May 22 08:50 vol_data.json
But stating path is returned as MOUNTED
and target path as NOT_MOUNTED
and fuse-recovery is not attempted.
I0522 08:50:44.714587 1 fuserecovery.go:92] ID: 21 Req-ID: 0001-0009-rook-ceph-0000000000000001-0b76a303-5d4a-4036-b152-b1feef05f8fe cephfs: staging target path /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/c57e487f74b05bade7ab81d4ed500cededd686dbcfab02b9cea5600d371b9c6f/globalmount is MOUNTED
I0522 08:50:44.714650 1 fuserecovery.go:98] ID: 21 Req-ID: 0001-0009-rook-ceph-0000000000000001-0b76a303-5d4a-4036-b152-b1feef05f8fe cephfs: target path /var/lib/kubelet/pods/8c66d061-189a-4b20-bd45-e14d5b3ab2e3/volumes/kubernetes.io~csi/pvc-ad9c8917-8c50-456f-911b-37a6deb03f91/mount is NOT_MOUNTED
using image from devel branch (which has k8s.io/mount-utils
v0.29.3)
and issue is gone, from logs you can see the stating path is returned as CORRUPTED
and target path as NOT_MOUNTED
and fuse-recovery is attempted and succeeded
I0522 09:56:47.612507 1 fuserecovery.go:92] ID: 16 Req-ID: 0001-0009-rook-ceph-0000000000000001-43fb5753-00c3-4afb-87b5-f4e255e10c8a cephfs: staging target path /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/8a78178247287601fcceabc0de70fa12ce5370938dad84960b5dc47c902bed77/globalmount is CORRUPTED
I0522 09:56:47.612525 1 fuserecovery.go:98] ID: 16 Req-ID: 0001-0009-rook-ceph-0000000000000001-43fb5753-00c3-4afb-87b5-f4e255e10c8a cephfs: target path /var/lib/kubelet/pods/bf47481a-9771-4927-9a4e-fe5284a90865/volumes/kubernetes.io~csi/pvc-b6afe4f7-72f2-4c3e-ab44-9770eb2de568/mount is NOT_MOUNTED
W0522 09:56:47.612563 1 fuserecovery.go:110] ID: 16 Req-ID: 0001-0009-rook-ceph-0000000000000001-43fb5753-00c3-4afb-87b5-f4e255e10c8a cephfs: mount problem detected when publishing a volume: /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/8a78178247287601fcceabc0de70fa12ce5370938dad84960b5dc47c902bed77/globalmount is CORRUPTED, /var/lib/kubelet/pods/bf47481a-9771-4927-9a4e-fe5284a90865/volumes/kubernetes.io~csi/pvc-b6afe4f7-72f2-4c3e-ab44-9770eb2de568/mount is NOT_MOUNTED; attempting recovery
I0522 09:56:47.622236 1 omap.go:89] ID: 16 Req-ID: 0001-0009-rook-ceph-0000000000000001-43fb5753-00c3-4afb-87b5-f4e255e10c8a got omap values: (pool="myfs-metadata", namespace="csi", name="csi.volume.43fb5753-00c3-4afb-87b5-f4e255e10c8a"): map[csi.imagename:csi-vol-43fb5753-00c3-4afb-87b5-f4e255e10c8a csi.volname:pvc-b6afe4f7-72f2-4c3e-ab44-9770eb2de568 csi.volume.owner:test]
I suspect issue with k8s.io/mount-utils
> v0.29.3. However, I don't see any similar open issue.
cc @Rakshith-R
replacing, k8s.io/mount-utils => k8s.io/mount-utils v0.29.3
/test ci/centos/mini-e2e/k8s-1.28
replacing,
k8s.io/mount-utils => k8s.io/mount-utils v0.29.3
this works!,
So, will add a TODO: update once fixed in mount-utils
and visit later? @Madhu-1
replacing,
k8s.io/mount-utils => k8s.io/mount-utils v0.29.3
this works!, So, will add a
TODO: update once fixed in mount-utils
and visit later? @Madhu-1
@iPraveenParihar Thank you, lets get it merged and we will revisit mount-utils, lets check what is changed in mount-utils release notes.
@mergifyio queue
queue
default
/test ci/centos/upgrade-tests-cephfs
/test ci/centos/upgrade-tests-rbd
/test ci/centos/k8s-e2e-external-storage/1.28
/test ci/centos/mini-e2e-helm/k8s-1.28
/test ci/centos/mini-e2e/k8s-1.28
/test ci/centos/k8s-e2e-external-storage/1.27
/test ci/centos/k8s-e2e-external-storage/1.29
/test ci/centos/mini-e2e-helm/k8s-1.27
/test ci/centos/mini-e2e-helm/k8s-1.29
/test ci/centos/mini-e2e/k8s-1.27
/test ci/centos/mini-e2e/k8s-1.29
/test ci/centos/k8s-e2e-external-storage/1.30
/test ci/centos/mini-e2e-helm/k8s-1.30
/test ci/centos/mini-e2e/k8s-1.30
few e2e failed at deploy artifats -
go: downloading go1.22.0 (linux/amd64)
go: github.com/ceph/ceph-csi/api in vendor/modules.txt requires go >= 1.22.2 (running go 1.22.0)
Packages librbd-devel librados-devel need to be installed
To build ceph-csi in a container: $ make containerized-build
go test -tags=reef,ceph_preview -mod=vendor -c ./e2e
go: github.com/ceph/ceph-csi/api in vendor/modules.txt requires go >= 1.22.2 (running go 1.22.0)
make: *** [Makefile:168: e2e.test] Error 1
make: *** [Makefile:205: containerized-build] Error 2
script returned exit code 2
and few are struct at reserve bare-metal machine
+ duffy client request-session pool=virt-ec2-t2-centos-8s-x86_64,quantity=1
Cannot get property 'nodes' on null object
and ci/centos/k8s-e2e-external-storage/1.27
got passed.
looks weird to me π€
/test ci/centos/mini-e2e-helm/k8s-1.28
go: downloading go1.22.0 (linux/amd64) go: github.com/ceph/ceph-csi/api in vendor/modules.txt requires go >= 1.22.2 (running go 1.22.0) Packages librbd-devel librados-devel need to be installed To build ceph-csi in a container: $ make containerized-build go test -tags=reef,ceph_preview -mod=vendor -c ./e2e go: github.com/ceph/ceph-csi/api in vendor/modules.txt requires go >= 1.22.2 (running go 1.22.0) make: [Makefile:168: e2e.test] Error 1 make: [Makefile:205: containerized-build] Error 2 script returned exit code 2
@iPraveenParihar can you please check on this one. https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/organizations/jenkins/upgrade-tests-rbd/detail/upgrade-tests-rbd/7974/pipeline
/test ci/centos/mini-e2e-helm/k8s-1.28
go: downloading go1.22.0 (linux/amd64) go: github.com/ceph/ceph-csi/api in vendor/modules.txt requires go >= 1.22.2 (running go 1.22.0) Packages librbd-devel librados-devel need to be installed To build ceph-csi in a container: $ make containerized-build go test -tags=reef,ceph_preview -mod=vendor -c ./e2e go: github.com/ceph/ceph-csi/api in vendor/modules.txt requires go >= 1.22.2 (running go 1.22.0) make: [Makefile:168: e2e.test] Error 1 make: [Makefile:205: containerized-build] Error 2 script returned exit code 2
@iPraveenParihar can you please check on this one. https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/blue/organizations/jenkins/upgrade-tests-rbd/detail/upgrade-tests-rbd/7974/pipeline
updates to vendor/modules.txt were missed during rebase conflicts. e2e previously failed now has passed. π
@Mergifyio queue
queue
Bumps the k8s-dependencies group with 1 update in the / directory: sigs.k8s.io/controller-runtime.
Updates
sigs.k8s.io/controller-runtime
from 0.17.3 to 0.18.2Release notes
Sourced from sigs.k8s.io/controller-runtime's releases.
... (truncated)
Commits
834905b
Merge pull request #2817 from k8s-infra-cherrypick-robot/cherry-pick-2813-to-...6396a49
Reintroduce AddMetricsExtraHandler on managercd2885e
Merge pull request #2809 from k8s-infra-cherrypick-robot/cherry-pick-2806-to-...06acde3
π correct kind source typed6feaae
Merge pull request #2808 from k8s-infra-cherrypick-robot/cherry-pick-2805-to-...87cae4c
bug: Cache: Fix label defaulting of byObject when namespaces are configureded81fa6
Merge pull request #2796 from alvaroaleman/markb35cd6b
:book: Mark Typed Handlers as experimentalb74908f
Merge pull request #2793 from kubernetes-sigs/dependabot/github_actions/actio...a9db208
Merge pull request #2794 from kubernetes-sigs/dependabot/github_actions/actio...You can trigger a rebase of this PR by commenting
@dependabot rebase
.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show