Open mattiashem opened 5 months ago
[core@bastion storage]$ kubectl logs -f hcloud-csi-node-jcsdp -n kube-system -c hcloud-csi-driver level=info ts=2024-03-15T13:33:11.303902051Z msg="Fetched data from metadata service" id=41964065 location=nbg1 ^C [core@bastion storage]$ kubectl logs -f hcloud-csi-node-p6jj2 -n kube-system -c hcloud-csi-driver level=info ts=2024-03-15T13:33:09.161029164Z msg="Fetched data from metadata service" id=41963957 location=nbg1 level=error ts=2024-03-15T13:34:49.615777847Z component=grpc-server msg="handler failed" err="rpc error: code = Internal desc = failed to publish volume: unable to detect existing disk format of /dev/disk/by-id/scsi-0HC_Volume_100455302: disk /dev/disk/by-id/scsi-0HC_Volume_100455302 propably contains partitions"
I'm also running into this issue.
My manifest:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: task-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
---
apiVersion: v1
kind: Pod
metadata:
name: task-pv-pod
spec:
volumes:
- name: task-pv-storage
persistentVolumeClaim:
claimName: task-pv-claim
containers:
- name: task-pv-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: task-pv-storage
The error:
MountVolume.SetUp failed for volume "pvc-15b2696c-eaf0-4e79-97de-71903a597ebb" : rpc error: code = Internal desc = failed to publish volume: unable to detect existing disk format of /dev/disk/by-id/scsi-0HC_Volume_100612315: disk /dev/disk/by-id/scsi-0HC_Volume_100612315 propably contains partitions
I am unable to reproduce this with our dev setup and the Getting Started guide.
Some questions that might help to pinpoint the issue:
StorageClass
?I am still unable to reproduce this. What I have done:
Setup a Cluster with Talos 1.7.0 using their docs I used Packer to create the snapshots. To generate the Talos Config I used this command instead, to get below the 32kb limit on userdata:
talosctl gen config talos-k8s-hcloud-tutorial https://$LOAD_BALANCER_IP:6443 --kubernetes-version 1.27.4 --with-docs=false --with-examples=false
Followed the Steps from our Getting Started on Kubernetes guide, installing the Helm chart and applying my-csi-app
.
And my-csi-app
successfully started and the volume is mounted.
Do you have some steps for me to reproduce this?
It looks like I get the error from the API, or it bounds the volumes wrong. I only have the problem with an older cluster. If I create a new cluster (even on old talos like 1.6), it works.
Is there some API in the client ore how the volumes that are attached are different ? Have 2 cluster now on the same project work on one but not on the other ?
I have given up and move
Same issue
failed to publish volume: │ │ unable to detect existing disk format of /dev/disk/by-id/scsi-0HC_Volume_xxxxxxxx: disk /dev/disk/by-id/scsi-0 │ │ HC_Volume_xxxxxxxx propably contains partitions
Some more questions:
LUKS
with the csi-driver?kube-hetzner
and only install it manually?uname -a
blkid -p -o export /dev/disk/by-id/scsi-0HC_Volume_xxxxxxxx
On Talos without a regular shell you should be able to exec into the csi-driver Node Pod to execute these commands.
TL;DR
Warning FailedMount 38s (x3536 over 4d23h) kubelet MountVolume.SetUp failed for volume "pvc-a1e2b216-bd1f-4e3f-b54f-ebc8ce7760bd" : rpc error: code = Internal desc = failed to publish volume: unable to detect existing disk format of /dev/disk/by-id/scsi-0HC_Volume_100455302: disk /dev/disk/by-id/scsi-0HC_Volume_100455302 propably contains partitions
Expected behavior
Have 3 cluster and it workes fine and mount the files in the other clusters
Observed behavior
Warning FailedMount 38s (x3536 over 4d23h) kubelet MountVolume.SetUp failed for volume "pvc-a1e2b216-bd1f-4e3f-b54f-ebc8ce7760bd" : rpc error: code = Internal desc = failed to publish volume: unable to detect existing disk format of /dev/disk/by-id/scsi-0HC_Volume_100455302: disk /dev/disk/by-id/scsi-0HC_Volume_100455302 propably contains partitions
Minimal working example
Used the install instructions from the guide. applied the PVC
Log output
Additional information
It looks like the provison work and in the cloud console I can see the disk and it looks as it has no filesystem