Closed halkeye closed 6 years ago
/area digitalocean
Sounds like a issue with the flexvolume driver. Could you check the kubelet
log?
journalctl -f -u kubelet.service
imported the external storage:
May 04 18:52:24 nemo-worker-0 kubelet-wrapper[896]: I0504 18:52:24.379746 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "digitalocean-provisioner-token-rglsq" (UniqueName: "kubernetes.io/secret/4876d7b1-4fcc-11e8-8fcf-7ac73da28f81-digitalocean-provisioner-token-rglsq") pod "digitalocean-provisioner-c4544597c-drx2g" (UID: "4876d7b1-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:24 nemo-worker-0 kubelet-wrapper[896]: I0504 18:52:24.887899 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/48ca6343-4fcc-11e8-8fcf-7ac73da28f81-flexvolume-mount") pod "digitalocean-flexplugin-deploy-4s5bc" (UID: "48ca6343-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:24 nemo-worker-0 kubelet-wrapper[896]: I0504 18:52:24.887964 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-kwc6k" (UniqueName: "kubernetes.io/secret/48ca6343-4fcc-11e8-8fcf-7ac73da28f81-default-token-kwc6k") pod "digitalocean-flexplugin-deploy-4s5bc" (UID: "48ca6343-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:25 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:25.679307 896 pod_container_deletor.go:77] Container "d7d6d7759b422239fb29c727f4d835ebe49d5b55badb289f4ae857a16deab3bb" not found in pod's containers
May 04 18:52:25 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:25.686110 896 pod_container_deletor.go:77] Container "861c49b8d0ec7d14a1116f7cbf53c90c78560d98b027247635ffaa4664aa1355" not found in pod's containers
May 04 18:52:26 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:26.159555 896 cadvisor_stats_provider.go:365] Partial failure issuing cadvisor.ContainerInfoV2: partial failures: ["/kubepods/pod48ca6343-4fcc-11e8-8fcf-7ac73da28f81/2b555799ec1d2961a916138abcec95b9259f0b1e96abdeb4b38ea20d74ae4704": RecentStats: unable to find data for container /kubepods/pod48ca6343-4fcc-11e8-8fcf-7ac73da28f81/2b555799ec1d2961a916138abcec95b9259f0b1e96abdeb4b38ea20d74ae4704]
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:31.336549 896 driver-call.go:251] Failed to unmarshal output for command: init, output: "", error: unexpected end of JSON input
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:31.337315 896 driver-call.go:144] FlexVolume: driver call failed: executable: /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean, args: [init], error: fork/exec /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean: no such file or directory, output: ""
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:31.337785 896 plugins.go:549] Error dynamically probing plugins: Error creating Flexvolume plugin from directory external-storage~digitalocean, skipping. Error: unexpected end of JSON input
When I check the directory manually on the worker afterwards:
core@nemo-worker-0 ~ $ ls /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
/var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
core@nemo-worker-0 ~ $ ls /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean -l
-rwxr-xr-x. 1 root root 6685080 May 4 18:52 /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
core@nemo-worker-0 ~ $ /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
{"status":"Failure","message":"open /var/lib/kubelet/volumeplugins/external-storage~digitalocean/do_token: permission denied"}
core@nemo-worker-0 ~ $ sudo /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
{"status":"Not supported"}
So i think its all installed. Donno if i need a sleep or something in my script between steps.
kubectl create -f external-storage/digitalocean/examples/pvc.yaml
persistentvolumeclaim "pv1" created
nothing in kubelet log
kubectl create -f external-storage/digitalocean/examples/pod-application.yaml
pod "busy-pod" created
Log:
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: W0504 18:56:44.885185 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:44.967200 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-rpx99" (UniqueName: "kubernetes.io/secret/e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81-default-token-rpx99") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:44.967531 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:44.967781 896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:45.467727125 +0000 UTC m=+584.457961103 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:45 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:45.469657 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:45 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:45.470144 896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:46.470106321 +0000 UTC m=+585.460340302 (durationBeforeRetry 1s). Error: "Volume has not been added to the
list of VolumesInUse in the node's volume status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:46 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:46.472891 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:46 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:46.480464 896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:48.480429407 +0000 UTC m=+587.470663386 (durationBeforeRetry 2s). Error: "Volume not attached according to
node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:48 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:48.580513 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:48 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:48.588212 896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:52.588146342 +0000 UTC m=+591.578380330 (durationBeforeRetry 4s). Error: "Volume not attached according to
node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:52 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:52.595873 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:52 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:52.604225 896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:57:00.604188474 +0000 UTC m=+599.594422460 (durationBeforeRetry 8s). Error: "Volume not attached according to
node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:57:00 nemo-worker-0 kubelet-wrapper[896]: I0504 18:57:00.634367 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:57:00 nemo-worker-0 kubelet-wrapper[896]: E0504 18:57:00.644299 896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:57:16.644254763 +0000 UTC m=+615.634488742 (durationBeforeRetry 16s). Error: "Volume not attached according to node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
On the controller/master
May 04 18:51:24 nemo-controller-0 kubelet-wrapper[1212]: W0504 18:51:24.665206 1212 pod_container_deletor.go:77] Container "f642ab0e861dd5278f8c794d674e174d67e685d7ccb3dab6a028f331f5be87f2" not found in pod's containers
May 04 18:52:24 nemo-controller-0 kubelet-wrapper[1212]: I0504 18:52:24.938463 1212 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/48cc98db-4fcc-11e8-8fcf-7ac73da28f81-flexvolume-mount") pod "digitalocean-flexplugin-deploy-7xtvt" (UID: "48cc98db-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:24 nemo-controller-0 kubelet-wrapper[1212]: I0504 18:52:24.940814 1212 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-kwc6k" (UniqueName: "kubernetes.io/secret/48cc98db-4fcc-11e8-8fcf-7ac73da28f81-default-token-kwc6k") pod "digitalocean-flexplugin-deploy-7xtvt" (UID: "48cc98db-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:28 nemo-controller-0 kubelet-wrapper[1212]: W0504 18:52:28.447375 1212 pod_container_deletor.go:77] Container "d9d03cb6a092428770450542a39e3fbd283c56f8a6927f0d082af594f68ef870" not found in pod's containers
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:31.337315 896 driver-call.go:144] FlexVolume: driver call failed: executable: /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean, args: [init], error: fork/exec /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean: no such file or directory, output: ""
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:31.337785 896 plugins.go:549] Error dynamically probing plugins: Error creating Flexvolume plugin from directory external-storage~digitalocean, skipping. Error: unexpected end of JSON input
Do you have other flexvolume installed? https://github.com/kubernetes/kubernetes/issues/63088
Does the attach success? (check kube-controller-manager log)
/var/lib/kubelet/volumeplugins/
/var/lib/kubelet/volumeplugins/external-storage~digitalocean
/var/lib/kubelet/volumeplugins/external-storage~digitalocean/do_token
/var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
nope just the one, its a completely fresh install
I0504 18:51:28.447995 1 controller_utils.go:1026] Caches are synced for resource quota controller
I0504 18:51:28.457612 1 controller_utils.go:1026] Caches are synced for disruption controller
I0504 18:51:28.458250 1 disruption.go:296] Sending events to api server.
I0504 18:51:28.459470 1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0504 18:51:28.459557 1 garbagecollector.go:144] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0504 18:51:28.502171 1 controller_utils.go:1026] Caches are synced for deployment controller
I0504 18:51:28.506517 1 controller_utils.go:1026] Caches are synced for ReplicaSet controller
I0504 18:51:30.140697 1 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
I0504 18:51:30.241580 1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0504 18:52:24.194569 1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"4868cd97-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"extensions", ResourceVersion:"821", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-c4544597c to 2
I0504 18:52:24.233714 1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-c4544597c", UID:"486cdea7-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"extensions", ResourceVersion:"822", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-c4544597c-99vsp
I0504 18:52:24.263499 1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-c4544597c", UID:"486cdea7-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"extensions", ResourceVersion:"822", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-c4544597c-drx2g
I0504 18:52:24.805025 1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"48c3acdf-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"apps", ResourceVersion:"841", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-4s5bc
I0504 18:52:24.818970 1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"48c3acdf-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"apps", ResourceVersion:"841", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-7xtvt
I0504 18:56:06.015680 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1149", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
E0504 18:56:06.035231 1 pvc_protection_controller.go:134] PVC default/pv1 failed with : Operation cannot be fulfilled on persistentvolumeclaims "pv1": the object has been modified; please apply your changes to the latest version and try again
I0504 18:56:06.054338 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1151", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I0504 18:56:06.120380 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1154", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I0504 18:56:08.323021 1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1159", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
E0504 18:56:08.714231 1 pv_protection_controller.go:112] PV pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81 failed with : Operation cannot be fulfilled on persistentvolumes "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81": the object has been modified; please apply your changes to the latest version and try again
…/github.com/halkeye/terraform-digitalocean kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pv1 Bound pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81 1Gi RWO default 55m
Checking the DO website, it does get created, just not attached
I just spun up a cluster in tor1 and everything is working as expected, the only required change was adding DIGITALOCEAN_ACCESS_TOKEN
to kube-controller-manager
. I'm wondering if it could be a bug in v1.10.0
? The cluster I spun up was v1.10.2
.
I did run kubectl -n kube-system delete pod -l app=digitalocean-flexplugin-deploy
for other reason, could you try the same?
I'm going to spin up a new 1.10.2 cluster.
I deleted the deploy pod, and eventually tried re-created the example pod but no luck.
I just spun up a cluster in tor1 and everything is working as expected, the only required change was adding DIGITALOCEAN_ACCESS_TOKEN to kube-controller-manager.
Should I run
kubectl --namespace=kube-system patch deployment kube-controller-manager -p '{"spec": { "template": { "spec": { "containers": [{ "name": "kube-controller-manager", "env": [ { "name": "DIGITALOCEAN_ACCESS_TOKEN", "valueFrom": { "secretKeyRef": { "key": "access-token", "name": "digitalocean" } } } ] ] } } } '
(I hope my brackets are right, i had a more complicated version that added volume mounts as well)
Should I run
It looks correct. I just did kubectl -n kube-system edit deploy kube-controller-manager
and then copy-paste (under containers
, remember to remove -
from - command
):
https://github.com/kubernetes-incubator/external-storage/blob/c01d6a39bab6ecf6c7724d112d695ce70e3314bc/digitalocean/manifests/digitalocean-flexplugin-deploy.yaml#L19-L24
new 1.10.2 cluster, running storage.sh (the init script posted above) gave me the same errors about skipping th flex plugin
Running kubectl -n kube-system delete pod -l app=digitalocean-flexplugin-deploy
gave me:
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:18.547799 896 pod_container_deletor.go:77] Container "54d763f792cbf755ea497b040332d73a5f3f1357db936565dea2ee518bf27072" not found in pod's containers
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.741992 896 reconciler.go:181] operationExecutor.UnmountVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742333 896 reconciler.go:181] operationExecutor.UnmountVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742573 896 operation_generator.go:686] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount" (OuterVolumeSpecName: "flexvolume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerVolumeSpecName "flexvolume-mount". PluginName "kubernetes.io/host-path", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.758271 896 operation_generator.go:686] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4" (OuterVolumeSpecName: "default-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerVolumeSpecName "default-token-5jmh4". PluginName "kubernetes.io/secret", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.842740 896 reconciler.go:285] Volume detached for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") on node "10.137.80.240" DevicePath ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.843489 896 reconciler.go:285] Volume detached for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") on node "10.137.80.240" DevicePath ""
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.745844 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/c3720d64-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.746585 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/c3720d64-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:20 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:20.619968 896 pod_container_deletor.go:77] Container "0e6c81a7676db13e7e3a69385d0b52b4f1b5d8c885e0e091945020e9161156d3" not found in pod's containers
tried create pvc and pod-application examples:
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:18.547799 896 pod_container_deletor.go:77] Container "54d763f792cbf755ea497b040332d73a5f3f1357db936565dea2ee518bf27072" not found in pod's containers May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.741992 896 reconciler.go:181] operationExecutor.Unmou$
tVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-defa
ult-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742333 896 reconciler.go:181] operationExecutor.Unmoun
tVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flex
volume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742573 896 operation_generator.go:686] UnmountVolume.T
earDown succeeded for volume "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount" (OuterVolumeSpec
Name: "flexvolume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerVolu
meSpecName "flexvolume-mount". PluginName "kubernetes.io/host-path", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.758271 896 operation_generator.go:686] UnmountVolume.T
earDown succeeded for volume "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4" (OuterVolumeSpec
Name: "default-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerV
olumeSpecName "default-token-5jmh4". PluginName "kubernetes.io/secret", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.842740 896 reconciler.go:285] Volume detached for volu
me "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") on node "
10.137.80.240" DevicePath ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.843489 896 reconciler.go:285] Volume detached for volu
me "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") on nod
e "10.137.80.240" DevicePath ""
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.745844 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/c3720d64-4fe2-11e8-9052-
4ed61f7ca20e-default-token-5jmh4") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.746585 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/c3720d64-4fe2-11e8-9052-
4ed61f7ca20e-flexvolume-mount") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:20 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:20.619968 896 pod_container_deletor.go:77] Container "0e6
c81a7676db13e7e3a69385d0b52b4f1b5d8c885e0e091945020e9161156d3" not found in pod's containers
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: W0504 21:34:41.477514 896 plugin-defaults.go:32] flexVolume driver ex
ternal-storage/digitalocean: using default GetVolumeName for volume pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: I0504 21:34:41.569756 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "f43a8b57-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: I0504 21:34:41.570077 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "default-token-k8r6j" (UniqueName: "kubernetes.io/secret/f43a8b57-4fe2-11e8-9052-
4ed61f7ca20e-default-token-k8r6j") pod "busy-pod" (UID: "f43a8b57-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: E0504 21:34:41.570365 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
2018-05-04 21:34:42.070321211 +0000 UTC m=+906.570794583 (durationBeforeRetry 500ms). Error: "Volume has not been added to t
he list of VolumesInUse in the node's volume status for volume \"pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\" (UniqueName: \"fl
exvolume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"f43a8b57-4fe2-11e
8-9052-4ed61f7ca20e\") "
May 04 21:34:42 nemo-worker-0 kubelet-wrapper[896]: I0504 21:34:42.071752 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "f43a8b57-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:34:42 nemo-worker-0 kubelet-wrapper[896]: E0504 21:34:42.072156 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
2018-05-04 21:34:43.072108529 +0000 UTC m=+907.572581903 (durationBeforeRetry 1s). Error: "Volume has not been added to the
list of VolumesInUse in the node's volume status for volume \"pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexv
olume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"f43a8b57-4fe2-11e8-9
052-4ed61f7ca20e\") "
kubectl --namespace=kube-system patch deployment kube-controller-manager -p '{"spec": { "template": { "spec": { "containers": [{ "name": "kube-controller-manager", "env": [ { "name": "DIGITALOCEAN_ACCESS_TOKEN", "valueFrom": { "secretKeyRef": { "key": "access-token", "name": "digitalocean" } } } ] } ] } } } }'
May 04 21:40:11 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:11.784575 896 kubelet.go:1635] Unable to mount volumes fo
r pod "busy-pod_default(6fc9a468-4fe3-11e8-9052-4ed61f7ca20e)": timeout expired waiting for volumes to attach or mount for po
d "default"/"busy-pod". list of unmounted volumes=[vol1 default-token-k8r6j]. list of unattached volumes=[vol1 default-token-k8r6j]; skipping pod
May 04 21:40:11 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:11.785467 896 pod_workers.go:186] Error syncing pod 6fc9a468-4fe3-11e8-9052-4ed61f7ca20e ("busy-pod_default(6fc9a468-4fe3-11e8-9052-4ed61f7ca20e)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"busy-pod". list of unmounted volumes=[vol1 default-token-k8r6j]. list of unattached volumes=[vol1 default-token-k8r6j]
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:38.540754 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:38.636278 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "default-token-k8r6j" (UniqueName: "kubernetes.io/secret/b692ff62-4fe3-11e8-9052-4ed61f7ca20e-default-token-k8r6j") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:38.636659 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:38.636906 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
2018-05-04 21:40:39.136874754 +0000 UTC m=+1263.637348133 (durationBeforeRetry 500ms). Error: "Volume has not been added to
the list of VolumesInUse in the node's volume status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"f
lexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:39 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:39.139488 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:39 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:39.139865 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
2018-05-04 21:40:40.139798148 +0000 UTC m=+1264.640271508 (durationBeforeRetry 1s). Error: "Volume has not been added to the
list of VolumesInUse in the node's volume status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-
9052-4ed61f7ca20e\") "
May 04 21:40:40 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:40.144406 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:40 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:40.145222 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
2018-05-04 21:40:42.145181337 +0000 UTC m=+1266.645654698 (durationBeforeRetry 2s). Error: "Volume has not been added to the
list of VolumesInUse in the node's volume status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flex
volume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:42 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:42.152734 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:42 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:42.156660 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until 2018-05-04 21:40:46.156620298 +0000 UTC m=+1270.657093658 (durationBeforeRetry 4s). Error: "Volume not attached according to node status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexvolume-external-storage/digitalocean/
pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:46 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:46.170717 896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:46 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:46.174931 896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until 2018-05-04 21:40:54.1748977 +0000 UTC m=+1278.675371072 (durationBeforeRetry 8s). Error: "Volume not attached according to n
ode status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.203458 896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.207885 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.208110 896 operation_generator.go:1168] Controller attach succeeded for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") device path: "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.304524 896 operation_generator.go:486] MountVolume.WaitForAttach entering for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") DevicePath "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.308416 896 attacher-defaults.go:39] flexVolume driver external-storage/digitalocean: using default WaitForAttach for volume 0x323b480, device /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.308590 896 operation_generator.go:495] MountVolume.WaitForAttach succeeded for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") DevicePath "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.308740 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.311338 896 attacher-defaults.go:50] flexVolume driver external-storage/digitalocean: using default MountDevice for volume 0x323b480, device /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e, deviceMountPath /var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.344237 896 mount_linux.go:485] `fsck` error fsck from util-linux 2.29.2
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: fsck.ext2: Bad magic number in super-block while trying to open /dev/sda
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: /dev/sda:
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: The superblock could not be read or does not describe a valid ext2/ext3/ext4
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: filesystem. If the device is valid and it really contains an ext2/ext3/ext4
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: filesystem (and not swap or ufs or something else), then the superblock
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: is corrupt, and you might try running e2fsck with an alternate superblock:
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: e2fsck -b 8193 <device>
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: or
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: e2fsck -b 32768 <device>
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:54.353755 896 mount_linux.go:148] Mount failed: exit status 32
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: Mounting command: mount
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: Mounting arguments: -o rw,defaults /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e /var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: Output: mount: wrong fs type, bad option, bad superblock on /dev/sda,
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: missing codepage or helper program, or other error
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: In some cases useful info is found in syslog - try
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: dmesg | tail or so.
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.362257 896 mount_linux.go:516] Disk "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" appears to be unformatted, attempting to format as type: "ext4" with options: [-F /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e]
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.571853 896 mount_linux.go:520] Disk successfully formatted (mkfs): ext4 - /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e /var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.588515 896 operation_generator.go:514] MountVolume.MountDevice succeeded for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") device mount path "/var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.602221 896 mounter-defaults.go:30] flexVolume driver external-storage/digitalocean: using default SetUpAt to /var/lib/kubelet/pods/b692ff62-4fe3-11e8-9052-4ed61f7ca20e/volumes/external-storage~digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.602414 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:56 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:56.325687 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:57 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:57.343737 896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
so I think it works after adding the secrets to the pod controller.
The pod started up \o/
…/github.com/halkeye/terraform-digitalocean kubectl exec busy-pod -- df -h /mnt/pv-1
Filesystem Size Used Available Use% Mounted on
/dev/sda 975.9M 2.5M 906.2M 0% /mnt/pv-1
…/github.com/halkeye/terraform-digitalocean kubectl exec busy-pod -- ls -l /mnt/pv-1
total 16
drwx------ 2 root root 16384 May 4 21:40 lost+found
As far as I can tell this is working.
Does the kubectl patch step need to be added to docs somewhere?
Does the kubectl patch step need to be added to docs somewhere?
I have a tracking issue for required changes for different setups, but haven't got time to update the README. https://github.com/kubernetes-incubator/external-storage/issues/531
Ideally people should just run kubectl apply -f manifests/bootkube.yaml
and it should work with bootkube
.
is that manifests/bootkube.yaml file somewhere? (i tried but didn't see it in this repo)
So far I have a working repeatable cluster, which is super exciting, I can use the kubectl patch thing, but but i rather script it the right way.
is that manifests/bootkube.yaml file somewhere? (i tried but didn't see it in this repo)
Sorry, it was just a idea, but I suppose the following could work:
spec:
template:
spec:
containers:
- env:
- name: DIGITALOCEAN_ACCESS_TOKEN
valueFrom:
secretKeyRef:
key: access-token
name: digitalocean
name: kube-controller-manager
Save as bootkube-patch.yaml
and run kubectl -n kube-system patch deploy kube-controller-manager -p "$(<bootkube-patch.yaml)"
@halkeye would you mind closing this issue?
II'm not sure what info is useful for a bug report so I'm going to include a whole lot. I've setup a kubernetes cluster using typhoon - https://typhoon.psdn.io/cl/digital-ocean/
I was eventually able to get this working by --enable-attach-detach-controller on kubelet, then modifying kube-controller-manager to add the DIGITALOCEAN_ACCESS_TOKEN env variable, and adding the plugins directory to volume mount but I don't feel like that's the intention of this project. I can do this again if info is needed, but I've already reset my cluster a few times since then.
providers.tf:
Setting up the external storage:
Setup the pvc and application
kubectl get events