search

kubernetes-retired / external-storage

[EOL] External storage plugins, provisioners, and helper libraries
Apache License 2.0
2.69k stars 1.6k forks source link

timeout expired waiting for volumes to attach or mount for pod #746

Closed halkeye closed 6 years ago

halkeye commented 6 years ago

II'm not sure what info is useful for a bug report so I'm going to include a whole lot. I've setup a kubernetes cluster using typhoon - https://typhoon.psdn.io/cl/digital-ocean/

I was eventually able to get this working by --enable-attach-detach-controller on kubelet, then modifying kube-controller-manager to add the DIGITALOCEAN_ACCESS_TOKEN env variable, and adding the plugins directory to volume mount but I don't feel like that's the intention of this project. I can do this again if info is needed, but I've already reset my cluster a few times since then.

providers.tf:

provider "digitalocean" {
  version = "0.1.3"
  token = "${chomp(file("~/.config/digital-ocean/token"))}"
  alias = "default"
}

provider "local" {
  version = "~> 1.0"
  alias = "default"
}

provider "null" {
  version = "~> 1.0"
  alias = "default"
}

provider "template" {
  version = "~> 1.0"
  alias = "default"
}

provider "tls" {
  version = "~> 1.0"
  alias = "default"
}

module "digital-ocean-nemo" {
  source = "git::https://github.com/poseidon/typhoon//digital-ocean/container-linux/kubernetes?ref=v1.10.0"

  providers = {
    digitalocean = "digitalocean.default"
    local = "local.default"
    null = "null.default"
    template = "template.default"
    tls = "tls.default"
  }

  cluster_name     = "nemo"
  region           = "tor1"
  dns_zone         = "do.gavinmogan.com"

  # image            = "coreos-stable"
  # controller_count = 1
  # controller_type  = "s-2vcpu-2gb"
  worker_count     = 2
  worker_type      = "s-1vcpu-1gb"
  ssh_fingerprints = ["5d:e5:24:00:13:90:ac:bf:94:5e:21:f5:23:36:0f:a8"]

  # output assets dir
  asset_dir = "/home/halkeye/.secrets/clusters/nemo"
}

Setting up the external storage:

set -e
export KUBECONFIG=$HOME/.secrets/clusters/nemo/auth/kubeconfig

test -d external-storage || git clone git@github.com:kubernetes-incubator/external-storage.git

cat external-storage/digitalocean/manifests/digitalocean-secret.yaml | perl -pi -e "s{<DigitalOcean token>}{$(python -c "import base64; print base64.b64encode(open('$HOME/.config/digital-ocean/token', 'r').read().strip().replace('\n', ''))")}" | kubectl create -f -
kubectl create -f external-storage/digitalocean/manifests/rbac
cat external-storage/digitalocean/manifests/digitalocean-provisioner.yaml | kubectl create -f -
cat external-storage/digitalocean/manifests/digitalocean-flexplugin-deploy.yaml | perl -pi -e "s{<flex volume plugin dir, default: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/>}{/var/lib/kubelet/volumeplugins/}" | kubectl create -f -
cat external-storage/digitalocean/manifests/sc.yaml | perl -pi -e "s{<digitalocean zone: ex fra1>}{tor1}" | kubectl create -f  -

# as per https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/ set the default storage class
kubectl patch storageclass default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Setup the pvc and application

kubectl create -f external-storage/digitalocean/examples/pvc.yaml
# wait for a while till everything gets connected/status == bound/etc
kubectl create -f external-storage/digitalocean/examples/pod-application.yaml

kubectl get events

14m         14m          1         busy-pod.152a22516843a6cb         Pod                                 Normal    SuccessfulMountVolume     kubelet, 10.137.64.244                                                                                MountVolume.SetUp succeeded for volume "default-token-mz84p"
54s         12m          6         busy-pod.152a226dfaeda75a         Pod                                 Warning   FailedMount               kubelet, 10.137.64.244                                                                                Unable to mount volumes for pod "busy-pod_default(68d5e62a-4c3e-11e8-aa21-42d9127af3ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"busy-pod". list of unmounted volumes=[vol1]. list of unattached volumes=[vol1 default-token-mz84p]
15m         15m          4         pv1.152a223c0288a6a4              PersistentVolumeClaim               Normal    ExternalProvisioning      persistentvolume-controller                                                                           waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
15m         15m          1         pv1.152a223c0c3160b5              PersistentVolumeClaim               Normal    Provisioning              external/digitalocean digitalocean-provisioner-c4544597c-kd29z 148c48e9-4c3e-11e8-85df-0a580a020002   External provisioner is provisioning volume for claim "default/pv1"
15m         15m          1         pv1.152a223c8f3b55ce              PersistentVolumeClaim               Normal    ProvisioningSucceeded     external/digitalocean digitalocean-provisioner-c4544597c-kd29z 148c48e9-4c3e-11e8-85df-0a580a020002   Successfully provisioned volume pvc-2a29e0e1-4c3e-11e8-aa21-42d9127af3ff
klausenbusk commented 6 years ago

/area digitalocean

Sounds like a issue with the flexvolume driver. Could you check the kubelet log?

halkeye commented 6 years ago

journalctl -f -u kubelet.service

imported the external storage:

May 04 18:52:24 nemo-worker-0 kubelet-wrapper[896]: I0504 18:52:24.379746     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "digitalocean-provisioner-token-rglsq" (UniqueName: "kubernetes.io/secret/4876d7b1-4fcc-11e8-8fcf-7ac73da28f81-digitalocean-provisioner-token-rglsq") pod "digitalocean-provisioner-c4544597c-drx2g" (UID: "4876d7b1-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:24 nemo-worker-0 kubelet-wrapper[896]: I0504 18:52:24.887899     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/48ca6343-4fcc-11e8-8fcf-7ac73da28f81-flexvolume-mount") pod "digitalocean-flexplugin-deploy-4s5bc" (UID: "48ca6343-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:24 nemo-worker-0 kubelet-wrapper[896]: I0504 18:52:24.887964     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-kwc6k" (UniqueName: "kubernetes.io/secret/48ca6343-4fcc-11e8-8fcf-7ac73da28f81-default-token-kwc6k") pod "digitalocean-flexplugin-deploy-4s5bc" (UID: "48ca6343-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:25 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:25.679307     896 pod_container_deletor.go:77] Container "d7d6d7759b422239fb29c727f4d835ebe49d5b55badb289f4ae857a16deab3bb" not found in pod's containers
May 04 18:52:25 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:25.686110     896 pod_container_deletor.go:77] Container "861c49b8d0ec7d14a1116f7cbf53c90c78560d98b027247635ffaa4664aa1355" not found in pod's containers
May 04 18:52:26 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:26.159555     896 cadvisor_stats_provider.go:365] Partial failure issuing cadvisor.ContainerInfoV2: partial failures: ["/kubepods/pod48ca6343-4fcc-11e8-8fcf-7ac73da28f81/2b555799ec1d2961a916138abcec95b9259f0b1e96abdeb4b38ea20d74ae4704": RecentStats: unable to find data for container /kubepods/pod48ca6343-4fcc-11e8-8fcf-7ac73da28f81/2b555799ec1d2961a916138abcec95b9259f0b1e96abdeb4b38ea20d74ae4704]
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:31.336549     896 driver-call.go:251] Failed to unmarshal output for command: init, output: "", error: unexpected end of JSON input
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:31.337315     896 driver-call.go:144] FlexVolume: driver call failed: executable: /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean, args: [init], error: fork/exec /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean: no such file or directory, output: ""
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:31.337785     896 plugins.go:549] Error dynamically probing plugins: Error creating Flexvolume plugin from directory external-storage~digitalocean, skipping. Error: unexpected end of JSON input

When I check the directory manually on the worker afterwards:

core@nemo-worker-0 ~ $ ls /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
/var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
core@nemo-worker-0 ~ $ ls /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean -l
-rwxr-xr-x. 1 root root 6685080 May  4 18:52 /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
core@nemo-worker-0 ~ $ /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
{"status":"Failure","message":"open /var/lib/kubelet/volumeplugins/external-storage~digitalocean/do_token: permission denied"}
core@nemo-worker-0 ~ $ sudo /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean
{"status":"Not supported"}

So i think its all installed. Donno if i need a sleep or something in my script between steps.

kubectl create -f external-storage/digitalocean/examples/pvc.yaml
persistentvolumeclaim "pv1" created

nothing in kubelet log

kubectl create -f external-storage/digitalocean/examples/pod-application.yaml
pod "busy-pod" created

Log:

May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: W0504 18:56:44.885185     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:44.967200     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-rpx99" (UniqueName: "kubernetes.io/secret/e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81-default-token-rpx99") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:44.967531     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:44 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:44.967781     896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:45.467727125 +0000 UTC m=+584.457961103 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:45 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:45.469657     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:45 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:45.470144     896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:46.470106321 +0000 UTC m=+585.460340302 (durationBeforeRetry 1s). Error: "Volume has not been added to the
list of VolumesInUse in the node's volume status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:46 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:46.472891     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:46 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:46.480464     896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:48.480429407 +0000 UTC m=+587.470663386 (durationBeforeRetry 2s). Error: "Volume not attached according to
node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:48 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:48.580513     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:48 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:48.588212     896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:56:52.588146342 +0000 UTC m=+591.578380330 (durationBeforeRetry 4s). Error: "Volume not attached according to
node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:56:52 nemo-worker-0 kubelet-wrapper[896]: I0504 18:56:52.595873     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:56:52 nemo-worker-0 kubelet-wrapper[896]: E0504 18:56:52.604225     896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:57:00.604188474 +0000 UTC m=+599.594422460 (durationBeforeRetry 8s). Error: "Volume not attached according to
node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
May 04 18:57:00 nemo-worker-0 kubelet-wrapper[896]: I0504 18:57:00.634367     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81") pod "busy-pod" (UID: "e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:57:00 nemo-worker-0 kubelet-wrapper[896]: E0504 18:57:00.644299     896 nestedpendingoperations.go:267] Operation for "\"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\"" failed. No retries permitted until 2018-05-04 18:57:16.644254763 +0000 UTC m=+615.634488742 (durationBeforeRetry 16s). Error: "Volume not attached according to node status for volume \"pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81\") pod \"busy-pod\" (UID: \"e3bfc4bf-4fcc-11e8-8fcf-7ac73da28f81\") "
halkeye commented 6 years ago

On the controller/master

May 04 18:51:24 nemo-controller-0 kubelet-wrapper[1212]: W0504 18:51:24.665206    1212 pod_container_deletor.go:77] Container "f642ab0e861dd5278f8c794d674e174d67e685d7ccb3dab6a028f331f5be87f2" not found in pod's containers
May 04 18:52:24 nemo-controller-0 kubelet-wrapper[1212]: I0504 18:52:24.938463    1212 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/48cc98db-4fcc-11e8-8fcf-7ac73da28f81-flexvolume-mount") pod "digitalocean-flexplugin-deploy-7xtvt" (UID: "48cc98db-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:24 nemo-controller-0 kubelet-wrapper[1212]: I0504 18:52:24.940814    1212 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-kwc6k" (UniqueName: "kubernetes.io/secret/48cc98db-4fcc-11e8-8fcf-7ac73da28f81-default-token-kwc6k") pod "digitalocean-flexplugin-deploy-7xtvt" (UID: "48cc98db-4fcc-11e8-8fcf-7ac73da28f81")
May 04 18:52:28 nemo-controller-0 kubelet-wrapper[1212]: W0504 18:52:28.447375    1212 pod_container_deletor.go:77] Container "d9d03cb6a092428770450542a39e3fbd283c56f8a6927f0d082af594f68ef870" not found in pod's containers
klausenbusk commented 6 years ago 
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: W0504 18:52:31.337315     896 driver-call.go:144] FlexVolume: driver call failed: executable: /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean, args: [init], error: fork/exec /var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean: no such file or directory, output: ""
May 04 18:52:31 nemo-worker-0 kubelet-wrapper[896]: E0504 18:52:31.337785     896 plugins.go:549] Error dynamically probing plugins: Error creating Flexvolume plugin from directory external-storage~digitalocean, skipping. Error: unexpected end of JSON input

Do you have other flexvolume installed? https://github.com/kubernetes/kubernetes/issues/63088

Does the attach success? (check kube-controller-manager log)

halkeye commented 6 years ago 
/var/lib/kubelet/volumeplugins/
/var/lib/kubelet/volumeplugins/external-storage~digitalocean
/var/lib/kubelet/volumeplugins/external-storage~digitalocean/do_token
/var/lib/kubelet/volumeplugins/external-storage~digitalocean/digitalocean

nope just the one, its a completely fresh install

I0504 18:51:28.447995       1 controller_utils.go:1026] Caches are synced for resource quota controller
I0504 18:51:28.457612       1 controller_utils.go:1026] Caches are synced for disruption controller
I0504 18:51:28.458250       1 disruption.go:296] Sending events to api server.
I0504 18:51:28.459470       1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0504 18:51:28.459557       1 garbagecollector.go:144] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0504 18:51:28.502171       1 controller_utils.go:1026] Caches are synced for deployment controller
I0504 18:51:28.506517       1 controller_utils.go:1026] Caches are synced for ReplicaSet controller
I0504 18:51:30.140697       1 controller_utils.go:1019] Waiting for caches to sync for garbage collector controller
I0504 18:51:30.241580       1 controller_utils.go:1026] Caches are synced for garbage collector controller
I0504 18:52:24.194569       1 event.go:218] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"digitalocean-provisioner", UID:"4868cd97-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"extensions", ResourceVersion:"821", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set digitalocean-provisioner-c4544597c to 2
I0504 18:52:24.233714       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-c4544597c", UID:"486cdea7-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"extensions", ResourceVersion:"822", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-c4544597c-99vsp
I0504 18:52:24.263499       1 event.go:218] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"digitalocean-provisioner-c4544597c", UID:"486cdea7-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"extensions", ResourceVersion:"822", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-provisioner-c4544597c-drx2g
I0504 18:52:24.805025       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"48c3acdf-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"apps", ResourceVersion:"841", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-4s5bc
I0504 18:52:24.818970       1 event.go:218] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"digitalocean-flexplugin-deploy", UID:"48c3acdf-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"apps", ResourceVersion:"841", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: digitalocean-flexplugin-deploy-7xtvt
I0504 18:56:06.015680       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1149", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
E0504 18:56:06.035231       1 pvc_protection_controller.go:134] PVC default/pv1 failed with : Operation cannot be fulfilled on persistentvolumeclaims "pv1": the object has been modified; please apply your changes to the latest version and try again
I0504 18:56:06.054338       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1151", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I0504 18:56:06.120380       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1154", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
I0504 18:56:08.323021       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pv1", UID:"cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81", APIVersion:"v1", ResourceVersion:"1159", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "external/digitalocean" or manually created by system administrator
E0504 18:56:08.714231       1 pv_protection_controller.go:112] PV pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81 failed with : Operation cannot be fulfilled on persistentvolumes "pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81": the object has been modified; please apply your changes to the latest version and try again
 …/github.com/halkeye/terraform-digitalocean  kubectl get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pv1       Bound     pvc-cc9d38d2-4fcc-11e8-8fcf-7ac73da28f81   1Gi        RWO            default        55m

Checking the DO website, it does get created, just not attached

image

klausenbusk commented 6 years ago

I just spun up a cluster in tor1 and everything is working as expected, the only required change was adding DIGITALOCEAN_ACCESS_TOKEN to kube-controller-manager. I'm wondering if it could be a bug in v1.10.0? The cluster I spun up was v1.10.2.

I did run kubectl -n kube-system delete pod -l app=digitalocean-flexplugin-deploy for other reason, could you try the same?

halkeye commented 6 years ago

I'm going to spin up a new 1.10.2 cluster.

I deleted the deploy pod, and eventually tried re-created the example pod but no luck.

I just spun up a cluster in tor1 and everything is working as expected, the only required change was adding DIGITALOCEAN_ACCESS_TOKEN to kube-controller-manager.

Should I run 

kubectl --namespace=kube-system patch deployment kube-controller-manager -p '{"spec": { "template": { "spec": { "containers": [{ "name": "kube-controller-manager", "env": [ { "name": "DIGITALOCEAN_ACCESS_TOKEN", "valueFrom": { "secretKeyRef": { "key": "access-token", "name": "digitalocean" } } } ] ] } } } '

(I hope my brackets are right, i had a more complicated version that added volume mounts as well)

klausenbusk commented 6 years ago

Should I run

It looks correct. I just did kubectl -n kube-system edit deploy kube-controller-manager and then copy-paste (under containers, remember to remove - from - command): https://github.com/kubernetes-incubator/external-storage/blob/c01d6a39bab6ecf6c7724d112d695ce70e3314bc/digitalocean/manifests/digitalocean-flexplugin-deploy.yaml#L19-L24

halkeye commented 6 years ago

new 1.10.2 cluster, running storage.sh (the init script posted above) gave me the same errors about skipping th flex plugin

Running kubectl -n kube-system delete pod -l app=digitalocean-flexplugin-deploy gave me:


May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:18.547799     896 pod_container_deletor.go:77] Container "54d763f792cbf755ea497b040332d73a5f3f1357db936565dea2ee518bf27072" not found in pod's containers
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.741992     896 reconciler.go:181] operationExecutor.UnmountVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742333     896 reconciler.go:181] operationExecutor.UnmountVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742573     896 operation_generator.go:686] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount" (OuterVolumeSpecName: "flexvolume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerVolumeSpecName "flexvolume-mount". PluginName "kubernetes.io/host-path", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.758271     896 operation_generator.go:686] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4" (OuterVolumeSpecName: "default-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerVolumeSpecName "default-token-5jmh4". PluginName "kubernetes.io/secret", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.842740     896 reconciler.go:285] Volume detached for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") on node "10.137.80.240" DevicePath ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.843489     896 reconciler.go:285] Volume detached for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") on node "10.137.80.240" DevicePath ""
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.745844     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/c3720d64-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.746585     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/c3720d64-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:20 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:20.619968     896 pod_container_deletor.go:77] Container "0e6c81a7676db13e7e3a69385d0b52b4f1b5d8c885e0e091945020e9161156d3" not found in pod's containers

tried create pvc and pod-application examples:

May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:18.547799     896 pod_container_deletor.go:77] Container "54d763f792cbf755ea497b040332d73a5f3f1357db936565dea2ee518bf27072" not found in pod's containers                                 May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.741992     896 reconciler.go:181] operationExecutor.Unmou$
tVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-defa
ult-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742333     896 reconciler.go:181] operationExecutor.Unmoun
tVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flex
volume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.742573     896 operation_generator.go:686] UnmountVolume.T
earDown succeeded for volume "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount" (OuterVolumeSpec
Name: "flexvolume-mount") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerVolu
meSpecName "flexvolume-mount". PluginName "kubernetes.io/host-path", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.758271     896 operation_generator.go:686] UnmountVolume.T
earDown succeeded for volume "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4" (OuterVolumeSpec
Name: "default-token-5jmh4") pod "82da86cf-4fe2-11e8-9052-4ed61f7ca20e" (UID: "82da86cf-4fe2-11e8-9052-4ed61f7ca20e"). InnerV
olumeSpecName "default-token-5jmh4". PluginName "kubernetes.io/secret", VolumeGidValue ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.842740     896 reconciler.go:285] Volume detached for volu
me "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-flexvolume-mount") on node "
10.137.80.240" DevicePath ""
May 04 21:33:18 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:18.843489     896 reconciler.go:285] Volume detached for volu
me "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/82da86cf-4fe2-11e8-9052-4ed61f7ca20e-default-token-5jmh4") on nod
e "10.137.80.240" DevicePath ""
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.745844     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "default-token-5jmh4" (UniqueName: "kubernetes.io/secret/c3720d64-4fe2-11e8-9052-
4ed61f7ca20e-default-token-5jmh4") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:19 nemo-worker-0 kubelet-wrapper[896]: I0504 21:33:19.746585     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "flexvolume-mount" (UniqueName: "kubernetes.io/host-path/c3720d64-4fe2-11e8-9052-
4ed61f7ca20e-flexvolume-mount") pod "digitalocean-flexplugin-deploy-t2t4w" (UID: "c3720d64-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:33:20 nemo-worker-0 kubelet-wrapper[896]: W0504 21:33:20.619968     896 pod_container_deletor.go:77] Container "0e6
c81a7676db13e7e3a69385d0b52b4f1b5d8c885e0e091945020e9161156d3" not found in pod's containers
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: W0504 21:34:41.477514     896 plugin-defaults.go:32] flexVolume driver ex
ternal-storage/digitalocean: using default GetVolumeName for volume pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: I0504 21:34:41.569756     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "f43a8b57-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: I0504 21:34:41.570077     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "default-token-k8r6j" (UniqueName: "kubernetes.io/secret/f43a8b57-4fe2-11e8-9052-
4ed61f7ca20e-default-token-k8r6j") pod "busy-pod" (UID: "f43a8b57-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:34:41 nemo-worker-0 kubelet-wrapper[896]: E0504 21:34:41.570365     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
 2018-05-04 21:34:42.070321211 +0000 UTC m=+906.570794583 (durationBeforeRetry 500ms). Error: "Volume has not been added to t
he list of VolumesInUse in the node's volume status for volume \"pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\" (UniqueName: \"fl
exvolume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"f43a8b57-4fe2-11e
8-9052-4ed61f7ca20e\") "
May 04 21:34:42 nemo-worker-0 kubelet-wrapper[896]: I0504 21:34:42.071752     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "f43a8b57-4fe2-11e8-9052-4ed61f7ca20e")
May 04 21:34:42 nemo-worker-0 kubelet-wrapper[896]: E0504 21:34:42.072156     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
 2018-05-04 21:34:43.072108529 +0000 UTC m=+907.572581903 (durationBeforeRetry 1s). Error: "Volume has not been added to the
list of VolumesInUse in the node's volume status for volume \"pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexv
olume-external-storage/digitalocean/pvc-f1b16112-4fe2-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"f43a8b57-4fe2-11e8-9
052-4ed61f7ca20e\") "

kubectl --namespace=kube-system patch deployment kube-controller-manager -p '{"spec": { "template": { "spec": { "containers": [{ "name": "kube-controller-manager", "env": [ { "name": "DIGITALOCEAN_ACCESS_TOKEN", "valueFrom": { "secretKeyRef": { "key": "access-token", "name": "digitalocean" } } } ] } ] } } } }'

May 04 21:40:11 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:11.784575     896 kubelet.go:1635] Unable to mount volumes fo
r pod "busy-pod_default(6fc9a468-4fe3-11e8-9052-4ed61f7ca20e)": timeout expired waiting for volumes to attach or mount for po
d "default"/"busy-pod". list of unmounted volumes=[vol1 default-token-k8r6j]. list of unattached volumes=[vol1 default-token-k8r6j]; skipping pod
May 04 21:40:11 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:11.785467     896 pod_workers.go:186] Error syncing pod 6fc9a468-4fe3-11e8-9052-4ed61f7ca20e ("busy-pod_default(6fc9a468-4fe3-11e8-9052-4ed61f7ca20e)"), skipping: timeout expired waiting for volumes to attach or mount for pod "default"/"busy-pod". list of unmounted volumes=[vol1 default-token-k8r6j]. list of unattached volumes=[vol1 default-token-k8r6j]
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:38.540754     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:38.636278     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "default-token-k8r6j" (UniqueName: "kubernetes.io/secret/b692ff62-4fe3-11e8-9052-4ed61f7ca20e-default-token-k8r6j") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:38.636659     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:38 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:38.636906     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
 2018-05-04 21:40:39.136874754 +0000 UTC m=+1263.637348133 (durationBeforeRetry 500ms). Error: "Volume has not been added to
the list of VolumesInUse in the node's volume status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"f
lexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:39 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:39.139488     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:39 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:39.139865     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
 2018-05-04 21:40:40.139798148 +0000 UTC m=+1264.640271508 (durationBeforeRetry 1s). Error: "Volume has not been added to the
 list of VolumesInUse in the node's volume status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-
9052-4ed61f7ca20e\") "
May 04 21:40:40 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:40.144406     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:40 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:40.145222     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until
 2018-05-04 21:40:42.145181337 +0000 UTC m=+1266.645654698 (durationBeforeRetry 2s). Error: "Volume has not been added to the
 list of VolumesInUse in the node's volume status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flex
volume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:42 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:42.152734     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:42 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:42.156660     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until 2018-05-04 21:40:46.156620298 +0000 UTC m=+1270.657093658 (durationBeforeRetry 4s). Error: "Volume not attached according to node status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexvolume-external-storage/digitalocean/
pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:46 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:46.170717     896 reconciler.go:207] operationExecutor.Verify
ControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:46 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:46.174931     896 nestedpendingoperations.go:267] Operation f
or "\"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\"" failed. No retries permitted until 2018-05-04 21:40:54.1748977 +0000 UTC m=+1278.675371072 (durationBeforeRetry 8s). Error: "Volume not attached according to n
ode status for volume \"pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\" (UniqueName: \"flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e\") pod \"busy-pod\" (UID: \"b692ff62-4fe3-11e8-9052-4ed61f7ca20e\") "
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.203458     896 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-stor
age/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e")
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.207885     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.208110     896 operation_generator.go:1168] Controller attach succeeded for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") device path: "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.304524     896 operation_generator.go:486] MountVolume.WaitForAttach entering for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") DevicePath "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.308416     896 attacher-defaults.go:39] flexVolume driver external-storage/digitalocean: using default WaitForAttach for volume 0x323b480, device /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.308590     896 operation_generator.go:495] MountVolume.WaitForAttach succeeded for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") DevicePath "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.308740     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.311338     896 attacher-defaults.go:50] flexVolume driver external-storage/digitalocean: using default MountDevice for volume 0x323b480, device /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e, deviceMountPath /var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.344237     896 mount_linux.go:485] `fsck` error fsck from util-linux 2.29.2
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: fsck.ext2: Bad magic number in super-block while trying to open /dev/sda
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: /dev/sda:
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: The superblock could not be read or does not describe a valid ext2/ext3/ext4
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: filesystem (and not swap or ufs or something else), then the superblock
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: is corrupt, and you might try running e2fsck with an alternate superblock:
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]:     e2fsck -b 8193 <device>
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]:  or
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]:     e2fsck -b 32768 <device>
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: E0504 21:40:54.353755     896 mount_linux.go:148] Mount failed: exit status 32
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: Mounting command: mount
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: Mounting arguments: -o rw,defaults /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e /var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: Output: mount: wrong fs type, bad option, bad superblock on /dev/sda,
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]:        missing codepage or helper program, or other error
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]:        In some cases useful info is found in syslog - try
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]:        dmesg | tail or so.
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.362257     896 mount_linux.go:516] Disk "/dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" appears to be unformatted, attempting to format as type: "ext4" with options: [-F /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e]
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.571853     896 mount_linux.go:520] Disk successfully formatted (mkfs): ext4 - /dev/disk/by-id/scsi-0DO_Volume_pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e /var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: I0504 21:40:54.588515     896 operation_generator.go:514] MountVolume.MountDevice succeeded for volume "pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e" (UniqueName: "flexvolume-external-storage/digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e") pod "busy-pod" (UID: "b692ff62-4fe3-11e8-9052-4ed61f7ca20e") device mount path "/var/lib/kubelet/plugins/kubernetes.io/flexvolume/external-storage/digitalocean/mounts/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e"
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.602221     896 mounter-defaults.go:30] flexVolume driver external-storage/digitalocean: using default SetUpAt to /var/lib/kubelet/pods/b692ff62-4fe3-11e8-9052-4ed61f7ca20e/volumes/external-storage~digitalocean/pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:54 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:54.602414     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:56 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:56.325687     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
May 04 21:40:57 nemo-worker-0 kubelet-wrapper[896]: W0504 21:40:57.343737     896 plugin-defaults.go:32] flexVolume driver external-storage/digitalocean: using default GetVolumeName for volume pvc-ae6bb3fb-4fe3-11e8-9052-4ed61f7ca20e
halkeye commented 6 years ago

so I think it works after adding the secrets to the pod controller.

halkeye commented 6 years ago

The pod started up \o/

 …/github.com/halkeye/terraform-digitalocean  kubectl exec busy-pod -- df -h /mnt/pv-1
Filesystem                Size      Used Available Use% Mounted on
/dev/sda                975.9M      2.5M    906.2M   0% /mnt/pv-1

 …/github.com/halkeye/terraform-digitalocean  kubectl exec busy-pod -- ls -l /mnt/pv-1
total 16
drwx------    2 root     root         16384 May  4 21:40 lost+found

As far as I can tell this is working.

Does the kubectl patch step need to be added to docs somewhere?

klausenbusk commented 6 years ago

Does the kubectl patch step need to be added to docs somewhere?

I have a tracking issue for required changes for different setups, but haven't got time to update the README. https://github.com/kubernetes-incubator/external-storage/issues/531

Ideally people should just run kubectl apply -f manifests/bootkube.yaml and it should work with bootkube.

halkeye commented 6 years ago

is that manifests/bootkube.yaml file somewhere? (i tried but didn't see it in this repo)

So far I have a working repeatable cluster, which is super exciting, I can use the kubectl patch thing, but but i rather script it the right way.

klausenbusk commented 6 years ago

is that manifests/bootkube.yaml file somewhere? (i tried but didn't see it in this repo)

Sorry, it was just a idea, but I suppose the following could work:

spec:
  template:
    spec:
      containers:
      - env:
        - name: DIGITALOCEAN_ACCESS_TOKEN
          valueFrom:
            secretKeyRef:
              key: access-token
              name: digitalocean
        name: kube-controller-manager

Save as bootkube-patch.yaml and run kubectl -n kube-system patch deploy kube-controller-manager -p "$(<bootkube-patch.yaml)"

klausenbusk commented 6 years ago

@halkeye would you mind closing this issue?