flexdriver on minikube does not work

[dharmeshkakadia]

While trying this on minikube, I got the following error.

Unable to mount volumes for pod "nginx-flex-blobfuse_default(4774ea6f-3930-11e8-b5c4-92f68014defe)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx-flex-blobfuse". list of unattached/unmounted volumes=[test]

I have been following https://github.com/andyzhangx/kubernetes-drivers/tree/master/flexvolume/blobfuse

kubectl version Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2018-01-26T19:04:38Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

[dharmeshkakadia]

cc @seguler

[andyzhangx]

@dharmeshkakadia I am not sure whether minikube would support this driver, could you try on a k8s cluster on azure, like AKS? And also could you get kubelet log from that minikube?

[dharmeshkakadia]

The installation was successful on minikube, which gave me impression that it would work.

The kubelet logs are full of

Apr 06 10:34:08 minikube localkube[2500]: E0406 10:34:08.278916 2500 desired_state_of_world_populator.go:286] Failed to add volume "test" (specName: "test") for pod "4774ea6f-3930-11e8-b5c4-92f68014defe" to desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "test" err=no volume plugin matched Apr 06 10:34:09 minikube localkube[2500]: E0406 10:34:08.413831 2500 kubelet.go:1625] Unable to mount volumes for pod "nginx-flex-blobfuse_default(4774ea6f-3930-11e8-b5c4-92f68014defe)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx-flex-blobfuse". list of unattached/unmounted volumes=[test]; skipping pod

[andyzhangx]

Flexvolume driver on minikube may not work, see issue here: https://github.com/kubernetes/minikube/issues/2628 I would encourage you to use a real k8s cluster on Azure.

[dharmeshkakadia]

I have tried this on acsengine cluster and now I am getting some other error :

MountVolume.SetUp failed for volume "test" : invalid character 'C' looking for beginning of value

[andyzhangx]

@dharmeshkakadia could you provide logs: /var/log/blobfuse-driver.log under your one agent node?

[dharmeshkakadia]

Its full of Thu Apr 19 01:44:34 UTC 2018 EXEC: mkdir -p /var/lib/kubelet/pods/35e4272b-4373-11e8-9f2a-000d3a114aa6/volumes/azure~blobfuse/test Thu Apr 19 01:44:34 UTC 2018 INF: export storage account - export AZURE_STORAGE_ACCOUNT= AZURE_STORAGE_ACCESS_KEY=... Thu Apr 19 01:44:34 UTC 2018 EXEC: /etc/kubernetes/volumeplugins/azure~blobfuse/bin/blobfuse /var/lib/kubelet/pods/35e4272b-4373-11e8-9f2a-000d3a114aa6/volumes/azure~blobfuse/test --container-name=sparkdkakadia --tmp-path=/tmp/blobfuse/ -o allow_other

[andyzhangx]

@dharmeshkakadia could you get kubelet logs on that node, could follow by: https://github.com/andyzhangx/demo/tree/master/debug#q-how-to-get-k8s-kubelet-logs-on-linux-agent? And what's your k8s version?

[dharmeshkakadia]

https://gist.github.com/dharmeshkakadia/8e4f234c92ca2fc4a4d3f389c1f3829f#file-8e36f5d17d59-log

[dharmeshkakadia]

kubectl version Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

[andyzhangx]

@dharmeshkakadia your kubelet logs are not complete, actually I would like to get the logs around invalid character..., thanks.

[dharmeshkakadia]

yeah, I think I hit the gist limit. Here is the last 10k lines. https://gist.github.com/dharmeshkakadia/268075d30812958c23aa0167278ff5e7

[andyzhangx]

@dharmeshkakadia Recently I have updated the doc, should use accountname and accountkey fields.

1. create a secret which stores azure storage account name and key

kubectl create secret generic blobfusecreds --from-literal accountname=ACCOUNT-NAME --from-literal accountkey="ACCOUNT-KEY" --type="azure/blobfuse"

[andyzhangx]

@dharmeshkakadia I would provide a fix to make the error message more meaningful, thanks.

[andyzhangx]

@dharmeshkakadia just delete the pod and blobfusecreds secret and use following command to create secret, it should work:

kubectl create secret generic blobfusecreds --from-literal accountname=ACCOUNT-NAME --from-literal accountkey="ACCOUNT-KEY" --type="azure/blobfuse"

[dharmeshkakadia]

ok. Now it seems to have mounted (thanks!), but I am still not able to read anything from the blob or cd to any folder. Also, all the folders seem to have been listed twice. Is this something you can help or is this is a bug on the blobfuse side? cc @seguler

root@nginx-flex-blobfuse:/data# ls -la total 4 drwxrwxrwx 2 root root 4096 Apr 19 05:14 . drwxr-xr-x 1 root root 4096 Apr 19 05:12 .. -rwxrwxrwx 1 root root 0 Mar 1 2017 HdiNotebooks -rwxrwxrwx 1 root root 0 Mar 1 2017 HdiNotebooks -rwxrwxrwx 1 root root 0 Mar 1 2017 HdiSamples -rwxrwxrwx 1 root root 0 Mar 1 2017 HdiSamples -rwxrwxrwx 1 root root 0 Mar 1 2017 HiveTPCDS -rwxrwxrwx 1 root root 0 Mar 1 2017 HiveTPCDS -rwxrwxrwx 1 root root 0 Mar 15 2017 HiveTPCH -rwxrwxrwx 1 root root 0 Mar 15 2017 HiveTPCH drwxrwxrwx 3 root root 4096 Jan 1 1970 _$azuretmpfolder$ -rwxrwxrwx 1 root root 0 Mar 1 2017 ams -rwxrwxrwx 1 root root 0 Mar 1 2017 ams -rwxrwxrwx 1 root root 0 Mar 1 2017 amshbase -rwxrwxrwx 1 root root 0 Mar 1 2017 amshbase -rwxrwxrwx 1 root root 0 Jan 1 1970 app-logs -rwxrwxrwx 1 root root 0 Mar 1 2017 apps -rwxrwxrwx 1 root root 0 Mar 1 2017 apps -rwxrwxrwx 1 root root 0 Mar 1 2017 atshistory -rwxrwxrwx 1 root root 0 Mar 1 2017 atshistory -rwxrwxrwx 1 root root 0 Mar 1 2017 custom-scriptaction-logs -rwxrwxrwx 1 root root 0 Mar 1 2017 custom-scriptaction-logs -rwxrwxrwx 1 root root 0 Mar 1 2017 example -rwxrwxrwx 1 root root 0 Mar 1 2017 example -rwxrwxrwx 1 root root 0 Mar 1 2017 hbase -rwxrwxrwx 1 root root 0 Mar 1 2017 hdp -rwxrwxrwx 1 root root 0 Mar 1 2017 hdp -rwxrwxrwx 1 root root 0 Mar 1 2017 hive -rwxrwxrwx 1 root root 0 Mar 1 2017 hive -rwxrwxrwx 1 root root 0 Mar 1 2017 mapred -rwxrwxrwx 1 root root 0 Mar 1 2017 mapred -rwxrwxrwx 1 root root 0 Mar 29 2017 mapreducestaging -rwxrwxrwx 1 root root 0 Mar 29 2017 mapreducestaging -rwxrwxrwx 1 root root 0 Mar 1 2017 mr-history -rwxrwxrwx 1 root root 0 Mar 1 2017 mr-history -rwxrwxrwx 1 root root 0 Mar 1 2017 tmp -rwxrwxrwx 1 root root 0 Mar 1 2017 tmp -rwxrwxrwx 1 root root 0 Mar 1 2017 user -rwxrwxrwx 1 root root 0 Mar 1 2017 user

[seguler]

oh it seems the blobfuse binary we have here in this repository is outdated. It does not support looking at a blob directory structure created by WASB/HDI. Blobfuse 0.4 and on-ward supports it. Let me update the binaries here.

[andyzhangx]

@dharmeshkakadia I have updated the latest deployment: blobfuse-flexvol-installer:1.0.3, it's using 1.0.0-RC blobfuse binary now. pls first delete the old daemonset, and then redploy daemonset blobfuse-flexvol-installer according to the latest doc:

kubectl delete daemonset blobfuse-flexvol-installer --namespace=flex

[dharmeshkakadia]

It working now. Thanks for helping out.