search

hpe-storage / dory

Kubernetes Flexvolume Driver for Docker Volume Plugins
Apache License 2.0
28 stars 10 forks source link

Bug: Mount of the POD using managed docker volume plugin fails #36

Open hpe-storage opened 6 years ago

hpe-storage commented 6 years ago

Under plain kubernetes environment, the flexvolumedriver binaries are installed, under 

[docker@cld13b4 ~]$ ls -l /usr/libexec/kubernetes/kubelet-plugins/volume/exec/hpe.com~hpe/
total 52360
-rwxr-xr-x. 1 docker docker 47046107 Apr 20 06:11 doryd
-rwxr-xr-x. 1 docker docker  6561963 Apr 20 06:11 hpe
-rw-r--r--. 1 docker docker      236 May 18 10:06 hpe.json

Config Details:

{
    "dockerVolumePluginSocketPath": "hpe:latest",
    "logDebug": true,
    "supportsCapabilities": true,
    "stripK8sFromOptions": true,
    "createVolumes": true,
    "listOfStorageResourceOptions": [ "size" ]
}

Contents from /var/log/dory.log

Info : 2018/05/18 08:18:23 dory.go:80: [3930] entry  : Driver=hpe Version=1.1.0-ba064b80 Socket=hpe:latest Overridden=true
Info : 2018/05/18 08:18:23 dory.go:82: [3930] request: mount [/var/lib/kubelet/pods/3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30/volumes/hpe.com~hpe/sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30 {"kubernetes.io/fsType":"","kubernetes.io/pod.name":"pod-comp3","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/pvOrVolumeName":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/readwrite":"rw","kubernetes.io/serviceAccount.name":"default","name":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","size":"16"}]
Debug: 2018/05/18 08:18:23 client.go:126: request: action=GET path=http://unix/plugins payload=
Debug: 2018/05/18 08:18:23 client.go:168: response: 200 OK, length=-1
Debug: 2018/05/18 08:18:23 dockerlt.go:62: returning []dockerlt.Plugin{dockerlt.Plugin{ID:"bc1f8296a8742b04d4e406c898c6dd6677735faf5ebd04cfc735c64e5c3ab5f3", Name:"hpe:latest", Enabled:true, Config:dockerlt.PluginConfig{Interface:dockerlt.PluginInterface{Socket:"hpe.sock"}}}}
Debug: 2018/05/18 08:18:23 client.go:126: request: action=POST path=http://unix/VolumeDriver.Capabilities payload={}
Debug: 2018/05/18 08:18:23 client.go:168: response: 200 OK, length=-1
Debug: 2018/05/18 08:18:23 dockervol.go:190: returning &dockervol.CapResponse{Capabilities:dockervol.PluginCapabilities{Scope:"global"}}
Debug: 2018/05/18 08:18:23 flexvol.go:194: mount called with [/var/lib/kubelet/pods/3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30/volumes/hpe.com~hpe/sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30 {"kubernetes.io/fsType":"","kubernetes.io/pod.name":"pod-comp3","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/pvOrVolumeName":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/readwrite":"rw","kubernetes.io/serviceAccount.name":"default","name":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","size":"16"}]
Debug: 2018/05/18 08:18:23 flexvol.go:344: findJSON(1) about to unmarshal {"kubernetes.io/fsType":"","kubernetes.io/pod.name":"pod-comp3","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/pvOrVolumeName":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/readwrite":"rw","kubernetes.io/serviceAccount.name":"default","name":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","size":"16"}
Debug: 2018/05/18 08:18:23 flexvol.go:167: getOrCreate called with sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30 and {"kubernetes.io/fsType":"","kubernetes.io/pod.name":"pod-comp3","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/pvOrVolumeName":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","kubernetes.io/readwrite":"rw","kubernetes.io/serviceAccount.name":"default","name":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","size":"16"}
Debug: 2018/05/18 08:18:23 client.go:126: request: action=POST path=http://unix/VolumeDriver.Get payload={"Name":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30"}
Debug: 2018/05/18 08:18:23 client.go:168: response: 200 OK, length=-1
Debug: 2018/05/18 08:18:23 dockervol.go:214: returning &dockervol.GetResponse{Volume:dockervol.DockerVolume{Name:"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30", Mountpoint:"/opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52", Status:map[string]interface {}{"volume_detail":map[string]interface {}{"mountConflictDelay":30, "flash_cache":interface {}(nil), "provisioning":"thin", "compression":interface {}(nil), "size":16}}}, Err:""}
Debug: 2018/05/18 08:18:23 flexvol.go:307: getMountID called with /var/lib/kubelet/pods/3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30/volumes/hpe.com~hpe/sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30
Debug: 2018/05/18 08:18:23 flexvol.go:313: getMountID returning "3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30"
Debug: 2018/05/18 08:18:23 client.go:126: request: action=POST path=http://unix/VolumeDriver.Mount payload={"Name":"sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30","ID":"3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30"}
Debug: 2018/05/18 08:18:23 client.go:168: response: 200 OK, length=-1
Debug: 2018/05/18 08:18:23 bmount.go:42: BindMount called with /opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52 /var/lib/kubelet/pods/3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30/volumes/hpe.com~hpe/sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30 false
Debug: 2018/05/18 08:18:23 cmd.go:33: ExecCommandOutput called with mount[--bind /opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52 /var/lib/kubelet/pods/3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30/volumes/hpe.com~hpe/sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30]
Debug: 2018/05/18 08:18:23 cmd.go:49: out :mount: special device /opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52 does not exist
Debug: 2018/05/18 08:18:23 cmd.go:49: out :
Error: 2018/05/18 08:18:23 bmount.go:51: BindMount failed with 32.  It was called with /opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52 /var/lib/kubelet/pods/3cf80cd9-5a95-11e8-8f3e-ecb1d7a4af30/volumes/hpe.com~hpe/sc-comp3-3ced2d29-5a95-11e8-8f3e-ecb1d7a4af30 false.  Output=mount: special device /opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52 does not exist
.

looking at the system output -- ls -l /opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52 reveals the directory is not directly available in the docker host, but under the plugin's rootfs something similar to this

[docker@cld13b2 ~]$ sudo find / -name "hpedocker-dm-uuid-mpath-360002ac00000000001012f9e00019d52" -type d
/var/lib/docker/plugins/114534cb1fc8933961bd954bdaee237133ba64d768226bba39b804555271e316/rootfs/opt/hpe/data/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52
/var/lib/docker/plugins/114534cb1fc8933961bd954bdaee237133ba64d768226bba39b804555271e316/propagated-mount/hpedocker-dm-uuid-mpath-360002ac00000000001012f2e00019d52

So, it short the flexvolumedriver looks it does the following

  1. docker volume inspect <vol>
  2. Get the MountPoint from the above command in step 1
  3. Do a bind mount for the path obtained to the POD's mount point folder.

As part of this , can I ask , if the flexvolumedriver

  1. Can detect the managed plugin operating in the host
  2. Get the plugin id
  3. And look for the mountpoint under rootfs filesystem of the plugin /var/lib/docker/plugins//rootfs/

where "mountpoint" is got from the $ docker volume inspect <vol>

datamattsson commented 6 years ago

Hello @hpe-storage, Docker EE 2.0 does not support managed Docker Volume plugins as per a statement received from Docker, Inc. The kubelet runs in a container with no visibility into /var/lib/docker/plugins (the kubelet needs to bind mount /var/lib/docker/plugins as rshared) and therefor the flexvolume driver is unable to bind mount the path for the pod you're trying to create. Please raise this issue with Docker, Inc.

wdurairaj commented 6 years ago

@drajen,

If the /var/lib/docker/plugins is not present in the bind mounts in the kubelet container, could you clarify how the nimble docker volume plugin (managed plugin) is able to work with kubernetes 1.8 environment ?

Anything specific you have done to resolve this. And we are not running in Docker EE 2.0 release but Docker 17.03

[docker@kuber-master ~]$ docker version
Client:
 Version:      17.03.2-ee-8
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   74b7e2f
 Built:        Wed Dec 13 21:06:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ee-8
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   74b7e2f
 Built:        Wed Dec 13 21:06:31 2017
 OS/Arch:      linux/amd64
 Experimental: false
datamattsson commented 6 years ago

Docker, Inc does not allow managed plugins to be accessed by the ucp-kubelet. It's an informed decision made by Docker, Inc engineering. Legacy plugins work just fine and that's the route Nimble has taken.

On Wed, Jun 13, 2018, 7:53 AM William Durairaj notifications@github.com wrote:

@drajen https://github.com/drajen,

If the /var/lib/docker/plugins is not present in the bind mounts in the kubelet container, could you clarify how the nimble docker volume plugin (managed plugin) is able to work with kubernetes 1.8 environment ?

Anything specific you have done to resolve this. And we are not running in Docker EE 2.0 release but Docker 17.03

[docker@kuber-master ~]$ docker version Client: Version: 17.03.2-ee-8 API version: 1.27 Go version: go1.7.5 Git commit: 74b7e2f Built: Wed Dec 13 21:06:31 2017 OS/Arch: linux/amd64

Server: Version: 17.03.2-ee-8 API version: 1.27 (minimum version 1.12) Go version: go1.7.5 Git commit: 74b7e2f Built: Wed Dec 13 21:06:31 2017 OS/Arch: linux/amd64 Experimental: false

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hpe-storage/dory/issues/36#issuecomment-396966701, or mute the thread https://github.com/notifications/unsubscribe-auth/AGs1aE8jjOID0EZNfgRQRDGx5lqcDnaQks5t8SdsgaJpZM4UG71B .