search

vmware-archive / vsphere-storage-for-docker

vSphere Storage for Docker
https://vmware.github.io/vsphere-storage-for-docker
Apache License 2.0
251 stars 95 forks source link

Can't use docker-volume-vsphere plugin with Alpine Linux #2042

Closed galindro closed 6 years ago

galindro commented 6 years ago

Environment Details

VM

Guest OS:             Other 3.x or later Linux (64-bit) 
Compatibility:        ESXi 6.5 and later (VM version 13) 
VMware Tools:         Running, version:10272 (Guest Managed) 
OS:                   Alpine Linux 3.6.2
Kernel:               4.9.32-0-hardened

ESX

Hypervisor:           VMware ESXi, 6.5.0, 6765664 
Model:                PowerEdge R710 
Processor Type:       Intel(R) Xeon(R) CPU X5680 @ 3.33GHz

Datastore

Name:                 VNX1_LUN_ID_92
Type:                 VMFS 5.61
URL:                  ds:///vmfs/volumes/5925d556-a8685bcc-8fea-bc305bfdff4e/ 
Total Capacity:       1,023.75 GB
Provisioned Space:    355.48 GB
Free Space:           668.27 GB

Steps to Reproduce

  1. Install Alpine Linux 3.6.2

  2. Edit /etc/apk/repositories and uncomment the following line:

http://linorg.usp.br/AlpineLinux/v3.6/community
  1. Install these packages:
apk update
apk add bash vim open-vm-tools docker udev xfsprogs util-linux htop alpine-sdk py-pip python-dev linux-headers go iproute2
  1. Configure these packages to start on boot:
rc-update add open-vm-tools
rc-update add udev-trigger sysinit
rc-update add udev sysinit
rc-update add docker boot

  1. Reboot the VM

  2. Check if /dev/disk/ exists:

ls -lh /dev/disk/
  1. Install docker vsphere plugin:
docker plugin install --grant-all-permissions --alias vsphere vmware/docker-volume-vsphere:latest
  1. Create a volume
docker volume create --driver=vsphere --name=registry -o size=1gb -o fstype=xfs

Expected Result

No errors and volume created

Actual Result

Error response from daemon: create registry: VolumeDriver.Create: Device not found

$ docker volume ls
DRIVER              VOLUME NAME

Triage

VM Logs and INFO

$ cat /var/log/docker-volume-vsphere.log
2017-12-15 08:50:40.290325052 -0200 -02 [INFO] Starting plugin driver=vsphere log_level=info config="/etc/docker-volume-vsphere.conf" group=root 
2017-12-15 08:50:40.29050906 -0200 -02 [INFO] Getting volume data from unix:///var/run/docker.sock
2017-12-15 08:50:40.304599424 -0200 -02 [INFO] Found 0 running or paused containers
2017-12-15 08:50:40.305013655 -0200 -02 [INFO] Discovered 0 volumes that may be in use.
2017-12-15 08:50:40.305027219 -0200 -02 [INFO] Refcounting successfully completed
2017-12-15 08:50:40.305039892 -0200 -02 [INFO] Docker VMDK plugin started version=latest port=1019 mock_esx=false 
2017-12-15 08:50:40.305262427 -0200 -02 [INFO] Going into ServeUnix - Listening on Unix socket address="/run/docker/plugins/vsphere.sock" 
2017-12-15 08:52:46.639818963 -0200 -02 [ERROR] Failed to get volume meta-data name=registry error="Volume registry not found (file: /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk)" 
2017-12-15 08:52:47.130922724 -0200 -02 [INFO] Attaching volume and creating filesystem name=registry fstype=xfs 
2017-12-15 08:52:47.751604007 -0200 -02 [WARNING] Get device path failed for unit %s @ PCI bridge device %s: 1Error="open /sys/bus/pci/devices/0000:00:/pci_bus: no such file or directory" 
2017-12-15 08:52:47.751671711 -0200 -02 [ERROR] Failed to get device path err="Device not found" volDev={1 160 } 
2017-12-15 08:52:47.751701346 -0200 -02 [ERROR] Could not find attached device, removing the volume error="Device not found" name=registry 
$ ls -lh /sys/bus/pci/devices/0000:00:/pci_bus
ls: /sys/bus/pci/devices/0000:00:/pci_bus: No such file or directory

$ ls -lh /sys/bus/pci/devices/
total 0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:00.0 -> ../../../devices/pci0000:00/0000:00:00.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:01.0 -> ../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:07.0 -> ../../../devices/pci0000:00/0000:00:07.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:07.1 -> ../../../devices/pci0000:00/0000:00:07.1
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:07.3 -> ../../../devices/pci0000:00/0000:00:07.3
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:07.7 -> ../../../devices/pci0000:00/0000:00:07.7
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:0f.0 -> ../../../devices/pci0000:00/0000:00:0f.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:11.0 -> ../../../devices/pci0000:00/0000:00:11.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.0 -> ../../../devices/pci0000:00/0000:00:15.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.1 -> ../../../devices/pci0000:00/0000:00:15.1
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.2 -> ../../../devices/pci0000:00/0000:00:15.2
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.3 -> ../../../devices/pci0000:00/0000:00:15.3
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.4 -> ../../../devices/pci0000:00/0000:00:15.4
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.5 -> ../../../devices/pci0000:00/0000:00:15.5
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.6 -> ../../../devices/pci0000:00/0000:00:15.6
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:15.7 -> ../../../devices/pci0000:00/0000:00:15.7
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.0 -> ../../../devices/pci0000:00/0000:00:16.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.1 -> ../../../devices/pci0000:00/0000:00:16.1
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.2 -> ../../../devices/pci0000:00/0000:00:16.2
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.3 -> ../../../devices/pci0000:00/0000:00:16.3
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.4 -> ../../../devices/pci0000:00/0000:00:16.4
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.5 -> ../../../devices/pci0000:00/0000:00:16.5
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.6 -> ../../../devices/pci0000:00/0000:00:16.6
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:16.7 -> ../../../devices/pci0000:00/0000:00:16.7
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.0 -> ../../../devices/pci0000:00/0000:00:17.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.1 -> ../../../devices/pci0000:00/0000:00:17.1
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.2 -> ../../../devices/pci0000:00/0000:00:17.2
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.3 -> ../../../devices/pci0000:00/0000:00:17.3
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.4 -> ../../../devices/pci0000:00/0000:00:17.4
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.5 -> ../../../devices/pci0000:00/0000:00:17.5
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.6 -> ../../../devices/pci0000:00/0000:00:17.6
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:17.7 -> ../../../devices/pci0000:00/0000:00:17.7
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.0 -> ../../../devices/pci0000:00/0000:00:18.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.1 -> ../../../devices/pci0000:00/0000:00:18.1
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.2 -> ../../../devices/pci0000:00/0000:00:18.2
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.3 -> ../../../devices/pci0000:00/0000:00:18.3
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.4 -> ../../../devices/pci0000:00/0000:00:18.4
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.5 -> ../../../devices/pci0000:00/0000:00:18.5
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.6 -> ../../../devices/pci0000:00/0000:00:18.6
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:00:18.7 -> ../../../devices/pci0000:00/0000:00:18.7
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:03:00.0 -> ../../../devices/pci0000:00/0000:00:15.0/0000:03:00.0
lrwxrwxrwx    1 root     root           0 Dec 15 08:43 0000:0b:00.0 -> ../../../devices/pci0000:00/0000:00:16.0/0000:0b:00.0

HOST Logs and INFO


$ cat /var/log/vmware/vmdk_ops.log

12/15/17 08:52:45 69815 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
12/15/17 08:52:46 69815 [MainThread] [INFO   ] Started new thread : 247720314624 with target <function execRequestThread at 0x39ab85da60> and args (10, 413771, b'{"cmd":"get","details":{"Name":"registry"},"version":"2"}')
12/15/17 08:52:46 69815 [Thread-20] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
12/15/17 08:52:46 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] executeRequest 'get' completed with ret={'Error': 'Volume registry not found (file: /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk)'}
12/15/17 08:52:46 69815 [MainThread] [INFO   ] Started new thread : 247720314624 with target <function execRequestThread at 0x39ab85da60> and args (13, 413771, b'{"cmd":"create","details":{"Name":"registry","Opts":{"fstype":"xfs","size":"1gb"}},"version":"2"}')
12/15/17 08:52:46 69815 [Thread-21] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
12/15/17 08:52:46 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] *** createVMDK: /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk opts={'fstype': 'xfs', 'size': '1gb'} vm_name=docker-sw-01 vm_uuid=42219456-a216-312b-3448-5501b3c8e7e0 tenant_uuid=11111111-1111-1111-1111-111111111111 datastore_url=/vmfs/volumes/5925d556-a8685bcc-8fea-bc305bfdff4e
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] No access control, skipping volumes tracing in auth DB
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] executeRequest 'create' completed with ret=None
12/15/17 08:52:47 69815 [MainThread] [INFO   ] Started new thread : 247720314624 with target <function execRequestThread at 0x39ab85da60> and args (10, 413771, b'{"cmd":"attach","details":{"Name":"registry"},"version":"2"}')
12/15/17 08:52:47 69815 [Thread-22] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] *** disk_attach: VMDK /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk to VM 'docker-sw-01' , bios uuid = 42219456-a216-312b-3448-5501b3c8e7e0, VC uuid=502130dc-6400-1c1c-ce94-946e1b6a1967)
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] Found vm name='docker-sw-01'
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] Attaching /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk as independent_persistent
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] Disk /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk successfully attached. controller pci_slot_number=160, disk_slot=1
12/15/17 08:52:47 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] executeRequest 'attach' completed with ret={'ControllerPciSlotNumber': '160', 'Unit': '1'}
12/15/17 08:52:47 69815 [MainThread] [INFO   ] Started new thread : 247720314624 with target <function execRequestThread at 0x39ab85da60> and args (13, 413771, b'{"cmd":"detach","details":{"Name":"registry"},"version":"2"}')
12/15/17 08:52:47 69815 [Thread-23] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] *** disk_detach: VMDK /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk to VM 'docker-sw-01' , bios uuid = 42219456-a216-312b-3448-5501b3c8e7e0, VC uuid=502130dc-6400-1c1c-ce94-946e1b6a1967)
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] Found vm name='docker-sw-01'
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] Disk detached /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] executeRequest 'detach' completed with ret=None
12/15/17 08:52:48 69815 [MainThread] [INFO   ] Started new thread : 247720314624 with target <function execRequestThread at 0x39ab85da60> and args (10, 413771, b'{"cmd":"remove","details":{"Name":"registry"},"version":"2"}')
12/15/17 08:52:48 69815 [Thread-24] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] *** removeVMDK: /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] *** cleanVMDK: /vmfs/volumes/VNX1_LUN_ID_92/dockvols/_DEFAULT/registry.vmdk
12/15/17 08:52:48 69815 [docker-sw-01-VNX1_LUN_ID_92._DEFAULT.registry] [INFO   ] executeRequest 'remove' completed with ret=None
12/15/17 09:02:16 69815 [MainThread] [INFO   ] Started new thread : 247720314624 with target <function execRequestThread at 0x39ab85da60> and args (13, 413771, b'{"cmd":"list","details":{"Name":""},"version":"2"}')
12/15/17 09:02:16 69815 [Thread-25] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
12/15/17 09:02:17 69815 [docker-sw-01-nolock-list] [INFO   ] executeRequest 'list' completed with ret=[]
$ esxcli software vib list |grep vmdkops
esx-vmdkops-service            0.18.087b0d7-0.0.1                       VMWare   VMwareAccepted    2017-12-03
govint commented 6 years ago

@galindro, please update the VIB on ESX to 0.19 , send to be an older version. Version 0.19 is enhanced to handle device identification in a more robust way and needs the most current VIB

shuklanirdesh82 commented 6 years ago

duplicate to #1969 .. @galindro As @govint mentioned please update your VIB to 0.19 release and your problem should be resolved.

I'm closing this issue, please reach us out if you are having any problem after upgrading to 0.19 VIB (VDVS driver).