yp-engineering / rbd-docker-plugin

Ceph RBD docker volume driver plugin.
MIT License
102 stars 70 forks source link

containers won't start #38

Open immesys opened 8 years ago

immesys commented 8 years ago

If I try start docker using an rbd volume, it never starts (just hangs), and some other docker commands hang too (like docker ps).

Output from rbd-docker-plugin:

sudo RBD_DOCKER_PLUGIN_DEBUG=1 ./rbd-docker-plugin --create                                                                    
rbd-volume-plugin: 2016/06/16 07:16:22 main.go:70: INFO: Setting up Ceph Driver for PluginID=rbd, cluster=, user=admin, pool=rbd, mount=/var/lib/docker-volumes, config=
rbd-volume-plugin: 2016/06/16 07:16:22 driver.go:83: INFO: newCephRBDVolumeDriver: setting base mount dir=/var/lib/docker-volumes/rbd
rbd-volume-plugin: 2016/06/16 07:16:22 driver.go:581: INFO: connecting to Ceph and default pool context
rbd-volume-plugin: 2016/06/16 07:16:22 main.go:82: INFO: Creating Docker VolumeDriver Handler
rbd-volume-plugin: 2016/06/16 07:16:22 main.go:86: INFO: Opening Socket for Docker to connect: /run/docker/plugins/rbd.sock
rbd-volume-plugin: 2016/06/16 07:16:45 driver.go:677: DEBUG: parseImagePoolNameSize: "bar3": ["bar3" "" "" "bar3" "" ""]
rbd-volume-plugin: 2016/06/16 07:16:45 driver.go:439: INFO: Get request(bar3) => /var/lib/docker-volumes/rbd/rbd/bar3
rbd-volume-plugin: 2016/06/16 07:16:45 driver.go:296: INFO: Mount(bar3)
rbd-volume-plugin: 2016/06/16 07:16:45 driver.go:677: DEBUG: parseImagePoolNameSize: "bar3": ["bar3" "" "" "bar3" "" ""]
rbd-volume-plugin: 2016/06/16 07:16:45 driver.go:708: INFO: rbdImageExists(rbd/bar3)

Docker command:

docker run --volume-driver=rbd -v bar3:/foobar -it ubuntu bash 

Output from docker, starting from the docker run command

DEBU[0053] Calling POST /v1.23/containers/create        
DEBU[0053] form data: {"AttachStderr":true,"AttachStdin":true,"AttachStdout":true,"Cmd":["bash"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":["bar3:/foobar"],"BlkioBps":0,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioIOps":0,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SandboxSize":0,"SecurityOpt":null,"ShmSize":0,"StorageOpt":null,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"rbd","VolumesFrom":null},"Hostname":"","Image":"ubuntu","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":true,"Tty":true,"User":"","Volumes":{},"WorkingDir":""} 
DEBU[0053] rbd implements: VolumeDriver                 
DEBU[0053] Registering new volume reference: driver "rbd", name "bar3" 
DEBU[0053] container mounted via layerStore: /var/lib/docker/aufs/mnt/94137310acd31be7f5411580190d23f9a71c5a990be0cf153d5029ecf84222bf 
DEBU[0053] copying image data from b6aedfb0672d6f0d83bee150e2ac4802d49e1e73eb0cb644fce947f5391e6622:/foobar, to bar3 
DEBU[0053] Calling POST /v1.23/containers/b6aedfb0672d6f0d83bee150e2ac4802d49e1e73eb0cb644fce947f5391e6622/attach?stderr=1&stdin=1&stdout=1&stream=1 
DEBU[0053] attach: stdin: begin                         
DEBU[0053] attach: stdout: begin                        
DEBU[0053] attach: stderr: begin                        
DEBU[0053] Calling POST /v1.23/containers/b6aedfb0672d6f0d83bee150e2ac4802d49e1e73eb0cb644fce947f5391e6622/start 
DEBU[0053] container mounted via layerStore: /var/lib/docker/aufs/mnt/94137310acd31be7f5411580190d23f9a71c5a990be0cf153d5029ecf84222bf 
DEBU[0053] Assigning addresses for endpoint clever_williams's interface on network bridge 
DEBU[0053] RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[]) 
DEBU[0053] Assigning addresses for endpoint clever_williams's interface on network bridge 
DEBU[0053] Programming external connectivity on endpoint clever_williams (39fc3649edeb4c91e90b121cbbe193b5f4d3320e1fff5766263950a6f7e963f4) 

It just hangs there, neither the docker daemon, rbd plugin nor the docker command progress or print anything more.

Docker version info:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Ceph is "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"

The ceph cluster is ok, the "rbd" pool exists (with a lot of space), and I can manipulate rbd's manually.

This is a fresh gopath, so all of the dependencies (like ceph-go) will definitely be up to date in the build.

Any ideas?

EDIT: also:

Ubuntu 14.04.4 LTS
Linux 3.13.0-88-generic