search

yp-engineering / rbd-docker-plugin

Ceph RBD docker volume driver plugin.
MIT License
102 stars 70 forks source link

fatal error: unexpected signal during runtime execution #34

Closed vincepii closed 6 years ago

vincepii commented 8 years ago

I am running rbd-docker-plugin using ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269) and following the lines of this post: http://ceph.com/planet/getting-started-with-the-docker-rbd-volume-plugin/.

I start the plugin with:

rbd-docker-plugin --create true

and I try to deploy a container with

docker run --rm -it --volume-driver=rbd --volume createit:/mnt/createit ceph/base bash

This is what I see in the logs:

rbd-volume-plugin: 2016/05/25 17:21:00 main.go:70: INFO: Setting up Ceph Driver for PluginID=rbd, cluster=, user=admin, pool=rbd, mount=/var/lib/docker-volumes, config=
rbd-volume-plugin: 2016/05/25 17:21:00 driver.go:83: INFO: newCephRBDVolumeDriver: setting base mount dir=/var/lib/docker-volumes/rbd
rbd-volume-plugin: 2016/05/25 17:21:00 driver.go:554: INFO: connecting to Ceph and default pool context
rbd-volume-plugin: 2016/05/25 17:21:00 main.go:82: INFO: Creating Docker VolumeDriver Handler
rbd-volume-plugin: 2016/05/25 17:21:00 main.go:86: INFO: Opening Socket for Docker to connect: /run/docker/plugins/rbd.sock
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:440: INFO: Path request(createit) => /var/lib/docker-volumes/rbd/rbd/createit
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:269: INFO: Mount(createit)
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:681: INFO: rbdImageExists(rbd/createit)
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:289: WARN: Image does not exist: createit
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:135: INFO: createImage(createit)
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:681: INFO: rbdImageExists(rbd/createit)
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:705: INFO: Attempting to create RBD Image: (%s/%s, %s, %s) rbd createit 20480 xfs
rbd-volume-plugin: 2016/05/25 17:21:06 driver.go:818: INFO: lockImage(rbd/createit)

And this is the output of rbd-docker-plugin on the command line:

# rbd-docker-plugin --create true
rbd-volume-plugin: 2016/05/25 17:21:00 main.go:152: INFO: setting log file: /var/log/rbd-docker-plugin.log
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x0 pc=0x7f7108bda067]

runtime stack:
runtime.throw(0x8cbd80, 0x2a)
    /usr/local/go/src/runtime/panic.go:547 +0x90
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 34 [syscall, locked to thread]:
runtime.cgocall(0x704350, 0xc820049000, 0x10000c800000000)
    /usr/local/go/src/runtime/cgocall.go:123 +0x11b fp=0xc820048fc8 sp=0xc820048f98
github.com/ceph/go-ceph/rbd._Cfunc_rbd_lock_exclusive(0x7f70dc003e50, 0x7f70dc005e70, 0x0)
    ??:0 +0x41 fp=0xc820049000 sp=0xc820048fc8
github.com/ceph/go-ceph/rbd.(*Image).LockExclusive(0xc820049190, 0xc82010aa00, 0x15, 0x0, 0x0)
    /root/gocode/src/github.com/ceph/go-ceph/rbd/rbd.go:572 +0x15b fp=0xc820049080 sp=0xc820049000
main.(*cephRBDVolumeDriver).lockImage(0xc820049618, 0x84c5d0, 0x3, 0xc820106248, 0x8, 0x0, 0x0, 0x0, 0x0)
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/driver.go:839 +0x619 fp=0xc820049210 sp=0xc820049080
main.(*cephRBDVolumeDriver).createRBDImage(0xc820049618, 0x84c5d0, 0x3, 0xc820106248, 0x8, 0x5000, 0x84d280, 0x3, 0x0, 0x0)
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/driver.go:733 +0x89c fp=0xc8200494c0 sp=0xc820049210
main.cephRBDVolumeDriver.createImage(0x84c5d0, 0x3, 0x0, 0x0, 0x84a928, 0x5, 0x84c5d0, 0x3, 0xc8200de8a0, 0x1b, ...)
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/driver.go:165 +0xa01 fp=0xc820049618 sp=0xc8200494c0
main.cephRBDVolumeDriver.Mount(0x84c5d0, 0x3, 0x0, 0x0, 0x84a928, 0x5, 0x84c5d0, 0x3, 0xc8200de8a0, 0x1b, ...)
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/driver.go:290 +0x773 fp=0xc820049890 sp=0xc820049618
main.(*cephRBDVolumeDriver).Mount(0xc820098380, 0xc820106248, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    <autogenerated>:4 +0x11d fp=0xc820049970 sp=0xc820049890
github.com/docker/go-plugins-helpers/volume.(*Handler).initMux.func6(0xc820106248, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /root/gocode/src/github.com/docker/go-plugins-helpers/volume/api.go:91 +0xa6 fp=0xc8200499e8 sp=0xc820049970
github.com/docker/go-plugins-helpers/volume.(*Handler).handle.func1(0x7f71005d9180, 0xc8201245b0, 0xc8201200e0)
    /root/gocode/src/github.com/docker/go-plugins-helpers/volume/api.go:106 +0xd8 fp=0xc820049af8 sp=0xc8200499e8
net/http.HandlerFunc.ServeHTTP(0xc8200752b0, 0x7f71005d9180, 0xc8201245b0, 0xc8201200e0)
    /usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc820049b18 sp=0xc820049af8
net/http.(*ServeMux).ServeHTTP(0xc8200791a0, 0x7f71005d9180, 0xc8201245b0, 0xc8201200e0)
    /usr/local/go/src/net/http/server.go:1910 +0x17d fp=0xc820049b70 sp=0xc820049b18
net/http.serverHandler.ServeHTTP(0xc820098400, 0x7f71005d9180, 0xc8201245b0, 0xc8201200e0)
    /usr/local/go/src/net/http/server.go:2081 +0x19e fp=0xc820049bd0 sp=0xc820049b70
net/http.(*conn).serve(0xc820104100)
    /usr/local/go/src/net/http/server.go:1472 +0xf2e fp=0xc820049f98 sp=0xc820049bd0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820049fa0 sp=0xc820049f98
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f711bcc3930, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82006e990, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82006e990, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82006e930, 0x0, 0x7f711bcc39f0, 0xc82010a060)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*UnixListener).AcceptUnix(0xc8200dea40, 0xc82004d938, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:305 +0x53
net.(*UnixListener).Accept(0xc8200dea40, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:315 +0x41
net/http.(*Server).Serve(0xc820098400, 0x7f711bcc2988, 0xc8200dea40, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/docker/go-plugins-helpers/sdk.Handler.listenAndServe(0xc8200791a0, 0x84cff8, 0x4, 0xc8200de920, 0x1c, 0x0, 0x0, 0x0, 0x0)
    /root/gocode/src/github.com/docker/go-plugins-helpers/sdk/handler.go:82 +0x288
github.com/docker/go-plugins-helpers/sdk.Handler.ServeUnix(0xc8200791a0, 0x0, 0x0, 0xc8200de920, 0x1c, 0x0, 0x0)
    /root/gocode/src/github.com/docker/go-plugins-helpers/sdk/handler.go:48 +0x6d
main.main()
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/main.go:121 +0xe3f

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 20 [syscall]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 21 [select, locked to thread]:
runtime.gopark(0x8fd568, 0xc820025f28, 0x84c9d0, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820025f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820025f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 22 [chan receive]:
main.main.func1(0xc820072480, 0xc820098300, 0xc82007e030, 0xc820075100)
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/main.go:99 +0x72
created by main.main
    /root/gocode/src/github.com/yp-engineering/rbd-docker-plugin/main.go:118 +0xdf7

Note that the ceph device is created and locked by this client.

vincepii commented 8 years ago

I debugged this a bit and the problem is actually the C library called by ceph-go crashing (or how it is used in ceph-go maybe).

Anyway, I wanted to re-implement the calls using shell commands and noticed that this fork has that done already: https://github.com/porcupie/rbd-docker-plugin.

It seemed to work fine with the initial tests that we did.

@porcupie why don't you try to open a PR? Your fork seems to work well with Jewel!

porcupie commented 8 years ago

@vincepii I think you are correct that this is an issue with the go-ceph interface to the C-library Ceph libs. Due to issues with go-ceph, I have in that other fork completely removed go-ceph for all operations and replaced with shell equivalents; we are using it internally instead of this one. Still in discussions whether to replace yp-engineering/rbd-docker-plugin with porcupie/rbd-docker-plugin -- unfortunately, once code is out there it is hard to tell how many people rely on certain functionalities. However, since the rbd binary is also already used for some operations not provided by go-ceph, I think it might be possible to merge this up more easily since the functionality is the same or better (less buggy).

Feel free to use porcupie/rbd-docker-plugin or open issues/features since that fork is under more active development. However, I do tend to use it as a test-bed for new features (e.g. I also changed the way remove operation works so plugin can optionally just ignore remove API command). I'll try to be more intentional and create github releases so a stable tag can be pulled and I will also endeavor to merge up the removal of go-ceph usage to this project.

tsaridas commented 8 years ago

@porcupie could you enable issue posting in your fork ?

porcupie commented 8 years ago

Oops - thanks for heads up @tsaridas I didn't realize I had to enable 👍

porcupie commented 6 years ago

Closing since I merged my fork here.