Issue with "docker restart"

[cwesthues]

I encounter the following problem/issue with ubiquity:

Services docker, ubiquity and ubiquity-docker-plugin are up and running without any error. However, when I do "systemctl docker restart", docker daemon/service runs fine for few minutes and then fails. In the log file ubiquity.log, I can see that the Scale NSD servers are contacted and that e.g. mmlscluster and mmlsmount are successfull.

Any idea what this (mis-)behaviour is caused by?

Here are my ubiquity config files, attached is the log file ubiquity.log:

ubiquity-server.conf: port = 9999
logPath = "/tmp" configPath = "/tmp" defaultBackend = "spectrum-scale"

[SpectrumScaleConfig] defaultFilesystemName = "gpfs" nfsServerAddr = "worker1" forceDelete = false

ubiquity-client.conf: logPath = "/tmp" backends = ["spectrum-scale"]

[DockerPlugin] port = 9000 pluginsDirectory = "/etc/docker/plugins/"

[UbiquityServer] address = "worker1.ibm.edu" port = 9999

Here is, for completeness, "systemctl status docker" AFTER the restart:

[root@worker1 ubiquity]# systemctl status -l docker ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2017-08-09 21:03:31 CEST; 17min ago Docs: https://docs.docker.com Process: 27453 ExecStart=/usr/bin/dockerd (code=exited, status=2) Main PID: 27453 (code=exited, status=2) Memory: 146.0M CGroup: /system.slice/docker.service ├─24488 docker-containerd-shim 5bb8d3e8bac0384551cb2c96d6d7bb1278da2d6c527ca63b95c1572c5043bf4e /var/run/docker/libcontainerd/5bb8d3e8bac0384551cb2c96d6d7bb1278da2d6c527ca63b95c1572c5043bf4e docker-runc ├─27569 docker-containerd-shim b42afeaa61ee9a4037a6df6e2aa2b42d4a940c29c0821e32bae8043e25d90e19 /var/run/docker/libcontainerd/b42afeaa61ee9a4037a6df6e2aa2b42d4a940c29c0821e32bae8043e25d90e19 docker-runc ├─28396 docker-containerd-shim 260386f932cc177c6d6200624d6c0d077c70d30f8cfc5a523c24f791672e4805 /var/run/docker/libcontainerd/260386f932cc177c6d6200624d6c0d077c70d30f8cfc5a523c24f791672e4805 docker-runc ├─28494 docker-containerd-shim 0d43fdc7e1227b616407ea260c231a247587482331cecd20b51c8ee8236e225f /var/run/docker/libcontainerd/0d43fdc7e1227b616407ea260c231a247587482331cecd20b51c8ee8236e225f docker-runc └─28552 docker-containerd-shim de4523a97086d355e36c0ce75ea04944fd1cd2cd84caaf3b577053085521d62b /var/run/docker/libcontainerd/de4523a97086d355e36c0ce75ea04944fd1cd2cd84caaf3b577053085521d62b docker-runc

Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/discovery.go:80 +0xa65 Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: github.com/docker/docker/pkg/plugins.loadWithRetry(0xc8214992a4, 0x8, 0xc821397f00, 0x0, 0x0, 0x0) Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/plugins.go:172 +0xa2 Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: github.com/docker/docker/pkg/plugins.GetAll.func1(0xc821397f80, 0xc821561200, 0xc8214992a4, 0x8) Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/plugins.go:255 +0x5f Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: created by github.com/docker/docker/pkg/plugins.GetAll Aug 09 21:03:31 worker1.ibm.edu dockerd[27453]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/plugins.go:257 +0x5a5 Aug 09 21:03:31 worker1.ibm.edu systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 09 21:03:31 worker1.ibm.edu systemd[1]: Unit docker.service entered failed state. Aug 09 21:03:31 worker1.ibm.edu systemd[1]: docker.service failed.

ubiquity.log.txt

[midoblgsm]

hi, @cwesthues , can you give the plugin side logs? it should be in your /tmp/ubiquity-docker-plugin.log

[cwesthues]

Hello Mohamed,   it is pretty boring:   /tmp/ubiquity-docker-plugin.log: ubiquity-docker-plugin: 2017/08/09 21:02:19 server.go:55: Starting server...ubiquity-docker-plugin: 2017/08/09 21:02:19 server.go:72: Started http server on 127.0.0.1:9000ubiquity-docker-plugin: 2017/08/09 21:02:48 server.go:55: Starting server...ubiquity-docker-plugin: 2017/08/09 21:02:48 server.go:72: Started http server on 127.0.0.1:9000   Is there a way to increase verbosity?   Thanks in advance,   Christof Christof Westhues IBM Storage Solutions Technical SME PPP Phone: +49 173 6413918 E-mail: cwesthues@de.ibm.com     ----- Ursprüngliche Nachricht -----Von: Mohamed Mohamed notifications@github.comAn: IBM/ubiquity-docker-plugin ubiquity-docker-plugin@noreply.github.comCC: cwesthues cwesthues@de.ibm.com, Mention mention@noreply.github.comBetreff: Re: [IBM/ubiquity-docker-plugin] Issue with "docker restart" (#40)Datum: Mi, 9. Aug 2017 22:44  hi, @cwesthues , can you give the plugin side logs? it should be in your /tmp/ubiquity-docker-plugin.log —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

 

[shay-berman]

Hi @cwesthues

Do you see any special errors in the "/var/log/messages" or in "journalctl -u docker"? if so please upload to the issue.

Thanks

[cwesthues]

Hi,   yes, seems we are coming closer, I see "panic" messages with both journalctl and in /var/log/messages:   ... Aug 10 08:14:36 worker1.ibm.edu dockerd[781]: time="2017-08-10T08:14:36.386308201+02:00" level=error msg="HandleAug 10 08:15:12 worker1.ibm.edu dockerd[781]: time="2017-08-10T08:15:12.106794834+02:00" level=error msg="HandleAug 10 08:15:13 worker1.ibm.edu dockerd[781]: time="2017-08-10T08:15:13.054619103+02:00" level=error msg="HandleAug 10 08:15:13 worker1.ibm.edu dockerd[781]: time="2017-08-10T08:15:13.788582682+02:00" level=error msg="HandleAug 10 08:15:13 worker1.ibm.edu dockerd[781]: panic: runtime error: invalid memory address or nil pointer derefeAug 10 08:15:13 worker1.ibm.edu dockerd[781]: [signal 0xb code=0x1 addr=0x8 pc=0xc51c31]Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: goroutine 335 [running]:Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: panic(0x1a6fa00, 0xc82000a080)Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: /usr/local/go/src/runtime/panic.go:481 +0x3e6Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: github.com/docker/docker/pkg/plugins.readPluginJSONInfo(0xc8214030Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/dAug 10 08:15:13 worker1.ibm.edu dockerd[781]: github.com/docker/docker/pkg/plugins.(localRegistry).Plugin(0xc82Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/dAug 10 08:15:13 worker1.ibm.edu dockerd[781]: github.com/docker/docker/pkg/plugins.loadWithRetry(0xc821403094, 0Aug 10 08:15:13 worker1.ibm.edu dockerd[781]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/dAug 10 08:15:13 worker1.ibm.edu dockerd[781]: github.com/docker/docker/pkg/plugins.GetAll.func1(0xc821495d80, 0xAug 10 08:15:13 worker1.ibm.edu dockerd[781]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/dAug 10 08:15:13 worker1.ibm.edu dockerd[781]: created by github.com/docker/docker/pkg/plugins.GetAllAug 10 08:15:13 worker1.ibm.edu dockerd[781]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/dAug 10 08:15:13 worker1.ibm.edu systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARAug 10 08:15:13 worker1.ibm.edu systemd[1]: Unit docker.service entered failed state.Aug 10 08:15:13 worker1.ibm.edu systemd[1]: docker.service failed. ...   And this is /var/log/messages: ... Aug 10 08:15:13 worker1 kernel: XFS (dm-6): Mounting V5 FilesystemAug 10 08:15:13 worker1 kernel: XFS (dm-6): Ending clean mountAug 10 08:15:13 worker1 dockerd: panic: runtime error: invalid memory address or nil pointer dereferenceAug 10 08:15:13 worker1 dockerd: [signal 0xb code=0x1 addr=0x8 pc=0xc51c31]Aug 10 08:15:13 worker1 dockerd: goroutine 335 [running]:Aug 10 08:15:13 worker1 dockerd: panic(0x1a6fa00, 0xc82000a080)Aug 10 08:15:13 worker1 dockerd: /usr/local/go/src/runtime/panic.go:481 +0x3e6Aug 10 08:15:13 worker1 dockerd: github.com/docker/docker/pkg/plugins.readPluginJSONInfo(0xc821403094, 0x8, 0xc821403380, 0x21, 0x0, 0x0, 0x0)Aug 10 08:15:13 worker1 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/discovery.go:119 +0x221Aug 10 08:15:13 worker1 dockerd: github.com/docker/docker/pkg/plugins.(localRegistry).Plugin(0xc820a47e70, 0xc821403094, 0x8, 0x46a400, 0x0, 0x0)Aug 10 08:15:13 worker1 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/discovery.go:80 +0xa65Aug 10 08:15:13 worker1 dockerd: github.com/docker/docker/pkg/plugins.loadWithRetry(0xc821403094, 0x8, 0xc821495d00, 0x1d4ebc8, 0x0, 0x0)Aug 10 08:15:13 worker1 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/plugins.go:172 +0xa2Aug 10 08:15:13 worker1 dockerd: github.com/docker/docker/pkg/plugins.GetAll.func1(0xc821495d80, 0xc82142d200, 0xc821403094, 0x8)Aug 10 08:15:13 worker1 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/plugins.go:255 +0x5fAug 10 08:15:13 worker1 dockerd: created by github.com/docker/docker/pkg/plugins.GetAllAug 10 08:15:13 worker1 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/plugins/plugins.go:257 +0x5a5Aug 10 08:15:13 worker1 systemd: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENTAug 10 08:15:13 worker1 systemd: Unit docker.service entered failed state.Aug 10 08:15:13 worker1 systemd: docker.service failed. ...   Regards,   Christof     Christof Westhues IBM Storage Solutions Technical SME PPP Phone: +49 173 6413918 E-mail: cwesthues@de.ibm.com     ----- Ursprüngliche Nachricht -----Von: shay-berman notifications@github.comAn: IBM/ubiquity-docker-plugin ubiquity-docker-plugin@noreply.github.comCC: cwesthues cwesthues@de.ibm.com, Mention mention@noreply.github.comBetreff: Re: [IBM/ubiquity-docker-plugin] Issue with "docker restart" (#40)Datum: Do, 10. Aug 2017 08:26  Hi @cwesthues Do you see any special errors in the "/var/log/messages" or in "journalctl -u docker"?if so please upload to the issue. Thanks —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

 

[shay-berman]

Hi @cwesthues

Thanks for the details.

It looks like Docker known issue -> https://github.com/moby/moby/issues/22081

What is your

1. docker EE version?
2. operating system version?
3. which ubiquity release version?

Thanks

[cwesthues]

Dear all,   I went from docker-engine-1.12.0-1 to docker-ce-17.0.6.0.ce-1, and the issue went away. Thanks for your quick help, very much appreciated!!!   Regards,   Christof Christof Westhues IBM Storage Solutions Technical SME PPP Phone: +49 173 6413918 E-mail: cwesthues@de.ibm.com     ----- Ursprüngliche Nachricht -----Von: shay-berman notifications@github.comAn: IBM/ubiquity-docker-plugin ubiquity-docker-plugin@noreply.github.comCC: cwesthues cwesthues@de.ibm.com, Mention mention@noreply.github.comBetreff: Re: [IBM/ubiquity-docker-plugin] Issue with "docker restart" (#40)Datum: Do, 10. Aug 2017 08:52  Hi @cwesthues Thanks for the details. It looks like Docker known issue -> moby/moby#22081 What is your docker EE version? operating system version? which ubiquity release version? Thanks —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

 

[shay-berman]

Cool

Feel free to contact us any time if you have issue or question about Ubiquity.

@cwesthues, since the issue resolved, please close this issue.

Thanks Shay Berman IBM