reiinakano / xcessiv

A web-based application for quick, scalable, and automated hyperparameter tuning and stacked ensembling in Python.
http://xcessiv.readthedocs.io
Apache License 2.0
1.27k stars 105 forks source link

ValueError: There exists an active worker named [Docker] #60

Open lucaventurini opened 6 years ago

lucaventurini commented 6 years ago

I'm using xcessiv inside docker, with docker-compose up. When I issue a docker restart (because, sometimes jobs get stuck, maybe this is another issue) I get the following:

xcessiv_1  | Process Process-2:
xcessiv_1  | Traceback (most recent call last):
xcessiv_1  |   File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
xcessiv_1  |     self.run()
xcessiv_1  |   File "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, in run
xcessiv_1  |     self._target(*self._args, **self._kwargs)
xcessiv_1  |   File "/usr/local/lib/python2.7/site-packages/xcessiv/scripts/runworker.py", line 17, in runworker
xcessiv_1  |     w.work()
xcessiv_1  |   File "/usr/local/lib/python2.7/site-packages/rq/worker.py", line 446, in work
xcessiv_1  |     self.register_birth()
xcessiv_1  |   File "/usr/local/lib/python2.7/site-packages/rq/worker.py", line 259, in register_birth
xcessiv_1  |     raise ValueError(msg.format(self.name))
xcessiv_1  | ValueError: There exists an active worker named u'4781ae60cd1b.26' already

I read that

By default, the name of a worker is equal to the concatenation of the current hostname and the current PID

and this is probably the problem with docker, which does not change either of the two things after a restart.

Has anybody any solution, apart restarting the whole machine?

lucaventurini commented 6 years ago

For the moment, I destroyed the containers with docker-compose rm and recreated them. I lost the packages I had installed manually, but it solved the problem for now. I think it must be some zombie process, should be related with this: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/