search

asmodehn / celeros

Celery ROS python interface
0 stars 2 forks source link

flower sometimes stays alive after sigterm #20

Open asmodehn opened 8 years ago

asmodehn commented 8 years ago

Killing roslaunch doesn't kill flower sometimes. This makes restarting blind (no log) sometimes inconsistent.

We need to investigate the cause ( flower has some SIGTERM signal handler... )

asmodehn commented 8 years ago

Happened when doing consistent polling... maybe it s the cause ?

asmodehn commented 8 years ago

when that happens SIGKILL does kill it properly.

asmodehn commented 8 years ago 
[flower-11] killing on exit
[rostful-10] killing on exit
[concert/connection_cache-6] killing on exit
[concert/hub-5] killing on exit
[heartbeat_echo-8] killing on exit
[concert/gateway-7] killing on exit
[concert/interactions-4] killing on exit
[2016-02-16 18:46:25,107: INFO/MainProcess] signal_shutdown [atexit]
[WARN] [WallTime: 1455615985.107208] PyrosROS node stopped by keyboad interrupt
Shutdown initiated
[navi/map_locations-9] killing on exit
[E 160216 18:46:25 http1connection:53] Uncaught exception
    Traceback (most recent call last):
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/http1connection.py", line 234, in _read_message
        delegate.finish()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/httpserver.py", line 282, in finish
        self.delegate.finish()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1884, in finish
        self.execute()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1914, in execute
        self.handler._execute(transforms, *self.path_args, **self.path_kwargs)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1332, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 2607, in wrapper
        return method(self, *args, **kwargs)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/api/workers.py", line 175, in get
        yield self.update_cache(workername=workername)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/api/control.py", line 24, in update_cache
        app=self.application)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 187, in wrapper
        Runner(result, future, yielded)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 578, in __init__
        self.run()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 633, in run
        yielded = self.gen.send(value)
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/api/control.py", line 41, in update_workers
        if future.exception() is None:
      File "/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 432, in exception
        self._condition.wait(timeout)
      File "/usr/lib/python2.7/threading.py", line 339, in wait
        waiter.acquire()
    KeyboardInterrupt
[D 160216 18:46:25 control:29] Updating all worker's cache...
[W 160216 18:46:25 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '1b5bff86-a741-3306-b199-3c68d28be6ea.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbXX0=', 'headers': {'ticket': '678f239e-14dd-438f-8304-04ceb31d84cd', 'clock': 1342}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '1b5bff86-a741-3306-b199-3c68d28be6ea', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': '7ab57530-dd54-4f7e-8f8c-65a4a45d5339'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:25 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbImNlbGVyb3MuYXBwLmFkZF90b2dldGhlciIsICJjZWxlcm9zLmFwcC5sb25nX3Rhc2siLCAiZ29waGVyX3Rhc2tzLnRhc2tzLmRlbGl2ZXJ5IiwgImdvcGhlcl90YXNrcy50YXNrcy50ZXN0X2RlbGl2ZXJ5Il19', 'headers': {'ticket': 'ae7c6653-c7fe-42a2-8148-d1de40eec616', 'clock': 1340}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': '2d01481a-c787-43a4-9eab-75d15e473efc'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:25 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '897908da-c845-3216-987c-104c5842a4ff.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbeyJleGNsdXNpdmUiOiBmYWxzZSwgIm5hbWUiOiAiY2VsZXJ5IiwgImV4Y2hhbmdlIjogeyJuYW1lIjogImNlbGVyeSIsICJkdXJhYmxlIjogdHJ1ZSwgImRlbGl2ZXJ5X21vZGUiOiAyLCAicGFzc2l2ZSI6IGZhbHNlLCAiYXJndW1lbnRzIjogbnVsbCwgInR5cGUiOiAiZGlyZWN0IiwgImF1dG9fZGVsZXRlIjogZmFsc2V9LCAiZHVyYWJsZSI6IHRydWUsICJyb3V0aW5nX2tleSI6ICJjZWxlcnkiLCAiYWxpYXMiOiBudWxsLCAicXVldWVfYXJndW1lbnRzIjogbnVsbCwgImF1dG9fZGVsZXRlIjogZmFsc2UsICJiaW5kaW5nX2FyZ3VtZW50cyI6IG51bGwsICJiaW5kaW5ncyI6IFtdLCAibm9fYWNrIjogZmFsc2V9LCB7ImV4Y2x1c2l2ZSI6IGZhbHNlLCAibmFtZSI6ICJhc21vZGVobiIsICJleGNoYW5nZSI6IHsibmFtZSI6ICJhc21vZGVobiIsICJkdXJhYmxlIjogdHJ1ZSwgImRlbGl2ZXJ5X21vZGUiOiAyLCAicGFzc2l2ZSI6IGZhbHNlLCAiYXJndW1lbnRzIjogbnVsbCwgInR5cGUiOiAiZGlyZWN0IiwgImF1dG9fZGVsZXRlIjogZmFsc2V9LCAiZHVyYWJsZSI6IHRydWUsICJyb3V0aW5nX2tleSI6ICJhc21vZGVobiIsICJhbGlhcyI6IG51bGwsICJxdWV1ZV9hcmd1bWVudHMiOiBudWxsLCAiYXV0b19kZWxldGUiOiBmYWxzZSwgImJpbmRpbmdfYXJndW1lbnRzIjogbnVsbCwgImJpbmRpbmdzIjogW10sICJub19hY2siOiBmYWxzZX1dfQ==', 'headers': {'ticket': '0bde1d5f-9237-4350-86f6-f5429d4a8f4a', 'clock': 1338}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '897908da-c845-3216-987c-104c5842a4ff', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': 'b636556a-317e-49cc-bcf2-a6654ccf9804'}, 'content-encoding': 'utf-8'}",) res : None
[concert/concert_conductor_graph_to_string-3] killing on exit
[concert/conductor-2] killing on exit
[INFO] [WallTime: 1455615985.301924] Conductor : sending shutdown request [gateway/hub]
[INFO] [WallTime: 1455615985.304577] Gateway : shutting down.
[concert/master-1] killing on exit
[INFO] [WallTime: 1455615986.017323] Gateway : unregistered from the hub [GoCart Concert]
[INFO] [WallTime: 1455615986.021682] Hub : shutting down.
[W 160216 18:46:27 control:51] 'stats' inspect method failed
[W 160216 18:46:27 control:51] 'active_queues' inspect method failed
[W 160216 18:46:27 control:51] 'registered' inspect method failed
[W 160216 18:46:27 control:51] 'scheduled' inspect method failed
[D 160216 18:46:27 control:29] Updating all worker's cache...
[W 160216 18:46:30 control:51] 'registered' inspect method failed
[W 160216 18:46:30 control:51] 'scheduled' inspect method failed
[W 160216 18:46:30 control:51] 'active' inspect method failed
[W 160216 18:46:30 control:51] 'revoked' inspect method failed
[W 160216 18:46:30 control:51] 'conf' inspect method failed
[D 160216 18:46:30 control:29] Updating all worker's cache...
[W 160216 18:46:33 control:51] 'scheduled' inspect method failed
[D 160216 18:46:33 control:29] Updating all worker's cache...
[W 160216 18:46:33 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '9ad31130-bf8e-3b01-872f-fe6907aa9952.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbXX0=', 'headers': {'ticket': '0f30cbf6-1ae8-4879-900f-c078ad57d127', 'clock': 1394}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '9ad31130-bf8e-3b01-872f-fe6907aa9952', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': '23bc7f4c-07cc-41b7-8f91-b4d0b95e757d'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:33 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiB7InByZWZldGNoX2NvdW50IjogMSwgImNsb2NrIjogIjk4OCIsICJ0b3RhbCI6IHsiZ29waGVyX3Rhc2tzLnRhc2tzLnRlc3RfZGVsaXZlcnkiOiAxfSwgInBpZCI6IDgzNDcsICJicm9rZXIiOiB7Imhvc3RuYW1lIjogImNvbmNlcnQiLCAidXNlcmlkIjogbnVsbCwgInZpcnR1YWxfaG9zdCI6ICIwIiwgInBvcnQiOiA2Mzc5LCAiaW5zaXN0IjogZmFsc2UsICJzc2wiOiBmYWxzZSwgInRyYW5zcG9ydCI6ICJyZWRpcyIsICJjb25uZWN0X3RpbWVvdXQiOiA0LCAidHJhbnNwb3J0X29wdGlvbnMiOiB7fSwgImxvZ2luX21ldGhvZCI6IG51bGwsICJ1cmlfcHJlZml4IjogbnVsbCwgImhlYXJ0YmVhdCI6IG51bGwsICJhbHRlcm5hdGVzIjogW119LCAicG9vbCI6IHsicHJvY2Vzc2VzIjogWzk2ODldLCAibWF4LXRhc2tzLXBlci1jaGlsZCI6ICJOL0EiLCAidGltZW91dHMiOiBbMCwgMF0sICJ3cml0ZXMiOiB7InJhdyI6ICIxIiwgImFsbCI6ICIxMDAuMDAlIiwgInRvdGFsIjogMSwgImF2ZyI6ICIxMDAuMDAlIiwgImlucXVldWVzIjogeyJhY3RpdmUiOiAwLCAidG90YWwiOiAxfX0sICJwdXQtZ3VhcmRlZC1ieS1zZW1hcGhvcmUiOiBmYWxzZSwgIm1heC1jb25jdXJyZW5jeSI6IDF9LCAicnVzYWdlIjogeyJtYWpmbHQiOiAwLCAiaXhyc3MiOiAwLCAibWluZmx0IjogMzcxNjIsICJpbmJsb2NrIjogMCwgIm5zaWduYWxzIjogMCwgIm5zd2FwIjogMCwgImlkcnNzIjogMCwgIm1zZ3JjdiI6IDAsICJtYXhyc3MiOiA3MzM3MiwgImlzcnNzIjogMCwgIm52Y3N3IjogMjI1MiwgIm91YmxvY2siOiA4LCAic3RpbWUiOiAwLjE3MzY4NTk5OTk5OTk5OTk4LCAibXNnc25kIjogMCwgIm5pdmNzdyI6IDg2MCwgInV0aW1lIjogMS43NzcxMzV9fX0=', 'headers': {'ticket': 'cfc98fa5-f9d2-4723-a8c1-3e5e8bfbe5dd', 'clock': 1396}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': '9db59f09-30ff-4a41-b8f6-fed70e337d9d'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:33 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '1b5bff86-a741-3306-b199-3c68d28be6ea.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbeyJleGNsdXNpdmUiOiBmYWxzZSwgIm5hbWUiOiAiY2VsZXJ5IiwgImV4Y2hhbmdlIjogeyJuYW1lIjogImNlbGVyeSIsICJkdXJhYmxlIjogdHJ1ZSwgImRlbGl2ZXJ5X21vZGUiOiAyLCAicGFzc2l2ZSI6IGZhbHNlLCAiYXJndW1lbnRzIjogbnVsbCwgInR5cGUiOiAiZGlyZWN0IiwgImF1dG9fZGVsZXRlIjogZmFsc2V9LCAiZHVyYWJsZSI6IHRydWUsICJyb3V0aW5nX2tleSI6ICJjZWxlcnkiLCAiYWxpYXMiOiBudWxsLCAicXVldWVfYXJndW1lbnRzIjogbnVsbCwgImF1dG9fZGVsZXRlIjogZmFsc2UsICJiaW5kaW5nX2FyZ3VtZW50cyI6IG51bGwsICJiaW5kaW5ncyI6IFtdLCAibm9fYWNrIjogZmFsc2V9LCB7ImV4Y2x1c2l2ZSI6IGZhbHNlLCAibmFtZSI6ICJhc21vZGVobiIsICJleGNoYW5nZSI6IHsibmFtZSI6ICJhc21vZGVobiIsICJkdXJhYmxlIjogdHJ1ZSwgImRlbGl2ZXJ5X21vZGUiOiAyLCAicGFzc2l2ZSI6IGZhbHNlLCAiYXJndW1lbnRzIjogbnVsbCwgInR5cGUiOiAiZGlyZWN0IiwgImF1dG9fZGVsZXRlIjogZmFsc2V9LCAiZHVyYWJsZSI6IHRydWUsICJyb3V0aW5nX2tleSI6ICJhc21vZGVobiIsICJhbGlhcyI6IG51bGwsICJxdWV1ZV9hcmd1bWVudHMiOiBudWxsLCAiYXV0b19kZWxldGUiOiBmYWxzZSwgImJpbmRpbmdfYXJndW1lbnRzIjogbnVsbCwgImJpbmRpbmdzIjogW10sICJub19hY2siOiBmYWxzZX1dfQ==', 'headers': {'ticket': '5f25dd43-5276-45e9-b3db-8aa18616383e', 'clock': 1398}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '1b5bff86-a741-3306-b199-3c68d28be6ea', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': 'a49ff116-e0a5-4ae5-b658-3c53dd2caf39'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:35 control:51] 'stats' inspect method failed
[W 160216 18:46:35 control:51] 'active_queues' inspect method failed
[W 160216 18:46:35 control:51] 'registered' inspect method failed
[W 160216 18:46:35 control:51] 'scheduled' inspect method failed
[D 160216 18:46:37 control:29] Updating all worker's cache...
[W 160216 18:46:38 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbImNlbGVyb3MuYXBwLmFkZF90b2dldGhlciIsICJjZWxlcm9zLmFwcC5sb25nX3Rhc2siLCAiZ29waGVyX3Rhc2tzLnRhc2tzLmRlbGl2ZXJ5IiwgImdvcGhlcl90YXNrcy50YXNrcy50ZXN0X2RlbGl2ZXJ5Il19', 'headers': {'ticket': '364f93b7-be98-4fbb-8b53-2cb4d755f0d4', 'clock': 1418}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': 'e9409d93-2639-44f6-83b3-46c8ed674ff3'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:40 control:51] 'active_queues' inspect method failed
[W 160216 18:46:40 control:51] 'registered' inspect method failed
[W 160216 18:46:40 control:51] 'active' inspect method failed
[W 160216 18:46:40 control:51] 'reserved' inspect method failed
[W 160216 18:46:40 control:51] 'revoked' inspect method failed
[W 160216 18:46:40 control:51] 'conf' inspect method failed
[D 160216 18:46:40 control:29] Updating all worker's cache...
[flower-11] escalating to SIGTERM
[I 160216 18:46:40 command:90] SIGTERM detected, shutting down
[E 160216 18:46:40 http1connection:53] Uncaught exception
    Traceback (most recent call last):
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/http1connection.py", line 234, in _read_message
        delegate.finish()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/httpserver.py", line 282, in finish
        self.delegate.finish()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1884, in finish
        self.execute()
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1914, in execute
        self.handler._execute(transforms, *self.path_args, **self.path_kwargs)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1332, in _execute
        result = method(*self.path_args, **self.path_kwargs)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 2607, in wrapper
        return method(self, *args, **kwargs)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/api/workers.py", line 175, in get
        yield self.update_cache(workername=workername)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/api/control.py", line 24, in update_cache
        app=self.application)
      File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 175, in wrapper
        yielded = next(result)
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/api/control.py", line 41, in update_workers
        if future.exception() is None:
      File "/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 432, in exception
        self._condition.wait(timeout)
      File "/usr/lib/python2.7/threading.py", line 339, in wait
        waiter.acquire()
      File "/opt/groot/bootstrap_ws/src/celeros/deps/flower/flower/command.py", line 91, in sigterm_handler
        sys.exit(0)
    SystemExit: 0
[D 160216 18:46:40 control:29] Updating all worker's cache...
shutting down processing monitor...
... shutting down processing monitor complete
done
yujin@gocart-server:/opt/groot/concert_ws$ [D 160216 18:46:44 control:29] Updating all worker's cache...
[W 160216 18:46:44 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '1b5bff86-a741-3306-b199-3c68d28be6ea.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbXX0=', 'headers': {'ticket': '413b81fb-fa47-4260-b06f-4d4fe110fa85', 'clock': 1480}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '1b5bff86-a741-3306-b199-3c68d28be6ea', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': '58176867-042f-4270-bb5d-42730019a179'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:47 control:51] 'stats' inspect method failed
[W 160216 18:46:47 control:51] 'scheduled' inspect method failed
[W 160216 18:46:47 control:51] 'conf' inspect method failed
[D 160216 18:46:47 control:29] Updating all worker's cache...
[W 160216 18:46:50 control:51] 'registered' inspect method failed
[W 160216 18:46:50 control:51] 'scheduled' inspect method failed
[D 160216 18:46:50 control:29] Updating all worker's cache...
[W 160216 18:46:50 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '1b5bff86-a741-3306-b199-3c68d28be6ea.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbXX0=', 'headers': {'ticket': 'c462efce-4dad-4e8c-aa17-a9791e12f231', 'clock': 1523}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '1b5bff86-a741-3306-b199-3c68d28be6ea', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': 'bc085cf9-ceba-43cf-890b-747b6784b9c8'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:50 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiBbImNlbGVyb3MuYXBwLmFkZF90b2dldGhlciIsICJjZWxlcm9zLmFwcC5sb25nX3Rhc2siLCAiZ29waGVyX3Rhc2tzLnRhc2tzLmRlbGl2ZXJ5IiwgImdvcGhlcl90YXNrcy50YXNrcy50ZXN0X2RlbGl2ZXJ5Il19', 'headers': {'ticket': '12eadbcc-804c-4396-abab-3bd9d846ec91', 'clock': 1521}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': 'bd5021fe-fb3f-30c0-a50b-d7602e7bf2c6', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': 'e3451374-1a57-44e4-bb14-d9cfbaecb847'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:50 control:45] Future state : 'FINISHED' except : KeyError(u"Message for queue '897908da-c845-3216-987c-104c5842a4ff.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAYXNtb2RlaG4iOiB7InByZWZldGNoX2NvdW50IjogMSwgImNsb2NrIjogIjEwNjAiLCAidG90YWwiOiB7ImdvcGhlcl90YXNrcy50YXNrcy50ZXN0X2RlbGl2ZXJ5IjogMX0sICJwaWQiOiA4MzQ3LCAiYnJva2VyIjogeyJob3N0bmFtZSI6ICJjb25jZXJ0IiwgInVzZXJpZCI6IG51bGwsICJ2aXJ0dWFsX2hvc3QiOiAiMCIsICJwb3J0IjogNjM3OSwgImluc2lzdCI6IGZhbHNlLCAic3NsIjogZmFsc2UsICJ0cmFuc3BvcnQiOiAicmVkaXMiLCAiY29ubmVjdF90aW1lb3V0IjogNCwgInRyYW5zcG9ydF9vcHRpb25zIjoge30sICJsb2dpbl9tZXRob2QiOiBudWxsLCAidXJpX3ByZWZpeCI6IG51bGwsICJoZWFydGJlYXQiOiBudWxsLCAiYWx0ZXJuYXRlcyI6IFtdfSwgInBvb2wiOiB7InByb2Nlc3NlcyI6IFs5Njg5XSwgIm1heC10YXNrcy1wZXItY2hpbGQiOiAiTi9BIiwgInRpbWVvdXRzIjogWzAsIDBdLCAid3JpdGVzIjogeyJyYXciOiAiMSIsICJhbGwiOiAiMTAwLjAwJSIsICJ0b3RhbCI6IDEsICJhdmciOiAiMTAwLjAwJSIsICJpbnF1ZXVlcyI6IHsiYWN0aXZlIjogMCwgInRvdGFsIjogMX19LCAicHV0LWd1YXJkZWQtYnktc2VtYXBob3JlIjogZmFsc2UsICJtYXgtY29uY3VycmVuY3kiOiAxfSwgInJ1c2FnZSI6IHsibWFqZmx0IjogMCwgIml4cnNzIjogMCwgIm1pbmZsdCI6IDM3MjMwLCAiaW5ibG9jayI6IDAsICJuc2lnbmFscyI6IDAsICJuc3dhcCI6IDAsICJpZHJzcyI6IDAsICJtc2dyY3YiOiAwLCAibWF4cnNzIjogNzMzNzIsICJpc3JzcyI6IDAsICJudmNzdyI6IDIzOTYsICJvdWJsb2NrIjogOCwgInN0aW1lIjogMC4xODIzOTksICJtc2dzbmQiOiAwLCAibml2Y3N3IjogODYzLCAidXRpbWUiOiAxLjg1OTY4MTk5OTk5OTk5OTh9fX0=', 'headers': {'ticket': '74eaabe9-ce79-492e-b4e7-58502dcfbc1c', 'clock': 1519}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '897908da-c845-3216-987c-104c5842a4ff', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': '502fdb2c-fba7-4781-ab90-cd6c4b794d9f'}, 'content-encoding': 'utf-8'}",) res : None
[W 160216 18:46:52 control:51] 'stats' inspect method failed
[W 160216 18:46:52 control:51] 'active_queues' inspect method failed
[W 160216 18:46:52 control:51] 'registered' inspect method failed
[W 160216 18:46:52 control:51] 'scheduled' inspect method failed
[D 160216 18:46:52 control:29] Updating all worker's cache...