search

pyros-dev / rostful

A lightweight package for providing and consuming ROS services, actions, and topics as RESTful web services
68 stars 41 forks source link

Worker refresh crashes #50

Closed heuristicus closed 8 years ago

heuristicus commented 9 years ago

Getting the following when running multiple simulated robots with web disabled, when trying to send a scheduled task.

Running roslaunch gopher_rocon_bootstrap gocart_sim.launch disable_zeroconf:=true robot_name:=test1 enable_web:=false --screen on two different masters (with different names), plus roslaunch gocart_concert concert.launch --screen disable_zeroconf:=true

[E 150904 16:15:44 web:1407] Uncaught exception GET /api/workers?refresh=1 (127.0.0.1) HTTPServerRequest(protocol='http', host='localhost:5555', method='GET', uri='/api/workers?refresh=1', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Origin': 'http://localhost:8888', 'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6', 'Accept-Encoding': 'gzip, deflate, sdch', 'Host': 'localhost:5555', 'Accept': '*/*', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/43.0.2357.130 Chrome/43.0.2357.130 Safari/537.36', 'Connection': 'keep-alive', 'Referer': 'http://localhost:8888/schedule', 'If-None-Match': '"45c9297eeb1a8255781767a94b0f4867db9c5527"', 'Content-Type': 'application/json;charset=UTF-8'}) Traceback (most recent call last): File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/web.py", line 1334, in _execute result = yield result File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 628, in run value = future.result() File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/concurrent.py", line 109, in result raise_exc_info(self._exc_info) File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 631, in run yielded = self.gen.throw(*sys.exc_info()) File "/opt/groot/concert_ws/src/flask-ext-catkin/src/flower/api/workers.py", line 62, in get yield self.update_cache(workername=workername) File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 628, in run value = future.result() File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/concurrent.py", line 109, in result raise_exc_info(self._exc_info) File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 631, in run yielded = self.gen.throw(*sys.exc_info()) File "/opt/groot/concert_ws/src/flask-ext-catkin/src/flower/api/control.py", line 23, in update_cache app=self.application) File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 628, in run value = future.result() File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/concurrent.py", line 109, in result raise_exc_info(self._exc_info) File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 631, in run yielded = self.gen.throw(*sys.exc_info()) File "/opt/groot/concert_ws/src/flask-ext-catkin/src/flower/api/control.py", line 38, in update_workers results = yield futures File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 628, in run value = future.result() File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/concurrent.py", line 109, in result raise_exc_info(self._exc_info) File "/opt/ros/indigo/lib/python2.7/dist-packages/tornado/gen.py", line 464, in callback result_list = [i.result() for i in children] File "/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 397, in result return self.__get_result() File "/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 356, in __get_result raise self._exception KeyError: u"Message for queue '9fac5d85-bee9-3346-aac2-9963e625160f.reply.celery.pidbox' without consumers: {'body': 'eyJjZWxlcnlAdGVzdDEiOiB7IkNFTEVSWV9BQ0tTX0xBVEUiOiB0cnVlLCAiQ0VMRVJZX1RBU0tfU0VSSUFMSVpFUiI6ICJqc29uIiwgIkNFTEVSWV9BTFdBWVNfRUFHRVIiOiBmYWxzZSwgIkNFTEVSWV9SRVNVTFRfU0VSSUFMSVpFUiI6ICJqc29uIiwgIkNFTEVSWV9BQ0NFUFRfQ09OVEVOVCI6IFsiYXBwbGljYXRpb24vanNvbiJdLCAiQ0VMRVJZX0lOQ0xVREUiOiBbImdvcGhlcl90YXNrcy50YXNrcyIsICJjZWxlcnkuYXBwLmJ1aWx0aW5zIiwgImNlbGVyb3MuYXBwIl0sICJURVNUSU5HIjogZmFsc2UsICJDRUxFUllfUkVESVNfU0NIRURVTEVSX1VSTCI6ICJyZWRpczovL2xvY2FsaG9zdDo2Mzc5LzIiLCAiQ0VMRVJZRF9DT05DVVJSRU5DWSI6IDEsICJDRUxFUllfVFJBQ0tfU1RBUlRFRCI6IHRydWUsICJDRUxFUllEX1BSRUZFVENIX01VTFRJUExJRVIiOiAxLCAiREVCVUciOiBmYWxzZSwgIkNFTEVSWUJFQVRfU1lOQ19FVkVSWSI6IDEsICJDRUxFUllCRUFUX01BWF9MT09QX0lOVEVSVkFMIjogMzAsICJCUk9LRVJfVVJMIjogInJlZGlzOi8vbG9jYWxob3N0OjYzNzkvLyIsICJDRUxFUllfUkVESVNfU0NIRURVTEVSX0tFWV9QUkVGSVgiOiAiKioqKioqKioiLCAiQ0VMRVJZX1JFU1VMVF9CQUNLRU5EIjogInJlZGlzOi8vbG9jYWxob3N0OjYzNzkifX0=', 'headers': {'ticket': 'fbd7ec48-7cac-4042-a2e3-d1aab73cc4d4', 'clock': 727}, 'content-type': 'application/json', 'properties': {'body_encoding': 'base64', 'delivery_info': {'priority': 0, 'routing_key': '9fac5d85-bee9-3346-aac2-9963e625160f', 'exchange': 'reply.celery.pidbox'}, 'delivery_mode': 1, 'delivery_tag': 'd475d7f8-8a35-4cd7-9d25-85c0aa304a61'}, 'content-encoding': 'utf-8'}"

Usually get something like

[2015-09-04 16:17:08,972: DEBUG/MainProcess] pidbox received method dump_revoked() [reply_to:{u'routing_key': u'fd6abf95-cae7-3053-9f3f-32ab873b96ee', u'exchange': u'reply.celery.pidbox'} ticket:04ea368e-e173-4633-90c9-2b5c88b4f560] on the receiving end, but after the crash it becomes [2015-09-04 16:17:12,646: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]

Seems like that is normal behaviour though?

asmodehn commented 9 years ago

This is more related to celeros I think.

After a quick research, I suspect this is a bug in celery itself. Refreshing workers on flower will sometime trigger this.

https://github.com/mher/flower/issues/455

I have yet to reproduce it but I believe triggering all inspect methods in celery simultaneously, while using a redis broker (or backend ?), will sometime produce the error.

asmodehn commented 8 years ago

Not a rostful issue. closing