Currently we have a Python Django application hosted in Heroku cloud that uses Celery workers for background processing.
Celery workers were using django DB as the broker. Recently we decided to use IronMq as the broker since it’s an enterprise solution. The packages we are using are
We are using two workers. One worker is for the queue ‘Normal priority’ and another worker is for the queue HighPriority’. But when the application tired to start the worker for ‘HighPriority’ queue an error was triggered and as a result both the workers where crashed.
Following are additional settings we are using to configure IronMQ and celery queues.
Currently we have a Python Django application hosted in Heroku cloud that uses Celery workers for background processing.
Celery workers were using django DB as the broker. Recently we decided to use IronMq as the broker since it’s an enterprise solution. The packages we are using are
Python 2.7 Django 1.3 Celery 3.0.16 Django-celery 3.0.11 Iron-celery 0.3.1
We are using two workers. One worker is for the queue ‘Normal priority’ and another worker is for the queue HighPriority’. But when the application tired to start the worker for ‘HighPriority’ queue an error was triggered and as a result both the workers where crashed.
Following are additional settings we are using to configure IronMQ and celery queues.
BROKER_URL='ironmq://xxxxx:xxxxx@' CELERY_RESULT_BACKEND='ironcache://xxxxx:xxxxx@'
import os import djcelery import iron_celery from kombu import Exchange, Queue djcelery.setup_loader()
CELERY_DEFAULT_QUEUE = 'NormalPriority' CELERY_QUEUES = { CELERY_DEFAULT_QUEUE: { 'exchange': CELERY_DEFAULT_QUEUE, 'binding_key': CELERY_DEFAULT_QUEUE, } }
CELERY_CREATE_MISSING_QUEUES=True CELERY_ROUTES = {'tasks.retrieveDocketUsingWorker': {'queue': 'HighPriority'}, 'tasks.retrieveDocketFromDdUsingWorker': {'queue': 'HighPriority'} }
We have attached the errorlog with this mail.
Will you please treat this as a high priority as we our development is ceased due to this .
Please see the error stack
2013-03-14T09:20:02+00:00 heroku[celeryd.1]: Starting process with command
pyth on DocketService/manage.py celeryd -E -B --loglevel=INFO
2013-03-14T09:20:03+00:00 heroku[celeryd.1]: State changed from starting to up 2013-03-14T09:20:06+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=3ms service=32ms status=200 bytes=60 2013-03-14T09:20:07+00:00 app[celeryd.1]: 2013-03-14T09:20:07+00:00 app[celeryd.1]: -------------- celery@bf00e27d-a0f8-4 d83-850b-06c03bb229c6 v3.0.16 (Chiastic Slide) 2013-03-14T09:20:07+00:00 app[celeryd.1]: ---- * ----- 2013-03-14T09:20:07+00:00 app[celeryd.1]: --- \ * * -- [Configuration] 2013-03-14T09:20:07+00:00 app[celeryd.1]: -- * - * --- . broker: ironmq: //4f792df2c032005c17019c8b@localhost// 2013-03-14T09:20:07+00:00 app[celeryd.1]: - * ---------- . app: default :0x2a53690 (djcelery.loaders.DjangoLoader) 2013-03-14T09:20:07+00:00 app[celeryd.1]: - * ---------- . concurrency: 4 (proc esses) 2013-03-14T09:20:07+00:00 app[celeryd.1]: - * ---------- . events: ON 2013-03-14T09:20:07+00:00 app[celeryd.1]: - \ ---------- 2013-03-14T09:20:07+00:00 app[celeryd.1]: - * --- * --- [Queues] 2013-03-14T09:20:07+00:00 app[celeryd.1]: -- *** ---- . NormalPriority: exch ange:NormalPriority(direct) binding:NormalPriority 2013-03-14T09:20:07+00:00 app[celeryd.1]: --- *\ ----- 2013-03-14T09:20:07+00:00 app[celeryd.1]: 2013-03-14T09:20:07+00:00 app[celeryd.1]: [Tasks] 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.DocketCa ching.cacheDocketEvent 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.DocketCa ching.cacheFullDocket 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.DocketCa ching.setIsPurchased 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.DocketDo wnloader.retrieveDocketFromDdUsingWorker 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.DocketDo wnloader.retrieveDocketUsingWorker 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.SearchIn dexUpdater.updateDocketIndexByCaseID 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.SearchIn dexUpdater.updateDocumentIndexByDocumentID 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.SyncDrop Box.syncDropBoxByNEFNoticeID 2013-03-14T09:20:07+00:00 app[celeryd.1]: . DocketService.DelayedJobs.SyncDrop Box.syncDropBoxByUserID 2013-03-14T09:20:07+00:00 app[celeryd.1]: 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,787: INFO/Beat] C elerybeat: Starting... 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,812: WARNING/Beat ] Reset: Account for new version field 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,812: WARNING/Beat ] Reset: Account for new tz field 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,825: WARNING/Beat ] Reset: Account for new utc_enabled field 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,885: WARNING/Main Process] /app/.heroku/python/lib/python2.7/site-packages/djcelery/loaders.py:132 : UserWarning: Using settings.DEBUG leads to a memory leak, never use this setti ng in production environments! 2013-03-14T09:20:07+00:00 app[celeryd.1]: warnings.warn("Using settings.DEBUG leads to a memory leak, never " 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,885: WARNING/Main Process] celery@bf00e27d-a0f8-4d83-850b-06c03bb229c6 ready. 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,887: INFO/MainPro cess] consumer: Connected to ironmq://4f792df2c032005c17019c8b@localhost//. 2013-03-14T09:20:07+00:00 app[celeryd.1]: [2013-03-14 05:20:07,909: INFO/MainPro cess] Starting new HTTPS connection (1): mq-aws-us-east-1.iron.io 2013-03-14T09:20:08+00:00 app[celeryd.1]: [2013-03-14 05:20:08,375: ERROR/MainPr ocess] Unrecoverable error: TypeError('string indices must be integers',) 2013-03-14T09:20:08+00:00 app[celeryd.1]: Traceback (most recent call last): 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/celery/worker/init.py", line 356, in start 2013-03-14T09:20:08+00:00 app[celeryd.1]: component.start() 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/celery/worker/consumer.py", line 392, in start 2013-03-14T09:20:08+00:00 app[celeryd.1]: self.reset_connection() 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/celery/worker/consumer.py", line 742, in reset_connection 2013-03-14T09:20:08+00:00 app[celeryd.1]: self.connection, on_decode_error=s elf.on_decode_error, 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/celery/app/amqp.py", line 307, in init 2013-03-14T09:20:08+00:00 app[celeryd.1]: queues or self.app.amqp.queues.con sume_from.values(), kw 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/kombu/messaging.py", line 338, in init 2013-03-14T09:20:08+00:00 app[celeryd.1]: self.revive(self.channel) 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/kombu/messaging.py", line 350, in revive 2013-03-14T09:20:08+00:00 app[celeryd.1]: self.declare() 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/kombu/messaging.py", line 360, in declare 2013-03-14T09:20:08+00:00 app[celeryd.1]: queue.declare() 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/kombu/entity.py", line 471, in declare 2013-03-14T09:20:08+00:00 app[celeryd.1]: self.queue_declare(nowait, passive =False) 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/kombu/entity.py", line 497, in queue_declare 2013-03-14T09:20:08+00:00 app[celeryd.1]: nowait=nowait) 2013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/kombu/transport/virtual/init.py", line 401, in queue_decla re 2013-03-14T09:20:08+00:00 app[celeryd.1]: return queue, self._size(queue), 02013-03-14T09:20:08+00:00 app[celeryd.1]: File "/app/.heroku/python/lib/python 2.7/site-packages/iron_celery/iron_mq_transport.py", line 78, in _size 2013-03-14T09:20:08+00:00 app[celeryd.1]: return details["size"] 2013-03-14T09:20:08+00:00 app[celeryd.1]: TypeError: string indices must be inte gers 2013-03-14T09:20:08+00:00 app[celeryd.1]: [2013-03-14 05:20:08,907: INFO/MainPro cess] Celerybeat: Shutting down... 2013-03-14T09:20:10+00:00 heroku[celeryd.1]: Process exited with status 0 2013-03-14T09:20:10+00:00 heroku[celeryd.1]: State changed from up to crashed 2013-03-14T09:20:11+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=61ms status=200 bytes=60 2013-03-14T09:20:16+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=24ms status=200 bytes=60 2013-03-14T09:20:24+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=2914ms status=200 bytes=60 2013-03-14T09:20:29+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=2ms service=2863ms status=200 bytes=60 2013-03-14T09:20:31+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=371ms status=200 bytes=60 2013-03-14T09:20:36+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=30ms status=200 bytes=60 2013-03-14T09:20:41+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=35ms status=200 bytes=60 2013-03-14T09:20:46+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=30ms status=200 bytes=60 2013-03-14T09:20:51+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=223ms status=200 bytes=60 2013-03-14T09:20:56+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=402ms status=200 bytes=60 2013-03-14T09:20:56+00:00 heroku[celeryd1.1]: State changed from crashed to star ting 2013-03-14T09:21:01+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=337ms status=200 bytes=60 2013-03-14T09:21:06+00:00 heroku[celeryd1.1]: Starting process with command
pyt hon DocketService/manage.py celeryd -E -B -Q HighPriority --loglevel=INFO
2013-03-14T09:21:07+00:00 heroku[celeryd1.1]: State changed from starting to up 2013-03-14T09:21:11+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=5743ms status=200 bytes=60 2013-03-14T09:21:16+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=246ms status=200 bytes=60 2013-03-14T09:21:16+00:00 app[celeryd1.1]: 2013-03-14T09:21:16+00:00 app[celeryd1.1]: -------------- celery@6876eb18-7abb- 45fd-ab45-3d36266c3fec v3.0.16 (Chiastic Slide) 2013-03-14T09:21:16+00:00 app[celeryd1.1]: ---- * ----- 2013-03-14T09:21:16+00:00 app[celeryd1.1]: --- \ * * -- [Configuration] 2013-03-14T09:21:16+00:00 app[celeryd1.1]: -- * - * --- . broker: ironmq ://4f792df2c032005c17019c8b@localhost// 2013-03-14T09:21:16+00:00 app[celeryd1.1]: - * ---------- . app: defaul t:0x2557690 (djcelery.loaders.DjangoLoader) 2013-03-14T09:21:16+00:00 app[celeryd1.1]: - * ---------- . concurrency: 4 (pro cesses) 2013-03-14T09:21:16+00:00 app[celeryd1.1]: - * ---------- . events: ON 2013-03-14T09:21:16+00:00 app[celeryd1.1]: - \ ---------- 2013-03-14T09:21:16+00:00 app[celeryd1.1]: - * --- * --- [Queues] 2013-03-14T09:21:16+00:00 app[celeryd1.1]: -- *** ---- . HighPriority: excha nge:HighPriority(direct) binding:HighPriority 2013-03-14T09:21:16+00:00 app[celeryd1.1]: --- *\ ----- 2013-03-14T09:21:16+00:00 app[celeryd1.1]: 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [Tasks] 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.DocketC aching.cacheDocketEvent 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.DocketC aching.cacheFullDocket 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.DocketC aching.setIsPurchased 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.DocketD ownloader.retrieveDocketFromDdUsingWorker 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.DocketD ownloader.retrieveDocketUsingWorker 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.SearchI ndexUpdater.updateDocketIndexByCaseID 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.SearchI ndexUpdater.updateDocumentIndexByDocumentID 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.SyncDro pBox.syncDropBoxByNEFNoticeID 2013-03-14T09:21:16+00:00 app[celeryd1.1]: . DocketService.DelayedJobs.SyncDro pBox.syncDropBoxByUserID 2013-03-14T09:21:16+00:00 app[celeryd1.1]: 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,482: INFO/Beat] Celerybeat: Starting... 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,518: WARNING/Bea t] Reset: Account for new version field 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,531: WARNING/Bea t] Reset: Account for new tz field 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,531: WARNING/Bea t] Reset: Account for new utc_enabled field 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,600: WARNING/Mai nProcess] /app/.heroku/python/lib/python2.7/site-packages/djcelery/loaders.py:13 2: UserWarning: Using settings.DEBUG leads to a memory leak, never use this sett ing in production environments! 2013-03-14T09:21:16+00:00 app[celeryd1.1]: warnings.warn("Using settings.DEBUG leads to a memory leak, never " 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,601: WARNING/Mai nProcess] celery@6876eb18-7abb-45fd-ab45-3d36266c3fec ready. 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,603: INFO/MainPr ocess] consumer: Connected to ironmq://4f792df2c032005c17019c8b@localhost//. 2013-03-14T09:21:16+00:00 app[celeryd1.1]: [2013-03-14 05:21:16,626: INFO/MainPr ocess] Starting new HTTPS connection (1): mq-aws-us-east-1.iron.io 2013-03-14T09:21:17+00:00 app[celeryd1.1]: [2013-03-14 05:21:17,041: ERROR/MainP rocess] Unrecoverable error: TypeError('string indices must be integers',) 2013-03-14T09:21:17+00:00 app[celeryd1.1]: Traceback (most recent call last): 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/celery/worker/init.py", line 356, in start 2013-03-14T09:21:17+00:00 app[celeryd1.1]: component.start() 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/celery/worker/consumer.py", line 392, in start 2013-03-14T09:21:17+00:00 app[celeryd1.1]: self.reset_connection() 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/celery/worker/consumer.py", line 742, in reset_connection 2013-03-14T09:21:17+00:00 app[celeryd1.1]: self.connection, on_decode_error= self.on_decode_error, 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/celery/app/amqp.py", line 307, in init 2013-03-14T09:21:17+00:00 app[celeryd1.1]: queues or self.app.amqp.queues.co nsume_from.values(), kw 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/kombu/messaging.py", line 338, in init 2013-03-14T09:21:17+00:00 app[celeryd1.1]: self.revive(self.channel) 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/kombu/messaging.py", line 350, in revive 2013-03-14T09:21:17+00:00 app[celeryd1.1]: self.declare() 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/kombu/messaging.py", line 360, in declare 2013-03-14T09:21:17+00:00 app[celeryd1.1]: queue.declare() 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/kombu/entity.py", line 471, in declare 2013-03-14T09:21:17+00:00 app[celeryd1.1]: self.queue_declare(nowait, passiv e=False) 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/kombu/entity.py", line 497, in queue_declare 2013-03-14T09:21:17+00:00 app[celeryd1.1]: nowait=nowait) 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/kombu/transport/virtual/init.py", line 401, in queue_decl are 2013-03-14T09:21:17+00:00 app[celeryd1.1]: return queue, self._size(queue), 0 2013-03-14T09:21:17+00:00 app[celeryd1.1]: File "/app/.heroku/python/lib/pytho n2.7/site-packages/iron_celery/iron_mq_transport.py", line 78, in _size 2013-03-14T09:21:17+00:00 app[celeryd1.1]: return details["size"] 2013-03-14T09:21:17+00:00 app[celeryd1.1]: TypeError: string indices must be int egers 2013-03-14T09:21:17+00:00 app[celeryd1.1]: [2013-03-14 05:21:17,631: INFO/MainPr ocess] Celerybeat: Shutting down... 2013-03-14T09:21:18+00:00 heroku[celeryd1.1]: Process exited with status 0 2013-03-14T09:21:18+00:00 heroku[celeryd1.1]: State changed from up to crashed 2013-03-14T09:21:23+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=2447ms status=200 bytes=60 2013-03-14T09:21:26+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=23ms status=200 bytes=60 2013-03-14T09:21:31+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=196ms status=200 bytes=60 2013-03-14T09:21:36+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=5ms service=35ms status=200 bytes=60 2013-03-14T09:21:41+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=34ms status=200 bytes=60 2013-03-14T09:21:46+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=517ms status=200 bytes=60 2013-03-14T09:21:47+00:00 heroku[router]: at=info method=GET path=/ host=www.ire landdev.com fwd="69.64.56.47" dyno=web.1 queue=0 wait=0ms connect=1ms service=4m s status=302 bytes=5 2013-03-14T09:21:47+00:00 heroku[router]: at=info method=GET path=/login/ host=w ww.irelanddev.com fwd="69.64.56.47" dyno=web.1 queue=0 wait=0ms connect=1ms serv ice=54ms status=200 bytes=2562 2013-03-14T09:21:51+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=30ms status=200 bytes=60 2013-03-14T09:21:57+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=1658ms status=200 bytes=60 2013-03-14T09:22:01+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=1ms service=37ms status=200 bytes=60 2013-03-14T09:22:06+00:00 heroku[router]: at=info method=POST path=/getDocketDow nloaderWorkerStatus/ host=www.irelanddev.com fwd="115.119.213.182" dyno=web.1 qu eue=0 wait=0ms connect=0ms service=35ms status=200 bytes=60Thanks in advance.