search

google-code-export / amcat

Automatically exported from code.google.com/p/amcat
0 stars 0 forks source link

amcat-dev can't seem to handle 3 celery workers #629

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Running daily.py with 3 celery workers leads to the following:

- They start of well
- After a minute or so, amcat-dev gets really slow, logging in shows: 'System 
information disabled due to load higher than 8.0'
- After 3 minutes, the broker (rabbitmq-server) crashes.

Note: After recreating this problem, shut down all workers, run 'manage.py 
celery purge', then re-start the broker (sudo rabbitmq-server) to effectively 
clear all tasks.

Original issue reported on code.google.com by Toon.Alfrink@gmail.com on 5 Nov 2013 at 3:41

GoogleCodeExporter commented 9 years ago 
start off*...

Note: ElasticSearch also seems to crash under the high load for it was not 
running shortly after this error.

Original comment by Toon.Alfrink@gmail.com on 5 Nov 2013 at 3:58

GoogleCodeExporter commented 9 years ago 
This was due to a number of memory-consuming bugs in scraping, fixed.

Original comment by Toon.Alfrink@gmail.com on 22 Jan 2014 at 4:36