Up until 9/23/15 we had per-task logging of celery tasks, then it went to just logging errors. We should get full logging turned back on. Here's where it stopped:
[2015-09-23 14:29:15,794: INFO/MainProcess] Received task: perma.tasks.proxy_capture[da398a13-99ba-4c6b-adeb-241ce7eae843]
[2015-09-23 14:29:15,794: INFO/MainProcess] Scaling up 1 processes.
[2015-09-23 14:29:15,830: WARNING/Worker-589] S9FY-CD97: Fetching http://www.law360.com/articles/547183/sec-could-bring-more-insider-trading-cases-in-house
[2015-09-23 14:29:16,147: WARNING/Worker-589] WarcProx opened.
[2015-09-23 14:29:16,148: WARNING/Worker-589] Fetching url.
[2015-09-23 14:29:18,910: WARNING/Worker-589] Finished fetching url.
[2015-09-23 14:29:20,961: WARNING/Worker-589] Fetching favicon from http://www.law360.com/favicon.ico ...
[2015-09-23 14:29:21,001: WARNING/Worker-589] Getting title.
[2015-09-23 14:29:21,005: WARNING/Worker-589] Fetching robots.txt ...
[2015-09-23 14:29:21,013: WARNING/Worker-589] Checking meta tags.
[2015-09-23 14:29:21,175: WARNING/Worker-589] Couldn't reach robots.txt
[2015-09-23 14:29:21,189: WARNING/Worker-589] Waiting for post-load requests.
[2015-09-23 14:29:26,216: WARNING/Worker-589] Taking screenshot.
[2015-09-23 14:29:26,965: WARNING/Worker-589] Shutting down browser and proxies.
[2015-09-23 14:29:27,514: WARNING/Worker-589] Saving WARC.
[2015-09-23 14:29:27,873: WARNING/Worker-589] Saved favicon at http://www.law360.com/favicon.ico
[2015-09-23 14:29:27,873: WARNING/Worker-589] Writing CDX lines to the DB
[2015-09-23 14:29:28,004: WARNING/Worker-589] S9FY-CD97 capture done.
[2015-09-23 14:29:28,006: INFO/MainProcess] Task perma.tasks.proxy_capture[da398a13-99ba-4c6b-adeb-241ce7eae843] succeeded in 12.1989553533s: None
[2015-09-23 14:30:07,832: INFO/MainProcess] Scaling down -1 processes.
[2015-10-03 17:46:33,301: ERROR/MainProcess] Process 'Worker-4518' pid:6014 exited with 'signal 9 (SIGKILL)'
[2015-10-06 00:48:22,175: ERROR/MainProcess] Process 'Worker-5680' pid:2977 exited with 'signal 9 (SIGKILL)'
[2015-10-06 05:00:39,704: ERROR/MainProcess] Process 'Worker-5803' pid:27356 exited with 'signal 9 (SIGKILL)'
[2015-10-12 19:28:53,761: ERROR/MainProcess] Process 'Worker-8771' pid:2522 exited with 'signal 9 (SIGKILL)'
So, what happened to the "Task succeeded" auto-logging, and why aren't our print statements being logged anymore?
jcushman
commented
8 years ago
Up until 9/23/15 we had per-task logging of celery tasks, then it went to just logging errors. We should get full logging turned back on. Here's where it stopped:
So, what happened to the "Task succeeded" auto-logging, and why aren't our print statements being logged anymore?
@anastasia mentioned: "http://blog.mapado.com/task-specific-logging-in-celery/ this has a rundown of some of the things that have worked for me, logging-wise"