search

holgerd77 / django-dynamic-scraper

Creating Scrapy scrapers via the Django admin interface
http://django-dynamic-scraper.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.15k stars 310 forks source link

Scheduler does not work in example project #80

Closed zeonn closed 7 years ago

zeonn commented 8 years ago

Trying to test celery scheduler with example project (from this repo).

$ python manage.py celeryd -l info -B --settings=example_project.settings
$ scrapyd

and after:

INFO/MainProcess] Received task: open_news.tasks.run_spiders[25f39d93-dd33-4bb8-8b8f-86d34f4e5fe9]
INFO/MainProcess] Task open_news.tasks.run_spiders[25f39d93-dd33-4bb8-8b8f-86d34f4e5fe9] succeeded in 0.181797763001s: None

nothing more happens... from scrapyd i receive:

"POST /schedule.json HTTP/1.1" 200 68 "-" "-"
"GET /listjobs.json?project=default HTTP/1.1" 200 85 "-" "Python-urllib/2.7"
"POST /schedule.json HTTP/1.1" 200 68 "-" "-"

and nothing more

Python 2.7.12 amqp==1.4.9 anyjson==0.3.3 attrs==16.2.0 billiard==3.3.0.23 celery==3.1.17 cffi==1.8.3 cryptography==1.5 cssselect==0.9.2 decorator==4.0.10 Django==1.9 django-celery==3.1.17 django-dynamic-scraper==0.11.2 enum34==1.1.6 future==0.15.2 idna==2.1 ipaddress==1.0.17 jsonpath-rw==1.4.0 kombu==3.0.35 lxml==3.6.4 parsel==1.0.3 Pillow==3.3.1 ply==3.9 pyasn1==0.1.9 pyasn1-modules==0.0.8 pycparser==2.14 PyDispatcher==2.0.5 pyOpenSSL==16.1.0 pytz==2016.6.1 queuelib==1.4.2 Scrapy==1.1.2 scrapy-djangoitem==1.1.1 scrapyd==1.1.0 service-identity==16.0.0 six==1.10.0 Twisted==16.4.1 w3lib==1.15.0 zope.interface==4.3.2

zeonn commented 8 years ago

I found the cause. The problem was in line "LOG_STDOUT = True" in file "./open_news/scrapper/settings.py" When i changed it on False, or just deleted it, scheduler worked fine. Why? It's broke my mind!

holgerd77 commented 7 years ago

Sorry for ignoring this so long, I had the same behaviour when testing this now and was (still am) as puzzled as you.

I will set the setting to False in the example project in upcoming v.0.12.2 release and added a note in the docs to avoid further confusion.

Will close this issue anyhow due to relatively low priority. Thanks for reporting!