niklasb / dryscrape

[not actively maintained] A lightweight Python library that uses Webkit to enable easy scraping of dynamic, Javascript-heavy web pages
http://dryscrape.readthedocs.io/
MIT License
533 stars 67 forks source link

multiprocessing with dryscrape #50

Closed deter3 closed 8 years ago

deter3 commented 8 years ago

This is not an issue , but sharing how to deal with xvfb .

I am using celery to run multiple sessions of webkit-server . I need to remove ' dryscrape.start_xvfb() ' in the code and put xvfb-run before celery command , like this

xvfb-run -a celery -A googlescrape worker --loglevel=INFO --concurrency=10 -Q googlescrape_task

I spent couple of hours to figure out and hope can help somebody to save the time .