search

jschnurr / scrapyscript

Run a Scrapy spider programmatically from a script or a Celery task - no project required.
MIT License
121 stars 26 forks source link

Added support for billiard 3.5+ used in Celery 4+ #1

Closed mrge closed 7 years ago

brooks-macbeth-shadowdragon commented 7 years ago

Simple fix for the TypeError coming from pickle. It is coming from the Job.from_xpath function. In parse you need to extract the data from response.xpath or else it will not be pickled properly. Granted, this is an issue on Scrapy's end, but this fixes the issue now.

In scrapyscript.py

48 def parse(self, response):
49    return {'data': response.xpath(self._xpath).extract()}

In tests.py

28 self.assertEqual(results[0]['data'][0],
29                  'Welcome to Python.org')

I have a fix on my end, but am unable to create a new pull request.