Scrapyscript is a Python library you can use to run Scrapy spiders directly from your code. Scrapy is a great framework to use for scraping projects, but sometimes you don't need the whole framework, and just want to run a small spider from a script or a Celery job. That's where Scrapyscript comes in.
With Scrapyscript, you can:
Job
Job(s)
in a Processor
processor.run()
to execute them... returning all results when the last job completes.
Let's see an example.
import scrapy
from scrapyscript import Job, Processor
processor = Processor(settings=None)
class PythonSpider(scrapy.spiders.Spider):
name = "myspider"
def start_requests(self):
yield scrapy.Request(self.url)
def parse(self, response):
data = response.xpath("//title/text()").extract_first()
return {'title': data}
job = Job(PythonSpider, url="http://www.python.org")
results = processor.run(job)
print(results)
[{ "title": "Welcome to Python.org" }]
See the examples directory for more, including a complete Celery
example.
pip install scrapyscript
A single request to call a spider, optionally passing in *args or **kwargs, which will be passed through to the spider constructor at runtime.
# url will be available as self.url inside MySpider at runtime
myjob = Job(MySpider, url='http://www.github.com')
Create a multiprocessing reactor for running spiders. Optionally provide a scrapy.settings.Settings
object to configure the Scrapy runtime.
settings = scrapy.settings.Settings(values={'LOG_LEVEL': 'WARNING'})
processor = Processor(settings=settings)
Start the Scrapy engine, and execute one or more jobs. Blocks and returns consolidated results in a single list.
jobs
can be a single instance of Job
, or a list.
results = processor.run(myjob)
or
results = processor.run([myjob1, myjob2, ...])
As per the scrapy docs, a Spider
must return an iterable of Request
and/or dict
or Item
objects.
Requests will be consumed by Scrapy inside the Job
. dict
or scrapy.Item
objects will be queued
and output together when all spiders are finished.
Due to the way billiard handles communication between processes, each dict
or Item
must be
pickle-able using pickle protocol 0. It's generally best to output dict
objects from your Spider.
Updates, additional features or bug fixes are always welcome.
git clone git@github.com:jschnurr/scrapyscript.git
poetry install
make test
or make tox
See CHANGELOG.md
The MIT License (MIT). See LICENCE file for details.