The Apify SDK for Python is the official library for creating Apify Actors in Python. It provides useful features like actor lifecycle management, local storage emulation, and actor event handling.
from typing import Generator
from scrapy import Request, Spider as BaseSpider
from scrapy.http import TextResponse
class DontFilterSpider(BaseSpider):
name = 'dont-filter'
def start_requests(self) -> Generator[Request, None, None]:
for _ in range(3):
yield Request('https://httpbin.org/get', method='GET', dont_filter=True)
def parse(self, response: TextResponse) -> Generator[dict, None, None]:
yield {'something': True}
And src/main.py to execute these spiders with Apify:
from __future__ import annotations
from scrapy.crawler import CrawlerProcess
from apify import Actor
from apify.scrapy.utils import apply_apify_settings
from .spiders.yield_post import YieldPostSpider as Spider
# from .spiders.dont_filter import DontFilterSpider as Spider
async def main() -> None:
"""Apify Actor main coroutine for executing the Scrapy spider."""
async with Actor:
Actor.log.info('Actor is being executed...')
# Apply Apify settings, it will override the Scrapy project settings
settings = apply_apify_settings()
# Execute the spider using Scrapy CrawlerProcess
process = CrawlerProcess(settings, install_root_handler=False)
process.crawl(Spider)
process.start()
Description
Request Queue
RequestQueue.add_request
method by improving its unique key generation logic.add_request
.apify/_utils.py
:get_short_base64_hash
based on the _hashPayload.normalize_url
based on the normalizeUrl.compute_unique_key
based on the _computeUniqueKey.Scrapy integration
scrapy.Request.dont_filter
field in theto_apify_request
.Issues
Testing
Unit tests
Manual testing / execution
The
YieldPostSpider
tests the case of the POST requests to the same URL.The
DontFilterSpider
tests the case of the Scrapy request withdont_filter
option.And
src/main.py
to execute these spiders with Apify:Execute it using Scrapy:
And using Apify (need to change the Spider in the
main.py
manually):And it produces the same output :tada:.