scrapy-plugins / scrapy-zyte-api

Zyte API integration for Scrapy
BSD 3-Clause "New" or "Revised" License
36 stars 19 forks source link

Plugin don't work with latest version of aiohttp (3.10.6) #221

Closed Laerte closed 2 months ago

Laerte commented 2 months ago

We get RuntimeError exception with newer version:

2024-09-26 20:23:04 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy_zyte_api.handler.ScrapyZyteAPIDownloadHandler" for scheme "https"
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/scrapy/core/downloader/handlers/__init__.py", line 58, in _load_handler
    dh = create_instance(
  File "/usr/local/lib/python3.10/site-packages/scrapy/utils/misc.py", line 188, in create_instance
    instance = objcls.from_crawler(crawler, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/scrapy_zyte_api/handler.py", line 48, in from_crawler
    return cls(crawler.settings, crawler, client)
  File "/usr/local/lib/python3.10/site-packages/scrapy_zyte_api/handler.py", line 33, in __init__
    self._session = create_session()
  File "/usr/local/lib/python3.10/site-packages/zyte_api/_utils.py", line 24, in deprecated_create_session
    return create_session(connection_pool_size=connection_pool_size, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/zyte_api/_utils.py", line 31, in create_session
    kwargs["connector"] = TCPConnector(limit=connection_pool_size, force_close=True)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 826, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/site-packages/aiohttp/connector.py", line 245, in __init__
    loop = loop or asyncio.get_running_loop()
RuntimeError: no running event loop

Pinning aiohttp version to 3.9.5 make the plugin work again.

Laerte commented 2 months ago

Disregard, the version that my project was using was too old, after upgrade to latest version, aiohttp version works without issue.