Gallaecio
commented
2 years ago Can you provide a minimal reproducible example?
For example, the following minimal spider code should work:
from scrapy import Request, Spider
class MinimalSpider(Spider):
name = "minimal"
custom_settings = {
"DOWNLOAD_HANDLERS": {
"http": "scrapy_zyte_api.handler.ScrapyZyteAPIDownloadHandler",
"https": "scrapy_zyte_api.handler.ScrapyZyteAPIDownloadHandler",
},
"ZYTE_API_KEY": "YOUR_API_KEY",
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}
def start_requests(self):
yield Request(
"https://toscrape.com",
meta={"zyte_api": {"httpResponseBody": True}},
)
def parse(self, response):
passYou should be able to store this code in a file, set your API key, and run that file with scrapy runspider.
Assuming it works as expected, what do you need to do to make it fail the way you actual code is failing?
ttilberg
Following the notes for the settings file, we are experiencing an issue where the
httpandhttpshandlers are not loading as expected. Generically, we are receiving the exception:The object should be created from async function.There are log lines mentioning asyncio, and
aiohttppaths referenced, so it seems like we are successfully loading AsyncIO. Do you have any thoughts on what this could be?Relevant log lines:
When you trace the code, you find that the
httpandhttpskeys are dropped from the downloaders dict after the first exception, and the second exception is raised because the dict no longer has those keys.