search

jxlil / scrapy-impersonate

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.
MIT License
78 stars 9 forks source link

Operation timed out after 30002 milliseconds #1

Closed jxlil closed 8 months ago

jxlil commented 9 months ago

In some cases it's possible to get a timeout error:

  File "/usr/local/lib/python3.9/site-packages/scrapy_impersonate/handler.py", line 45, in _download_request
    response = await self.client.request(**RequestParser(request).as_dict())  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/curl_cffi/requests/session.py", line 554, in request
    raise RequestsError(e)
curl_cffi.requests.errors.RequestsError: Failed to perform, ErrCode: 28, Reason:  \
'Operation timed out after 30002 milliseconds with 1096454 bytes received'

It should be possible to define the timeout for curl_cffi from scrapy.Request