Closed honzajavorek closed 6 months ago
I figured out I don't need whole response, so I was able to fix this by a change like this: https://github.com/juniorguru/plucker/commit/a0cabe8f14a4f9959051c18fa09297b15bbb9d27
Thank you @honzajavorek for reporting this. I've opened a PR https://github.com/apify/apify-sdk-python/pull/191 which should improve the error handling in to_apify_request
. Also, the ApifyScheduler
should let the user know, that the request was not scheduled due to this reason.
My spider https://github.com/juniorguru/plucker/blob/26d1758e310b8b2451541516cf4447e4a5e4a11a/juniorguru_plucker/jobs_jobscz/spider.py runs just fine with Scrapy, but fails with critical errors when teaming up with Apify.
See exception details 💌
``` [twisted] CRITICAL Unhandled error in Deferred: Traceback (most recent call last): File "/Users/honza/.local/share/mise/installs/python/3.11/lib/python3.11/asyncio/events.py", line 84, in _run self._context.run(self._callback, *self._args) File "/Users/honza/Projects/juniorguru-plucker/.venv/lib/python3.11/site-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer self.runUntilCurrent() File "/Users/honza/Projects/juniorguru-plucker/.venv/lib/python3.11/site-packages/twisted/internet/base.py", line 994, in runUntilCurrent call.func(*call.args, **call.kw) File "/Users/honza/Projects/juniorguru-plucker/.venv/lib/python3.11/site-packages/twisted/internet/task.py", line 680, in _tick taskObj._oneWorkUnit() ---When debugging the problem, I figured out the following line causes the problem:
Inspecting problematic dicts, the culprit seems to be the fact that I pass a response object around:
Then the response comes in the dict like this:
The
<200 https://example.com/.../>
is a representation of the Response, which probably cannot be pickled, or at least some Selector objects in there.I don't think you can do much about it, it's probably a limitation of delegating the request mechanics to an external system such as Apify. If you need to serialize and later deserialize the request, there's just no way I could pass around something which Python cannot pickle.
So I think the only solution here is to fail nicely. The line which pickles the request should catch the exception and provide a nicer error message which explains what is happening and why, ideally with some guidance on how to avoid the problem. I'll get back here if I come up with a workaround.