Open williamhakim10 opened 4 months ago
I will try some load test to check, tbh implementation is quite old and probably not super efficient as it relying on threadpool for async behind the scene. I think it's simpler to build a new a implem on top of native async client and leverage out-of-the-box asyncio feature
As an aside, I wanted to point out that it feels like the very inconsistent use of typing throughout the library makes it very hard to debug what's actually going on and reason about what errors can happen and when.
Yeah, that's due to the current implementation, we can't really throw in async way with async for, the only way is to return exception as result, also I made the choice to disable the throw in concurrency mode https://github.com/scrapfly/python-scrapfly/blob/master/scrapfly/client.py#L369 (same issue with asyncio.gather which require return_exception
to not stop everything and result in the same typing experience)
We have code that looks like this:
That being said, this code tends to explode on the async iterator sometimes, which will throw an error which looks like this without returning a result at all.
Seems like there's some kind of bug where the async iterator can itself throw rather than return an exception, which means the entire process blows up. Any ideas how we might go about fixing?
As an aside, I wanted to point out that it feels like the very inconsistent use of typing throughout the library makes it very hard to debug what's actually going on and reason about what errors can happen and when.