Closed hermit-crab closed 4 years ago
Due to https://github.com/scrapinghub/python-scrapinghub/issues/121. Simply put s3 upload happens during job items iteration and each time it does a delay is introduced which can kill the iterator.
Might be better to wait till next python-scrapinghub release which may address the issue before fixing it here.
python-scrapinghub
PR merged
kalessin
commented
4 years ago kalessin
commented
4 years ago
Due to https://github.com/scrapinghub/python-scrapinghub/issues/121. Simply put s3 upload happens during job items iteration and each time it does a delay is introduced which can kill the iterator.
Might be better to wait till next
python-scrapinghubrelease which may address the issue before fixing it here.