Closed marcopasqua closed 5 hours ago
This can happen when the commoncrawl bucket is under heavy traffic or you yourself are sending a lot of requests. I recommend using a large number of tasks (possibly even 1 task = 1 file) so that when you have these errors you can just relaunch the processing and it will run only the missing tasks/files, without wasting a lot of compute
Hi,
I have implemented a pipeline to process the Common Crawl (CC) data, similar to the FineWeb example in the example folder. The main issue I'm encountering is that, when reading files from CC, the connection sometimes times out, causing the execution to stop.
Here is the error message I receive:
Is it possible to change some parameters to mitigate this problem?
Thanks!