Open github-actions[bot] opened 1 year ago
I need to read 7.2M files from an S3 bucket and do a simple data processing (filtering). For the data processing, I'm using Dask. I'm wondering how I can speed up the process.
Note that I use a machine with 64 processors and 500GB RAM.
Here's my cod…
Read the full article on the following website:
https://stackoverflow.com/questions/73841461/speeding-up-data-processing-with-dask
I need to read 7.2M files from an S3 bucket and do a simple data processing (filtering). For the data processing, I'm using Dask. I'm wondering how I can speed up the process.
Note that I use a machine with 64 processors and 500GB RAM.
Here's my cod…
Would you like to know more?
Read the full article on the following website:
https://stackoverflow.com/questions/73841461/speeding-up-data-processing-with-dask