search

coiled / dask-community

Issue tracker for the Dask community team
MIT License
2 stars 0 forks source link

[Discourse] Dask Bag significantly faster with `scheduler='processes'`, help me understand why? #1162

Open github-actions[bot] opened 2 years ago

github-actions[bot] commented 2 years ago

Sorry, I’m still quite new to Dask, and trying to figure out what was going on here, and why using the processes scheduler is faster, and uses less memory, I understand bags use the multiprocessor by default.

Basically I’m processing some common craw…

Would you like to know more?

Read the full article on the following website:

https://dask.discourse.group/t/dask-bag-significantly-faster-with-scheduler-processes-help-me-understand-why/881

pavithraes commented 2 years ago

Ian has answered