search

rayon-rs / rayon

Rayon: A data parallelism library for Rust
Apache License 2.0
11.04k stars 501 forks source link

idea: allow control of splitting depth / number of jobs #476

Open cuviper opened 6 years ago

cuviper commented 6 years ago

We currently have with_min_len and with_max_len to control the length of the split iterators. However, this only works on an IndexedParallelIterator that knows its length in the first place.

Perhaps we could offer different controls for the splitting depth, or the number of jobs (~2depth), which could work on any parallel iterator. This may line up more directly to how folks think about their tasks, especially if they want to split exactly into the number of threads.

brandonros commented 2 years ago

Is it possible with this library to have for example controlled network requests in parallel? Can you use ThreadPoolBuilder / with_max_len to make sure no more than 4 requests are ever fired at once in parallel or is that level of control not possible?

cuviper commented 2 years ago

See my reply in https://github.com/rayon-rs/rayon/issues/963#issuecomment-1273687781.