async-rs / parallel-stream

Data parallelism library for async-std.
https://docs.rs/parallel-stream
Apache License 2.0
93 stars 16 forks source link

Method: limit #1

Open yoshuawuyts opened 4 years ago

yoshuawuyts commented 4 years ago

It should be possible to limit the max concurrency. This likely needs to be built into the ParallelStream trait itself, with each impl being responsible for carrying it.

The default limit should be unlimited, with the executor and runtime scheduling resources.

API Outline

trait ParallelStream {
    fn limit(self, limit: impl Into<Option<usize>>) -> Self;
}

Usage

// Set max limit to 5
let out: Vec<usize> = parallel_stream::repeat(5)
    .take(100)
    .limit(5) // <-- set max concurrency
    .map(async |n| n * n)
    .collect()
    .await;

// Remove max limit
let out: Vec<usize> = parallel_stream::repeat(5)
    .take(100)
    .limit(None) // <-- set max concurrency
    .map(async |n| n * n)
    .collect()
    .await;

Naming

The name limit is inspired by the parameter name of for_each_concurrent. It seems small and to the point, though not particularly attached if there's something different that's concise yet to the point.

ccope commented 3 years ago

It would be nice if you could specify a number of requests within a time period as well.