search

rayon-rs / rayon

Rayon: A data parallelism library for Rust
Apache License 2.0
10.99k stars 501 forks source link

How Rayon read data from kafka? #821

Open rts-gordon opened 3 years ago

rts-gordon commented 3 years ago

Hi there, Rayon is a great product for parallel computing. I use Kafka as an unbound data source, read Kafka data with "for loop", so I can't use Rayon function sucha s par_iter for parallel computing. Would you like to give me some examples with Rayon::par_iter to read data from Kafka? Thank you very much.

Regards CHCP

cuviper commented 3 years ago

If that for loop is using a normal Iterator for the Kafka data, then you can use ParallelBridge like:

your_iterator.par_bridge().for_each(|item| {
    // parallel work
});

Another option is to keep your current loop and spawn your CPU-intensive work into Rayon, either static rayon::spawn or within a rayon::scope.

rts-gordon commented 3 years ago

I will try these options, thanks a lot for your help. @cuviper