Cookbook ideas for threadpool crate

[ericho]

Issue to gather ideas for the threadpool crate

edit:

• [x] Draw fractal dispatching work to a thread pool
• [x] calculate SHA1 in all *.iso files in a folder
  • [ ] TODO: generate an IO bound task that can be dispatched. Ideally should not have external dependencies.

[KodrAus]

Dispatching some blocking io to a threadpool and returning a Result on a channel?

[budziq]

• Generate thumbnails in parallel - use walkdir to find all image files in subdirs and create scaled down thumbnails next to originals with image crate. Any other interesting file processing might do here in fact.
• Gather output from multiple child processes - Manage a pool of child processes doing some IO bound work (like a list of git clone addresses)
• Process memory mapped file in parallel - memmap a csv (or any easily divisible) file and process the chunks against a transform (csv reader/regex filter/regex replace) in a threadpool. Chunking here might be tricky. Depending on the example the results might need to be sorted afterwards.

[yandexx]

• Calculating checksum for all files (or by a glob) in a folder; this task is commonly CPU-bound.

[ericho]

Just extending @yandexx suggestion, an example could be calculate the checksum of all files in a folder tree using walkdir.

[iamdanfox]

Trying to think of non-file system traversing examples here... what about overlaying some text onto an animated gif? Each frame in the gif format could be processed completely independent and the piston.rs/image/gif crate can already encode/decode nicely.

[budziq]

@iamdanfox I'm not sure if this idea is significantly different form the fractal drawing example (which is a little artificial in itself). Could you point how this example would be different (except for per frame vs per pixel parallelism) or maybe you have some other cool ideas?

[AndyGauge]

I'd like to expand the examples in threadpool to do something Io bound that would be beneficial to a wider audience.

[AndyGauge]

Maybe we could check for clients listening on port 80. Say a request to github.io in each thread to verify the port is responding. I'm thinking using essentially this: https://doc.rust-lang.org/std/net/struct.TcpStream.html#examples to check each of these addresses: 185.199.X.153 where X is 108,109,110,111.

[budziq]

Maybe we could check for clients listening on port 80.

This seams a little artificial. I think that original approach of "we have a list of crates and what can we do with these" might be suboptimal in some cases (especially relatively low level crates such as this one). How about we post issues asking for contributors ideas for given domain instead of given crate?

[KodrAus]

@budziq Focusing on domains rather than specific libraries sounds like a good approach to me and probably more in line with the new organization too.