viant / toolbox

Toolbox - go utility library
Apache License 2.0
197 stars 29 forks source link

Adding a `processSliceAsync` function ? #20

Closed atakanyenel closed 5 years ago

atakanyenel commented 5 years ago

I think it would make sense to add a function that while iterating a slice, creates a goroutine for each element , then returns the total result once every routine is complete. This would increase the performance for big slices.

The logic of stopping if handler returns false can still be implemented with context or maybe we just say, if handler returns false, don't send it through channel, thus don't add it to slice ?

What do you think ?

adranwit commented 5 years ago

Very interesting, I think this is a good idea,

how about throttling number of active goroutines in case each does some memory extensive processing?

do you have some poc ?

atakanyenel commented 5 years ago

I don't but I can do a PoC pull request in the following days with tests, and we can move from there if it looks plausible.

I'm not sure about throttling, as I had the idea from javascripts map function which does things Async, but we can put extra measures to make sure we are not overloading cpu.

adranwit commented 5 years ago

Sound good,

adranwit commented 5 years ago

Pull request has been merge, Thanks,