search

twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

AsyncBase forceExtraFutures wait strategy is suboptimal #669

Closed pankajroark closed 8 years ago

pankajroark commented 8 years ago

The way AsyncBase waits is that it identifies futures over the limit and then waits for all of them to finish. Some of these futures may take much longer than others, e.g. a service call that times out, while others futures not in the toForce list may finish first. So even though the overall number of outstanding futures may be lower we still keep waiting for every single toForce future to finish. End result is that we don't wait just until we come down to MaxWaitingFutures but longer, thus under utilizing resources.

I'm working on a fix for this but any suggestions are welcome.

pankajroark commented 8 years ago

Also, it seems that failure of any single future would have removed the await, which doesn't seem right.

johnynek commented 8 years ago

closed by #670