twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.48k stars 703 forks source link

Fix race between jvm shutdown and `writer.finished` #1938

Closed ttim closed 3 years ago

ttim commented 3 years ago

Currently writer.finished happens in onComplete callback on Future result in Exucution. However since onComplete isn't being called before future being resolved and called asynchroniously after future being resolved, it leads to a race and runtime error:

To fix this behaviour I made onComplete logic to happen before result future get resolved by changing onComplete to andThen

tlazaro commented 3 years ago

LGTM, both onComplete vs andThen are for performing side-effects when the Future is done. But andThen makes it so the side-effect runs before anything else. Seems like exactly what we wanted in this case.

ttim commented 3 years ago

@johnynek does it seem reasonable to you? I'm wondering is there something I missing