search

twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Fix `FileNotFoundException` in some cases in `VersionedStore#getAllVersions` #1859

Closed ttim closed 6 years ago

ttim commented 6 years ago

This is cherry-pick of https://github.com/twitter/scalding/pull/1858

Some jobs in twitter sometimes fail with FileNotFoundException due to our replication process - during replication temporary file is created in destination folder (and later renamed). If getAllVersions method executes listDir while temporary file exists and later executes getFileSystem().getFileStatus(p) while this temporary file renamed exception is thrown and job fails.

Solution is to move getFileStatus closer to block where it's used and where it's guarded by try-catch.

CLAassistant commented 6 years ago

CLA assistant check
All committers have signed the CLA.