search

nathanmarz / dfs-datastores

Dead-simple vertical partitioning, compression, appends, and consolidation of data on a distributed filesystem.
BSD 3-Clause "New" or "Revised" License
215 stars 82 forks source link

override getModifiedTime for Cascading >= 2.1 #27

Open sorenmacbeth opened 11 years ago

sorenmacbeth commented 11 years ago

The default SkipStrategy in Cascading 2.1 and greater uses the timestamp of the directory returned by getPath(), which in the case of a Pail doesn't change. We need to override getModifiedTime to return 0L when used as a sink and something else (currentTime maybe) when used as a source.

http://docs.cascading.org/cascading/2.1/userguide/html/ch03s08.html#N20CA7