search

syndicate-storage / syndicate

Internet-scale software-defined storage system
Apache License 2.0
56 stars 10 forks source link

File modification detection at Hadoop #41

Closed iychoi closed 10 years ago

iychoi commented 11 years ago

Add a feature to read Extended Attributes at JSyndicateFS And JSyndicateFS will expose File version and Block version to Hadoop. Hadoop will check this version at each Mapper (Reader) to verify this file is the same as InputSplitter has used in a previous stage.

iychoi commented 10 years ago

Hadoop will check last file modification time at InputSplitter and RecordReader.