search

nathanmarz / dfs-datastores

Dead-simple vertical partitioning, compression, appends, and consolidation of data on a distributed filesystem.
BSD 3-Clause "New" or "Revised" License
215 stars 82 forks source link

JobConf Configuration should inherit from FileSystem's Configuration #8

Closed bm3780 closed 12 years ago

bm3780 commented 12 years ago

I'm proposing that JobConf should inherit from the FileSystem's configuration for a Consolidation.

nathanmarz commented 12 years ago

Can you explain the reason for doing this? Did you run into any problems with the current behavior?

bm3780 commented 12 years ago

Yes, the subsequent MapReduce job for the Pail consolidation was being run locally rather than against my Hadoop cluster. I'm a new Hadoop user so perhaps there's another way to accomplish this. Setting the "mapred.job.tracker" property for my FileSystem configuration seemed to resolve this issue, and this is why I was proposing that the JobConf inherit the FileSystem's configuration.

sritchie commented 12 years ago

@nathanmarz, are you okay with this? I'm happy to merge this in.