search

nathanmarz / dfs-datastores

Dead-simple vertical partitioning, compression, appends, and consolidation of data on a distributed filesystem.
BSD 3-Clause "New" or "Revised" License
215 stars 82 forks source link

PailTap ignores Hadoop configuration provided to it #41

Closed kyrill007 closed 10 years ago

kyrill007 commented 11 years ago

PailTap in its sourceConfInit method, when creating its Pail instance, ignores Hadoop configuration provided to it. This present a problem for us. We are using MapR instance of Hadoop and thus the default configuration is obtained from MapR client config directories. Well, we often override those setting with custom ones (in code). For example, instead of using the default MapR filesystem we often use LocalFileSystem (during testing), and thus the tests fail since PailTap resolves to default configuration, which MapR.

It is interesting that in other places in this class PailTap actually uses the provided configuration (when sinking), but it seems to ignore it when sourcing.

The pull request is coming.

kyrill007 commented 10 years ago

This issue should be closed as my pull request with the fix got merged.