search

facebookarchive / hive-io-experimental

Hive I/O Library
Other
66 stars 30 forks source link

HiveApiOutputCommitter should have retries to guard against intermittent hive metastore failures #20

Open groys opened 11 years ago

groys commented 11 years ago

The output committer kicks in after all the data has been successfully written by MR. The hive metastore can be flaky and an exception in the CommitJob function can waste the entire work the MR job has done.

It would be great to retry a few times if talking to the hive metastore fails. Thrift/swift may provide such a client built in

nitay commented 11 years ago

Agreed this would be great to have. Got a PR? :). Otherwise ill get to it soon hopefully.

On Apr 26, 2013, at 10:42, Gautam Roy notifications@github.com wrote:

The output committer kicks in after all the data has been successfully written by MR. The hive metastore can be flaky and an exception in the CommitJob function can waste the entire work the MR job has done.

It would be great to retry a few times if talking to the hive metastore fails. Thrift/swift may provide such a client built in

— Reply to this email directly or view it on GitHubhttps://github.com/facebook/hive-io-experimental/issues/20 .