NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

Add retry on hadoop client calls #347

Open raronson opened 10 years ago

raronson commented 10 years ago

If the cluster is under a lot of load, client calls which directly interact with it can fail in weird ways. I have seen LeaseExpiredException, ConnectException: Connection refused, and others which all point to a cluster under load not able to respond to requests.

Scoobi should retry these client calls a number of times so the app doesn't die on the first exception from a hadoop call.

Examples: