GoogleCloudPlatform / appengine-mapreduce

A library for running MapReduce jobs on App Engine
https://github.com/GoogleCloudPlatform/appengine-mapreduce/wiki/1-MapReduce
Apache License 2.0
234 stars 109 forks source link

allow lock timeout for a slice of a sharded task to be configured via `MapSettings` #101

Closed eschultink closed 8 years ago

eschultink commented 8 years ago

Basic problem is that you can configure millisPerSlice via MapSettings. When a MapJob runs, it continues to execute the slice until it's been executing for >= that many milliseconds (see example implementation in MapShardTask).

While this shard task is executing, the share is "locked". This lock is considered to be held for a timeout that is millisPerSlice * 1.1 (MapSettings:286). If that lock has not been released and that much time has passed, ShardJobRunner will deem the slice as failed (see ShardJobRunner.handleLockHeld()).

This PR makes that 1.1 value configurable via MapSettings (or MapReduceSettings). I've called this sliceTimeoutRatio, but certainly open to suggestions. This PR also replaces a few other magic numbers with the appropriate constants.

googlebot commented 8 years ago

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


eschultink commented 8 years ago

I signed it!

googlebot commented 8 years ago

CLAs look good, thanks!

eschultink commented 8 years ago

@aozarov thanks Arie - removed that.