search

documentcloud / cloud-crowd

Parallel Processing for the Rest of Us
https://github.com/documentcloud/cloud-crowd/wiki
MIT License
851 stars 92 forks source link

Timeout for jobs #14

Closed lucascs closed 14 years ago

lucascs commented 14 years ago

Hi,

It should be possible to specify a timeout for jobs, when adding it:

RestClient.post("#{CROWD_SERVER}/jobs", {
      :job => {
          'action' => 'word_count',
          'inputs' => ['a', 'b', 'c'],
          'timeout' => 60
      }.to_json
 });

so if an input takes more than 60 seconds to process, it is canceled...

and on merge, this input should be either ignored or have an error/default value.

Thanks

jashkenas commented 14 years ago

This sort of thing is highly specific to your action, or individual pieces of your action, and shouldn't be handled at the broad level of the job.

If you'd like to time out an action, just wrap the relevant code in a timeout block.

http://ruby-doc.org/stdlib/libdoc/timeout/rdoc/index.html

The Timeout::Error will propagate back up through CloudCrowd, and will cause the work unit and the job to be marked as "failed", and then call your application at the callback_url to tell it. Your application can take it from there.

Sound alright?

lucascs commented 14 years ago

Sounds good to me =)

Thanks