pyres should be compatible with resque

[mjrusso]

I think that pyres should be compatible with resque.

I should be able to use pyres_worker to create Python workers that can subsequently be monitored via the resque web interface. Conversely, I should be able to use rake resque:work to create a Ruby worker that can then be monitored by the pyres web interface.

This is important because there are a number of alternative worker implementations being built against resque -- for example, in the C, C#, Scala languages.

Right now, I believe that the only issue barring compatibility is the timestamp format. The pyres web interface crashes if a job is created by a Ruby resque worker, because it reads a timestamp in a format it is not expecting. (The resque web interface reports all dates/times as NaN for all jobs created by Python pyres workers.)

This could trivially be fixed by changing the pyres timestamp format to match resque's.

I can make the change in my fork, but I would like to know if this will make it into the mainline.

[binarymatt]

can you describe what and where the change would be?

[yashh]

This could be a killer feature. Can we standardize the timestamp to use JSON timestamps?

[mjrusso]

Sure --

In Resque, timestamps are calculated with the following code:

Time.now.to_s

The result looks something like the following:

Thu Feb 18 23:42:16 -0500 2010

For example, when starting a worker, the following code is executed:

# Tell Redis we've started
def started!
  redis.set("worker:#{self}:started", Time.now.to_s)
end

And, as we would expect, this code creates a key-value pair that looks something like the following: worker:Macintosh-101.local:41166:default:started:Thu Feb 18 23:49:23 -0500 2010.

Now, when I try to run pyres_web with that key-value pair set in the database, the server will crash on any page that needs to render the timestamp. For example:

<type 'exceptions.ValueError'> occurred on '/workers/Macintosh-101.local:41166:default': time data 'Thu Feb 18 23:49:23 -0500 2010' does not match format '%Y-%m-%d %H:%M:%S'

Caused by:

File "/Library/Python/2.6/site-packages/pyres-0.5.0-py2.6.egg/pyres/worker.py", line 53, in _get_started
ds = datetime.datetime.strptime(datestring, '%Y-%m-%d %H:%M:%S')

Using the same timestamp format as returned by Ruby's Time.now.to_s would make this work. (This would need to be done everywhere that timestamps are generated.)

[binarymatt]

if someone wants to write up a patch, i'll be happy to take a look at it.

[mjrusso]

I will write a patch for this, hopefully in a week or so.

[mjrusso]

Turns out that this was a bit messier than I was hoping, so I reached out to the Resque folks. See here: http://github.com/defunkt/resque/issues#issue/60

We would have to change the following code to use Unix timestamps:

pyres.worker:

def _set_started(self, time):
    if time:
        self.resq.redis.set("resque:worker:%s:started" % self, time.strftime('%Y-%m-%d %H:%M:%S'))
    else:
        self.resq.redis.delete("resque:worker:%s:started" % self)

def _get_started(self):
    datestring = self.resq.redis.get("resque:worker:%s:started" % self)
    ds = None
    if datestring:
        ds = datetime.datetime.strptime(datestring, '%Y-%m-%d %H:%M:%S')
    return ds

pyres.failure.redis:

'failed_at' : str(datetime.datetime.now()),

And then there there might be a few places in pyres_web.

Can we get this into 0.7 or did you want to wait until the next version?

[binarymatt]

i think i might have this done. I'm working on some tests right now.

[binarymatt]

This is done and in master