verbling / kickq

Kick jobs out the door. Quickly. A job queue for node
MIT License
36 stars 6 forks source link

`queued` job not being picked up #5

Open cadorn opened 11 years ago

cadorn commented 11 years ago

I jave a job that seems to be stuck with status queued:

Wed May 08 2013 18:03:06 GMT+0000 (UTC) - debug: '[rolodex][job:fetch-for-service] Existing job:' { id: '103',
  name: 'fetch-for-service',
  complete: false,
  success: false,
  createTime: 1368033631407,
  updateTime: 1368033631440,
  finishTime: null,
  totalProcessTime: null,
  delay: 1000,
  processTimeout: 120000,
  retry: false,
  retryTimes: 3,
  retryInterval: 1800000,
  hotjob: false,
  hotjobTimeout: 10000,
  ghostRetry: false,
  ghostTimes: 1,
  ghostInterval: 1800000,
  data:
   { sessionID: 'wvJ0JuQcHT3fJV1LJmd8wSaY',
     serviceId: 'github',
     credentials:
      { id: 967670,
        username: 'cadorn-read',
        accessToken: 'f5d2fc149fbaedd92a0f616a094ce5ec71568cc8' } },
  lastError: null,
  scheduledFor: 1368033632440,
  state: 'queued',
  runs: [],
  hotjobPromise: null }
Wed May 08 2013 18:03:06 GMT+0000 (UTC) - debug: '[rolodex][job:fetch-for-service] createTime:' 1368033631407 -2556
Wed May 08 2013 18:03:06 GMT+0000 (UTC) - debug: '[rolodex][job:fetch-for-service] createTime:' 1368033631440 -2556
Wed May 08 2013 18:03:06 GMT+0000 (UTC) - debug: '[rolodex][job:fetch-for-service] scheduledFor:' 1368033632440 -2555

Not sure what lead to this. Maybe the worker got killed while processing the job?

Another reason to switch to a better queuing logic?

thanpolas commented 11 years ago

How was this breakdown produced? (at which point?)

I don't see the id property...

This might require some redis-cli exploring... is that possible?

thanpolas commented 11 years ago

ah, id is there, i needed to scroll horizontally...

Do you store logs? Can you grep for id 103?

cadorn commented 11 years ago

I don't have more info than what I posted and I have a workaround that schedules a new job if updateTime is older than a timeout so fixing this is not critical for me right now. But it points to a problem in the processing logic as a job should never be stuck in queued state no matter in what intricate ways workers restarted or db crashed and recovered.

thanpolas commented 11 years ago

Kickq is still in its infancy. Since you are brave enough to move forward with it and suffer all the consequences it would make sense to check out the logging options =).

A sensible setup is to save to a logfile and tune the loglevel to fine:

kickq.config({
  loggerLevel: kickq.LogLevel.FINE,
  loggerFile: true,
  loggerFilename: 'kickq.log'  
});