shinyscorpion / task_bunny

TaskBunny is a background processing application written in Elixir and uses RabbitMQ as a messaging backend
MIT License
202 stars 30 forks source link

new message format #13

Closed ono closed 7 years ago

ono commented 7 years ago

First step of https://github.com/shinyscorpion/task_bunny/issues/4

Old format:

{"id": 123}

New format:

{
  "job": "TaskBunny.SampleJob",
  "payload": {"id": 123},
  "created_at": "2017-02-16T22:29:16.524651Z"
}
grammaticof commented 7 years ago

Good idea as long as it's configurable.

It will be a good thing to have for specific use cases.

A couple of questions:

Who is the broker for the unique queue? How the worker knows which jobs to pickup as the specific job is described in the payload?

Another problem is dealing with one "retry" queue as you will need to read the payload to understand which are failing.

ono commented 7 years ago

@losco I believe it's explained here: https://github.com/shinyscorpion/task_bunny/issues/4. TaskBunny will declare retry and reject queues for each queues.

This PR contains only message format change. Configuration change will be happening ....soon.

IanLuites commented 7 years ago

Actually, I think RabbitMQ already does this, but it might be good for logkeeping...

Since we now have a clearer reason for job failure, do you think it might be a good thing to add a log to the message?

Something like:

{
  "job": "TaskBunnyJob",
  "payload": {"id": 123},
  "created_at": "2017-02-16T22:29:16.524651Z",
  "log": [
      {"type": "failed", "reason": "Some description", "timestamp": "2017-02-16T22:30:16.524651Z"},
      {"type": "failed", "reason": "Some description", "timestamp": "2017-02-16T22:35:16.524651Z"},
      {"type": "rejected", "reason": "Some description", "timestamp": "2017-02-16T22:40:16.524651Z"},
  ]
}
ono commented 7 years ago

Yeah, after this change, you can extend the message easily. I wouldn't add the log entry you suggested in this PR but we can do it at a request or need.

IanLuites commented 7 years ago

Fair enough, seems easy to add.

I don't know if you are interested in it, but have you looked into the Poison into/@derive? here It would allow for easier validating and seeing the messages as an actually typed struct.

Also added #15 as suggestion for the job <-> string conversions.