Don't serialise worker class inside the job

So, we use Disque for two things at the moment:

performing long-running tasks of our ruby app in the background
message queue between our ruby and golang apps

The second case means we ended up with code like this in a Go app:

    job := map[string]string{
        "class":     "Jobs::ProcessMessage",
        "arguments": payload,
    }

Which is pretty bad, IMO. One of the reasons to use a message queue to communicate between applications is decoupling the producer and consumer of events. However, in this case, we have to tell the producer exactly which consumer will take care of this message.

Proposal

Stop sending the class parameter altogether in serialised jobs. Instead, just send the payload at the top level.

In order for this to work, we'd need to change how we consume jobs: We no longer just grab jobs off a queue and get metadata from them. The burden of knowing who consumes the job in each queue lies in the consumer (Disc::Worker).

For this, we need to have a mapping somewhere of which queue gets consumed by which job. At the moment, bin/disc requires at least one file to be required, so we can make sure that entry point runs the initialisation code. Something along these lines, maybe?

# disc.rb
require "disc/worker"
# require your jobs, any init code, etc.

Disc::Worker.run(
  "foo" => Jobs::Foo, # queue name => consumer
  "bar" => Jobs::Bar,
  # etc
)

Why is this good?

Because it decouples producers and consumers when used as a message broker between multiple apps.

Why is this bad?

We lose the multiple jobs per queue. This might be a bad thing (though it's easy to implement by clients who want to keep this—in fact, if we do this, I'd probably include code for this in core so that users have to write minimal glue code to upgrade)

So… Havanna

This is pretty much what Havanna does, so maybe it's not worth going this route in Disc and just use Havanna in cases where you have producers and consumers in different applications? Dunno.

cc @pote @djanowski @soveran what do you think?

pote / disc