Implement initial message passing capability

[dfaller]

This is the start of Scale's new messaging passing architecture to improve scalability. All of this should be in a Django app, called "messaging"?

Base class CommandMessage

• self.succeeded - used by manager to know whether to ack success for this message
• self.new_messages - list of CommandMessage, used by manager to know if new messages should be sent
• self.type - type of message to use in factory method to create correct subclass, will need to be sent/received as part of overall payload in the CommandMessageManager methods
• to_json()
• from_json(json_dict)
• execute() - docstring should mention that this should be idempotent and able to be run out of order

Subclasses register in Django app methods and have no-arg init() Factory method to generate CommandMessage objects from JSON payloads Basic test CommandMessage (log "Hello Scale" or something)

Class CommandMessageManager

• init() sets up and initializes actual backend (SQS, RabbitMQ, Kafka) from settings/env vars
• send_message(cmd_msg) - uses cmd_msg.to_json() to create payload
• singleton access as cmd_msg_mgr
• process_messages() - generically handles "loops" of message processing, should handle multiple in a loop (so we can do 10 at a time with SQS batch), gets JSON payloads from backend and calls factory method and from_json() to set up cmd_msg objects, and processes them

Django management command to run cmd_msg_mgr.process_messages()

Classes that abstract/implement backends (SQS, RabbitMQ, Kafka, etc) SQS backend should use long-polling and SQS batch (10 at a time)

Need to updates docs to discuss configuring Scale to use whatever messaging backend.

[gisjedi]

So after some reading on the subject, I'm leaning toward RabbitMQ for the out-of-the-box solution. RabbitMQ appears to be the nearest match for what we are trying to accomplish and has excellent Python (and other cross-language support), as a bonus its usage is very similar to SQS. Kafka isn't a first choice as it is favors an event stream approach with no cleanup following message processing (message roll-off happens by queue size / TTL). It is up to the consumer to manage their position in the log instead of just popping off the front of the queue.

[gisjedi]

The initial implementation will support RabbitMQ, with SQS being the secondary implementation.