Magento Queue API definition proposal

[alankent]

Magento currently does not have a standard message queue API. There were plans based around “async events” (or a job queue) and “web hooks”, but this is not scheduled by the core team for Magento 2.0. This work is to propose a set of interfaces for Magento and community comment, ideally developed by someone who has practical experience of using a message queue within Magento in a real-life application. (Implementation of the interfaces would be a separate phase of development.)

The business goal is to support async processing of parts of requests. For example, to achieve greater performance, it may be desirable to accept an order and queue it for later processing, with the possible risk of the item becoming out of stock. This can deliver better throughput at peak, with the negative of occasionally having to later inform customers that an item is out of stock. (There is no proposal to change the current implementation of order processing. The only change is to provide a standardized message queue API that extensions can depend on.)

The work for this GitHub issue is to define service contracts (PHP interfaces) for sending messages to a queue, plus have a standardized API for receiving requests from the queue. The API/infrastructure should make it possible to plug in a different implementations, ideally per queue.

Example possible queue implementations include:

• Call the processing function immediately (that is, don’t queue it at all – do it immediately). Useful for debugging and in environments where a separate process cannot be run.
• Let the main processing flow form the HTTP response and send that back to the initiating web browser, then do all queued requests before exiting the current thread. This provides a faster response to the caller, while not requiring a separate process to manage to monitor the queue.
• Use a table in MySQL as the queue. No need for another technology to be installed. Cron or similar could launch code to check for async requests and process them.
• Redis, RabbitMQ, ActiveMQ, SQS, Kafka, … - the community should be able to plug in different queue implementations without change to the Magento code.

Magento might provide one or two implementations (e.g. no-queue, or no-queue and MySQL-queue). The community can then provide other implementations via extensions. No change should be required to producer code (code creating queue requests) or consumer code (code processing queue requests) when changing queue implementation.

The default Magento distribution most likely will not be changed to create messages. More likely a site would create a message inside a plugin or event observer. Any synergies with the existing Magento event infrastructure should be considered. (e.g. can the event dispatch API be used for message queues?)

“Web hooks” was another related technology. One option was to supply a standard consumer implementation to send a queue message to a remote end point using a HTTP POST with JSON content.

[alankent]

http://www.eschrade.com/page/reasons-why-redis-is-a-great-option-for-a-worker-queue-in-magento/ has more great reasons why queuing is useful.

[kschroeder]

Another benefit of using a queuing system is that you could build out a service layer in the architecture that could process different parts of a request asynchronously. Define a series of actions and inject each into the queue and then wait on the queue until all of those actions have been processed.

For example, it could be done on the block level in catalog_category_view the action could define several jobs, such as generate-layout-XML, calculate-related-products, render-shopping-cart among others and have those jobs inject the rendered HTML into a response queue. The frontend would inject them all at once in the queue and retrieve the responses as soon as they were available. This way the performance hit would only be the longest processing action instead of the totality of all processing actions.

The full page cache would likely still be faster for cached content, but for dynamic content it would allow merchants to build out much more complex scenarios without sacrificing much response time.

[SKovbel]

Great Idea!!!

Magento need something like this

interface SchedulerInterface
{
    execute (Object $args);
}

class Scheduler
{
    addQueue (
        SchedulerInterface $receiver, 
        Object $args = null,
        [$start_date = null]
    )
    addJob (
        SchedulerInterface $receiver, 
        Object $args,
        $cron_expr
    )
}

[kschroeder]

@alankent how detailed do you want this discussion to get here?

[alankent]

I was hoping a group of interested parties would identify themselves and work out how they wanted to work together (email, separate Git repo, discuss here, whatever). Then come back with a proposal. Still working out the best way. Simplest might be to fork your own repo and start work there and invite others to come have a look and comment / suggest alternatives. Just drop a link on this page to the work you are doing for others to find.

[MarySmith13]

However magento has a good reputation among the site development companies. I know lots of qualified developers working on mogento's platform. For example http://magecom.net/

[Flyingmana]

I take the "separate Git repo" approach and now have a dirty but usable first draft.

https://github.com/Cotya/QueueModule

[jksimoniii]

@alankent Doesn't look like there was too much traction here. Are you aware of any progress with this concept? Or maybe more discussions outside of this channel? I am considering using messaging queues as solution for many of the things mentioned above.

[alankent]

Yes, there is queue support in EE now for background processing.

Related is work internally to support better communication between the existing Magento and the new order management and retail commerce products based on message queues.

So there is some work progressing - happy to share more if useful.

[alankent]

This particular request is out of date now - things have moved on. So closing to avoid future confusion.