[💡FEATURE REQUEST]: support Mercure Protocol for broadcasting ( as a replacement for WebSocket )

[azjezz]

I would like to see a first-party integration for Mercure protocol with roadrunner, which could serve as a replacement for WebSocket broadcasting.

Mercure is used widely in the Symfony ecosystem, it has an official integration with the framework, and is used for hotwire integration in Symfony UX.

A standalone component is also available to be used outside of Symfony.

see: https://mercure.rocks/docs/ecosystem/awesome

difference between Mercure and WebSocket:

as explained in the website:

In a nutshell the WebSocket API is low level, Mercure is a high level. Mercure provides convenient built-in features such as authorization, re-connection, state reconciliation and a presence API ; while with WebSockets, you need to implement them yourself.

Also WebSockets are not designed to leverage HTTP/2+ and are known to be hard to secure. On the other hand Mercure relies on plain HTTP connections and benefits from the performance an security improvement built in the latest versions of this protocol.

HTTP/2 connections are multiplexed and bidirectional by default (it was not the case of HTTP/1). When using Mercure over a h2 connection (recommended), your app can receive data through Server-Sent Events, and send data to the server with regular POST (or PUT/PATCH/DELETE) requests, with no overhead.

Basically, in most cases Mercure can be used as a modern and easier to use replacement for WebSocket

[francislavoie]

This could come sorta for free by implementing Roadrunner as a plugin for Caddy (Mercure can also be plugged into Caddy, see API-Platform's recommended Docker image https://github.com/api-platform/api-platform/blob/main/api/docker/caddy/Caddyfile)

As wolfy-j has said on Reddit, a plugin for Caddy is on the roadmap https://www.reddit.com/r/PHP/comments/pnexyy/roadrunner_240_queue_and_keyvalue_drivers/hcp0qj0/?utm_source=reddit&utm_medium=web2x&context=3

[sebheitzmann]

+1. It would be really nice to have an all in one

[rustatian]

Please, vote on the original message with 👍🏻 . Thanks 😃

[rustatian]

@azjezz I briefly had a look at the Mercure Golang API and it seems to be very easy to implement Hub/Server support with the RR plugin. I guess after the draft implementation, we will discuss an API here (from the PHP POV).

[azjezz]

A client hub package already exist ( symfony/mercure ), it's maintained by @dunglas and me ( and some other contributors ).

I will make sure it's compatible personally.

[rustatian]

I mean, I'm not a PHP dev. The worker from my POV (and from the RR) is just a process. So, I need to understand, what API to expose from the RR perspective. After the first draft, I'll return to you (@azjezz) and @dunglas to discuss the integration if you guys don't mind.

[rustatian]

Also, the Golang library has version 0.13. I'm a little bit worried about breaking changes before the stable release.

[azjezz]

I think @dunglas is more qualified to answer that, as i don't work on the go lib.

[rustatian]

Hey guys 👋🏻 Some updates for this ticket:

1. I'm not sure that we should support Mercure protocol inside the standard RR bundle, and it would be nice to have it via a third-party RR plugin.

2. @francislavoie I looked briefly at the Caddy v2 modules, and, generally, that's possible to have a limited RR version as a Caddy module. But we use a different configuration (but still, it is possible to have a caddy-specific config plugin as we have for the std RR bundle link. But (I guess) we don't need to do that since RR has an http/https/fcgi endpoints, and Caddy users may reverse proxy the requests. Users may use the RR HTTP plugin w/o any middleware to have a raw, untouched request. Looking forward to getting your feedback.

[francislavoie]

@rustatian I think Roadrunner would best fit as a Caddy app, not as an HTTP handler/middleware plugin. See https://caddyserver.com/docs/architecture, it explains how Caddy is architected.

A couple examples of Caddy apps as 3rd party plugins are the ssh app which runs an SSH server, and the supervisor app which acts kinda like supervisord or systemd and can run arbitrary long-running scripts. Roadrunner could be spun up that way from within Caddy.

A possible idea is then Roadrunner support could be provided as a reverse_proxy transport which would send the request through the Roadrunner app running in memory; Caddy has http and fastcgi transports, so this would just be another transport module. A transport is just a module which implements http.RoundTripper.

[rustatian]

@francislavoie Got u, thanks. The funny thing is that we have the same plugins (I mean ssh and supervisor (called service)) 😄 Yeah, a RoundTripper is a good idea for integration. I'm not sure that all this stuff (our plugins) will be compatible since many of them use custom ResponseWriter's, which needs to be adequately tested. So, let's create a separate ticket for that and see the community feedback 👍🏻

[francislavoie]

The point is, this ticket would be made redundant if Roadrunner plugged into Caddy, because Mercure is already designed to work as an HTTP handler module for Caddy.

Two birds :bird:, one stone :rock:.

The way I could see it working is like this (via Caddyfile config, which is a higher-level config language, the underlying config is JSON)

{
    # the roadrunner app
    roadrunner {
        rpc tcp://127.0.0.1:6001
        server {
            command php psr-worker.php
        }
        ...
    }
}

app.example.com {
    root * /srv
    encode gzip
    mercure {
        ...
    }
    php_fastcgi roadrunner {
        transport roadrunner
    }
    file_server
}

The http.reverse_proxy.transport.roadrunner module would basically do res, err := ctx.App("roadrunner").RoundTrip(req) which would get the Roadrunner app and tell it to send the request through its worker.

I'm using the php_fastcgi directive there cause it takes care of doing all the usual rewrites needed for a typical modern PHP app (see https://caddyserver.com/docs/caddyfile/directives/php_fastcgi#expanded-form); php_fastcgi doesn't support overriding the transport right now but it could work that way :sweat_smile: (ideally the directive would just be called php or something but :man_shrugging:)

For logging, Caddy already uses zap as well, so that would plug in nicely too.

[rustatian]

Yeah, thanks. I'm not thinking about the Mercure because it's a relatively niche protocol. I'm thinking about existing plugins for the RR, like the whole queue infrastructure (amqp, boltdb, sqs, beanstalk, etc), temporal support, PHP-SDK, etc. We can use RR w/o plugins just to round-trip the request, but it's like 5 percent of the RR.

[rustatian]

And I'm thinking, to support the RR module inside a Caddy, not only for the Mercure, but for the PHP users.

(Funny thing, that I'm not a PHP dev at all, and from the RR perspective, it doesn't know that inside the process is a PHP script)

[rustatian]

Not planned. Implemented Centrifuge for the broadcasting (WS + Clients): https://github.com/roadrunner-server/centrifuge + PHP: https://github.com/roadrunner-php/centrifugo.