[Messenger][RFC] Improve Messenger to support other app consuming/recieving the message

[tristanbes]

Description

Hello,

I'm a new user of Messenger; Thank you to all contributors of this promising component; Yet, it gave me a hard time to solve some basic problematics.

Example

Let's say you have 2 different Symfony applications.

The application A (big, monolithic) is responsible of publishing the message to the queue using messenger. Until here, it's ok, everything is covered by the doc.

Then the queue (Amazon SQS/SNS) triggers a new Lambda execution, passing the message to the function (which is a SF Application B, responsible of generating a PDF); This behaviour is specific to Amazon, but it can be generalized of "messages can be pulled or pushed".

The problematic

A. On all examples on the doc, you assume that the consumer is the application that put the message in the queue itself;

But what if it's not the case ? What if it is another Symfony (smaller) project, responsible of generating the PDF only ? What if it's another language ? (JS ?).

B. The Messenger component sends the messages with some namespace inside;

So if your message class lives in App\AwesomeBundle\Message\PdfMessage in application A, and App\Message\PdfMessage in application B, then application B won't be able to decode the Message;

C. What if the message is not pulled by the messenger component itself, but somhow is recieved (pull vs pushed) ?

---

I achieved this scenario, and here's what helped me:

• Happyr/message-serializer
• http://developer.happyr.com/symfony-messenger-on-aws-lambda
• http://developer.happyr.com/sns-retry
• @Nyholm (thanks again ;D)

Do you think some of these problematic are generic enough to be covered by the component code or by the documentation ?

Thank you :)

[dunglas]

Isn't the Symfony Serializer integration enough for these cases? When using Serializer, the message is encoded in JSON, and can easily be decode by any app able to deal with JSON.

[weaverryan]

We’re covering this in the coming days in SymfonyCasts:

A) symfony Serializer for turning messages into JSON if the worker will be another app. That’s one line of config.

B) custom Messenger transport Serializer is your app is consuming messages that originate from another app. That’s one class and one line of config (and if you want, you can use the Serializer component from within the this custom transport Serializer).

Could the problem be just a lack of documentation to point this out?

Thanks for the convo!

[ragboyjr]

So, I actually was able to successfully get messenger working with magento 1.x to a symfony application over redis. Both applications send and receive messages from the other.

What we ended up doing was just create a third repository for storing our shared message classes between the two, and then installed the repository as a git submodule and used composer path type repositories to include into each of the applications.

I think using shared classes is safer than maintaining two separate versions of the class, but, you still have to be careful that any message class changes are done in a backwards compatible way to remove the tight coupling of those apps being deployed together.

[Matth--]

We have an application that is synced up with a .net process. We use a custom Messenger Serializer (described by @weaverryan) for inbound and outbound messages.

[rjwebdev]

I am struggling with this issue myself currently at work where I have to publish messages onto a queue where another agency will pick up the messages.

I tried to create a small package that contains interfaces which would be implemented by the classes that will be published so both of us would end up installing this package. In my messenger config, I bound the interfaces to the different queues but this did not work since the class names were added to the envelope and not the interface names.

[pounard]

@rjwebdev

but this did not work since the class names were added to the envelope and not the interface names

That's one of my main concern with messenger (and symfony/serializer component as well): you cannot give a map of names or aliases that would map to PHP class. In a project we wrote this: https://gist.github.com/pounard/25d2a8d4502a6bb61e660957196a5773 (which is then filled with an alias to PHP class array map, from a custom bundle configuration).

Then you specify your messages with business oriented names and not technical ones. One missing bit in my example code is that we also have a custom messenger's serializer that maps the names the other way around to send them instead of PHP class names (PHP classes got converted to business name when published, then converted back into PHP class names when consumed).

[pounard]

@ragboyjr

I think using shared classes is safer than maintaining two separate versions of the class, but, you still have to be careful that any message class changes are done in a backwards compatible way to remove the tight coupling of those apps being deployed together.

If you have more than two applications (currently we have only one but we are targeting a client environment where we'll have at least 5, each with their own maintainers and release plan, plus one end-user centric dashboard that will arbitrarily consume events to populate a client dashboard).

In this future, having other technos than PHP is considered, and having stalling application for 1 or 2 year is also a possibility: we have to specify our messages as a high-level business-oriented specification unbound to the technical aspects. Which means, backward compatibility when possible, having the most recent upgrades able to talk using more than one version of the protocol, etc...

In that regard, sharing code with every application is not possible.

[pounard]

@tristanbes

Do you think some of these problematic are generic enough to be covered by the component code or by the documentation ?

I think it should, I opened #33373 #33368 (messenger) and #33399 #33397 (serializer) for discussion some time ago. You might also want to participate in #32049

[ragboyjr]

@pounard for sure, you gotta do what works best for your team and roadmap. A custom serializer might be the best way to go if you end up sharing messages across various languages. If you can keep in php, a shard repository that can be updated at its own cadence in the individual repos might be less work.

[pounard]

@ragboyjr

That's the usual answer I get :) I do not always agree though. Regarding incoming content type I still think that the messenger component should handle this natively. For the rest, every bit is subject to subjectivity and one's need. But all remain interesting topics to discuss.

To go further in this direction, there's still a probability that we'll switch from the messenger to a more bare-metal AMQP implementation instead one day, if the messenger doesn't easily answer our needs; that's why all our business code sends its messages via a custom interface, and messenger, in our project, is completely hidden behind it. It's replaceable.

[weaverryan]

Are you suggesting that the message type would be set on the content type and then a system would allow you to map each type to a class in your code (that the Serializer works then use fire deserialization)? Or do I misunderstand?

For us, our custom Serializer is a basic switch case statement: we determine what kind is message is coming in (e.g. by reading some info from a header or just looking at the field), then use that in a switch case to create and return different message objects for each.

[pounard]

@weaverryan

Are you suggesting that the message type would be set on the content type and then a system would allow you to map each type to a class in your code

content-type and message type are both very different things, I think they can be handled very differently.

Message type may or may not be exposed by the bus implementation, for example in AMQP you can retrieve messages with the 'type' property (as of now, the messenger stores it alongside the message header, in the form of the PHP class name). Converting from an arbitrary type to a PHP class could be a feature of component/serializer, or could be handled by the messenger serializer (which I think makes more sense).

Regarding content-type (i.e. format in component/serializer semantic) should be handled by the messenger serializer class.

For us, our custom Serializer is a basic switch case statement: we determine what kind is message is coming in (e.g. by reading some info from a header or just looking at the field), then use that in a switch case to create and return different message objects for each.

I think that a basic version of that based upon configuration could be in messenger core, all it needs is a map to be injected, using the dic or annotations (either one or both). You then wouldn't even need to implement your own.

[kafoso]

Symfony Messenger's headers are very tightly coupled with not only Symfony Messenger, but also with PHP as a whole through its utilization of PHP namespacees.

This is quite the hindrance for us who wish to utilize other programming languages that publish and consume messages from the same RabbitMQ server.

The "type" header always points to a PHP class (full namespace). But what if the message is an integration event, consumed by an entirely different code base with a different namespace? It is a very greedy coupling.

Stamps are a great feature. I like how they're handled. However, again the coupling to Symfony and PHP is tight, as headers like "X-Message-Stamp-Foo\Bar\Baz" will exist, where "Foo\Bar\Baz" is - again - a PHP class reference.

You are completely locked into this naming convention if you use Symfony\Component\Messenger\Transport\Serialization\Serializer and must then write your very own serializer (implementing Symfony\Component\Messenger\Transport\Serialization\SerializerInterface). This is of course doable, but I feel Symfony Messenger should support overriding of header keys.

This could be fixed for stamps by introducing a new interface, e.g. "NamedHeaderStampInterface":

<?php
namespace Symfony\Component\Messenger\Stamp;

use Symfony\Component\Messenger\Stamp\StampInterface;

interface NamedHeaderStampInterface extends StampInterface
{
    public function getHeaderName(): string;
}

Stamps that implement this interface will use the string provided by getHeaderName(), instead of "X-Message-Stamp-". Yes, there is a very real risk of name conflicts, but that should be something developers sort out themselves.

Mapping the header names to their respective classes is another thing which must be handled, such that when consuming, things may get glued back together.

A thought on how to handle the "type" header differently

Let developers define a core namespace for the messages being sent to an exchange. This namespace is then subtracted from the full class name.

Example:

<?php
use My\Application\AMQP\Event\Foo\Bar\Baz\UserWasCreatedEvent;

$namespace = 'My\Application\AMQP\Event';
$userWasCreatedEvent = new UserWasCreatedEvent;

if (0 !== stripos(get_class($userWasCreatedEvent), $namespace)) {
    throw new \Exception(sprintf(
        "Class \\%s must be located under the namespace \\%s, but it is not",
        get_class($userWasCreatedEvent),
        $namespace
    ));
}

$type = substr(get_class($userWasCreatedEvent), strlen($namespace));

echo $type;

Output:

Foo\Bar\Baz\UserWasCreatedEvent

To stitch it all together again, the consumer will simply have to prepend the namespace. We have now gotten rid of the ultra tight coupling in PHP and to the application from which a message originated. However, the remainder of the namespace will still have to be exactly the same.

The above is also very easily converted to namespacing in other programming languages like C#, Java and Node.js. E.g. simply replace the backslashes with dots.

---

Anyone else thinking this would be a great change?

I may even be done without backward compatibility breakage by simple writing an alternative Symfony\Component\Messenger\Transport\Serialization\Serializer, e.g. Symfony\Component\Messenger\Transport\Serialization\CrossApplicationSerializer, which then handles - you guessed it - the cross application aspects.

[pounard]

I just think the messenger component should have a type to class map, per default returning the class name, but configurable, either using some YAML, some static method call, or using annotations (I dislike annotations I would not use them anyway). The messenger's serializer is way too coupled to messenger own logic making it very hard to replace or decorate. We done that for a year, and ended up ditching the messenger component completely, using a 100% custom bus using pgsql now.

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[tristanbes]

Carsonbot (🤖 ) I didn't work on this problematic since 1 / 1.5 year. Has this issue improved since ?

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[adioe3]

I just think the messenger component should have a type to class map, per default returning the class name, but configurable, either using some YAML, some static method call, or using annotations (I dislike annotations I would not use them anyway). The messenger's serializer is way too coupled to messenger own logic making it very hard to replace or decorate. We done that for a year, and ended up ditching the messenger component completely, using a 100% custom bus using pgsql now.

This is exactly what we've implemented in a separate bundle in my company and it works well. I am in favor of this approach for two reasons:

1. free-form event names are great (user.farted.today versus App\Events\User\Fart\Today)
2. you can keep backwards-compatibility, i.e. this would work:

routing:
  super.custom.event: transport
  Old\School\Event: transport

events:
  super.custom.event: Some\Php\Class

Now I can integrate non-Symfony applications, yay!

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[carsonbot]

Hello? This issue is about to be closed if nobody replies.

[kafoso]

I would still like to see at least opt-in support for a loose coupling.

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[weaverryan]

I think there is definitely room for a solution here. To get the ball rolling, someone needs to jump in and start a pull request. PR's always cause more action than issues. I think we've validated the "need" for this already. Volunteers?

[danrot]

I am currently running in the same situation, and in my specific problem I don't see such an easy solution as described previously. I use debezium to generate some messages in a redis stream, but unfortunately the messenger redis transport cannot read them (at least not easily). And the issue does not seem to be solved by just using another serializer... In the redis transport there is a line specifically asking for a messages key, which does not exist in my case.

https://github.com/symfony/symfony/blob/817448301b6552092d1330679f9a064ba5e9ae9f/src/Symfony/Component/Messenger/Bridge/Redis/Transport/RedisReceiver.php#L45

And it seems to me like this is happening before any of the other extension points, like e.g. the Serializer, are called. Therefore I don't have another choice than implementing my own transport (except for maybe reusing the Connection object).

In general I find it a bit weird that a component that is thought to be used with stuff like Redis Streams or RabbitMQ is so tightly coupled to PHP resp. itself. It expects a certain structure, which might not be used emitted by other systems, and it uses PHP serialization by default.

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[pscheit]

yes carson, that would be nice

[94noni]

passing by via google:

it reminds me:

• https://jolicode.com/blog/about-symfony-messenger-and-interoperability
• https://jolicode.com/blog/state-replication-with-symfony-workflow-messenger-and-rabbitmq

i run a use case like this, big app (symfony) dispatching an event that is encode php to json and a micro app (symfony) that decode and handle it as well as some app (javascript) that pushes js into rabbitmq, and my micro app (symfony) decodes it and handle it

[rela589n]

It has been 4 years already since the issue was originally opened.

Currently building an SOA application we still try to use messenger, since it provides a great deal of flexibility.

Though, in terms of communication over network these serialized php objects suck, and it's necessary to write a custom transport logic.

It would be such a relief to have this at core. If implemeted properly, messenger could be used along with thrift, avro, protocol buffers in order to send big amounts of data, or just json for basic ordinary needs

[wouterj]

Ryan's message from a year ago still holds: This issue shows there is a need, but we need someone to do the work. Anyone up to starting a PR to implement the ideas discussed in this issue?

Otherwise, I'm afraid we'll have to close this as stalled given there apparently hasn't been enough interest for somebody (or some company) to dedicate some time to actually fix this.

[pounard]

This issue shows there is a need, but we need someone to do the work.

I did, in 2019. I don't want to it a second time, I'm sorry but this has fatigued me it took 3 years for people to understand it's a must have. I sincerely hope someone will do because it's a necessary feature, it would make the messenger component greater !

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[faizanakram99]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

Yes

[pscheit]

@weaverryan @wouterj can someone outline where one would start? (with the implementation?)

[94noni]

just adding my 2cts here

perhaps related to what I've posted previously

we can imagine just having a class in the symfony core messenger component, not being the default, that just serialized the envelope and return a raw json unrelated to php things at all to decouple things ?

thus, freely and simply, we can swap the serializer config of any transport requiring this core class

[xabbuh]

I have to admit that through all the comments in this issue I have no idea anymore what are the core changes that you all would like to see for the Messenger component. So from my POV crafting an actionable list of things to do would be a good start.

[pounard]

@xabbuh long time ago when this issue was opened, where I work for we have switched to a completely home-made message dispatcher instead of messenger due to the numerous restrictions it had at that time.

Now, we intend to switch back to messenger which has evolved and stabilized during those years, for new projects, so soon enough I'll go deeper in this topic and will be able to help draw pieces of this actionable list.

We have a new project coming in the next weeks, so stay tuned.

[pounard]

@xabbuh First one: add an #[AsMessage(bus: 'async')] attribute in order to apply bus routing rules directly on the message class instead of having it to write in config/packages/messenger.yaml under the framework.messenger.routing option. $bus parameter needs to be typed as int|array in order to allow messages to be routed through multiple buses if needed (it is documented that as possible).

[stof]

given that message classes are not registered as service, this won't be easy as autoconfiguration attributes are used only on autoconfigured services (unless you suggest reading that attribute at runtime).

[pounard]

given that message classes are not registered as service, this won't be easy as autoconfiguration attributes are used only on autoconfigured services (unless you suggest reading that attribute at runtime).

You can read it at runtime with no problems, reflection is lightning fast. We did this in our own bus implementation for routing

[rela589n]

That's a good point!

I support this idea, since I myself used to find me three times a day jumping back and forth between message routing configuration files just to tweak a message class after refactoring.

Idea with attribute is a good starting point, though, in case of routing configuration files, it's a common thing to define a separate routing (e.g. sync transport) sections for development environments (when@dev, when@test). Currently I have no idea how it could be done with attributes 👀

[pounard]

Idea with attribute is a good starting point, though, in case of routing configuration files, it's a common thing to define a separate routing (e.g. sync transport) sections for development environments (when@dev, when@test). Currently I have no idea how it could be done with attributes 👀

My opinion is that the bus implementation should be different when dev or test, but not the routing itself, otherwise you don't test in prod condition. But that's an opinion, not an absolute truth. A simple idea to fix that would be either to add an $env parameter on the attribute (not fond of it looks like a leaky abstraction to me) or simply state that the YAML configuration will override the attributes (seems legit to me, you could then override a bundle hardcoded configuration for example).

[pounard]

I will open an issue and a PR next week if time allows me to do so.

[wouterj]

Hi!

This issue is going a bit everywhere, which makes it hard to understand (maybe extra because I'm not that familiar with the Messenger component). I suggest the best way forward is to branch of a couple actionable and scoped issues, and close this one.

For now, I've spotted three things in the discussion here:

• Add a type to FQCN map to the Messenger component (defaulting to type = FQCN), so messages are decoupled from PHP namespaces. I think this is worth an issue, as some research is needed on how this influences other, non-AMQP, transports (which don't have a content/message type feature)
• Improve DX to use the serializer component to decouple the data from PHP as well? (it's not clear to me if this is already the case or not)
• Experiment with the idea of an #[AsMessage] attribute to define the routing, as proposed by @pounard.

Are some of you open to start the issue, run some research and experiment with code? I'm happy to assist with some early brainstorming and checking if the changes aren't too impactful on the current code. You can find me as wouterj on Symfony Slack.

[pounard]

@wouterj

This issue is going a bit everywhere

Agree, this would worth the shot to open a set of smaller issues, lots of the changes requested here are not conflicting with each other. I'll try to reach you on Slack, but I don't open the client often since it likes to eat all my laptop RAM.

[pounard]

@wouterj @stof @xabbuh @rela589n I opened https://github.com/symfony/symfony/issues/57506 for the routing specific issue and discussing around adding the #[AsMessage] attribute.

[stof]

Using the Serializer component instead of using PHP serialization is already possible (the default behavior is to use PHP serialization because it is a simpler setup, which works fine when you don't need to share the message between apps)

[pounard]

@stof

Using the Serializer component instead of using PHP serialization is already possible (the default behavior is to use PHP serialization because it is a simpler setup, which works fine when you don't need to share the message between apps)

OK nice, that wasn't the case when this issue was created, that's good news.

[pounard]

Using the Serializer component instead of using PHP serialization is already possible (the default behavior is to use PHP serialization because it is a simpler setup, which works fine when you don't need to share the message between apps)

I reread the code and it seems what you said is not exactly true: while it seems to be OK in Symfony\Component\Messenger\Transport\Serialization, yet theSymfony\Component\Messenger\Transport\Serialization\PhpSerializer class is hardcoded at many places, such as in console commands. Is there something wrong here?

[nesl247]

Correct me if I'm wrong, but even the json serialization is still called the symfony serializer because without symfony specific message properties, symfony can't consume a message created from an external system. I imagine, because we'd like this too, that this particular point in this issue is that there should be a configurable routing happening from queue -> message, not needing specific symfony headers for consumption.

[pounard]

I saw two different ways of solving the intercommunication:

1. first being that some brokers implement a "type" or sometime named "subject" header in the envelope, which gives information to the receiver about what's inside: that's I guess the right place to tell which message is it, then it's guessable, all you need is a dedicated logical name to class map on the Symfony side for it to work. (Small note here: some brokers allow additional headers to be passed, like HTTP when you write X-Foo-Bar, that might be a sensible way of passing the type information.)

2. another one is more barbarish, you unserialize using the content type (xml, json, whatever) and attempt to guess using data found inside, this one is not really what we would want with the serializer component.

Regarding the first point, there are many ways to skin a cat, but I can see two (which are in the end equivalent) and both easy to implement:

1. either keep the name map uncoupled from the serializer, give the user some attributes (requires discovery) or a plain good old option in YAML for input, then decorate the serializer with some code that takes the "input name", convert it to a class name, then call the real serializer.

2. same thing, but couple the map inside the serializer (configuration remain the same).

I much prefer the first one, but both are doable quite easily (done it for client projects in https://github.com/makinacorpus/php-normalization)

The hardest problem here is that messages are not services, and will never be, hence core developers will not allow abusing the dependency injection registration mechanism to parse class attributes (ie. making messages false services, parse them, register the names, then drop them as being services).