Single command bus

[rosstuck]

Earlier this week, we had a quick chat about the new Router replacement system and how that could create an infinite queuing loop if given the same command bus.

I'd rather not force people to have two command buses because:

• they'll likely end up duplicating the config exactly, minus the queuing middleware
• it's confusing to explain in the docs
• it complicates framework integration

@sagikazarmark's earlier response was that this might cause more bootstrapping than folks intend for a light-weight worker, but I don't think that's a given and is more a choice of DI container/framework than Tactician's responsibility.

Anyways, I was thinking about ways of resolving this. The best idea I've had so far is that we keep the initial queuing the exact same (user drops a QueuableCommand in, that gets serialized, etc). Then on the worker side, it gets popped out of the queue and sent to the router.

Here's where the two changes start: rather than returning [$this->commandBus, 'handle'] we return

return function () use ($envelope) {
    $queuedCommand = new QueuedCommand($envelope->getMessage);
    $this->commandBus->handle()
};

Then in the QueueMiddleware, we add an extra if statement to avoid the loop:

if ($command instanceof QueuedCommand) {
    return $next($queuedCommand->getRawCommand());
}

In short, we signal the state to the QueueMiddleware by wrapping the Command in a QueuedCommand wrapper. That prevents the infinite loop and lets us use one command bus everywhere. It's a little bit "different" for the internals but easier for the consuming user (provided they take our advice about putting the QueueMiddleware always go at the top of the list).

Thoughts?

[sagikazarmark]

I am going to repeat myself to have the complete response of mine here:

• In the most ideal case your worker instance is not even on the same machine. That means you have to create another config.
• In a less ideal case the worker instance runs on the same machine, BUT separated from the application. Same as above: you still have to configure a command bus again.
• In a NOT ideal case you bootstrap your application in the worker AND use the same command bus. First: don't do that. Second: If you do that, use a separate command bus. Third: If you want to use the same command bus, configure the DI to identify which environment it runs (app, worker) so it can skip the queue middleware.

I really dislike the solution above as it introduces a responsibility which is:

1. clearly NOT a responsibility of a queue middleware (to know if the queue is coming from a queue or not). (It's like: ExecutedCommand. CommandBus should know if the command has been executed before)
2. needed in a case which breaks the whole concept of queue-worker implementations. (See the NOT ideal case)
3. beacause of laziness of developers. If someone doesn't know what queues are for then he/she probably shouldn't use it.

P.S.: Hm, rereading this whole comment it seems like I started a war against your idea @rosstuck. While it's not my intention, it shows that I'm quite against the idea. Based on my experiences, if you need this functionality, you are doing something wrong. If you really want this implemented, it should go into a LazyQueueMiddleware or something, because even if I accept that someone wants it, it should not be in the "core" plugin. But again, rather leave this whole thing. :stuck_out_tongue_closed_eyes:

[rosstuck]

Hehehe, no hard feelings! It's code, not personal. :smile:

I do think there's a key difference here: regardless of which machine the worker runs on, you're probably best off checking out the same codebase there, so your configs are centralized. It's about whether or not you run the code, not whether or not it's present. Lots of huge scale companies run with single massive codebases. So, the physical machine / configuration argument isn't relevant here, I think.

I don't think this breaks the implementation or responsibilities here, since the middleware is in charge of whether or not to queue or dispatch onwards. This is just a subcase of "dispatch onwards" to me.

I can see an argument for having two versions of this, but that needs to be considered carefully since this version relies on a change in the router as well (which isn't kind to that approach).

[sagikazarmark]

If I am correct than the root of our disagreement is that:

• you say the QueueMiddleware should decide whether it should pass the command to the queue or not (the command originates from a queue or not)
• I say the QueueMiddleware should not even be in the Command Bus

[sagikazarmark]

Infinite loop is only possible of you pass commands to the same queue. However if you want to pass every command to a secondary queue from the consumer (eg. a statistics of commands executed) you wouldn't be able to, since it comes from a queue. This is a good example of why I think having this limitation by default is a bad idea.

[sagikazarmark]

@rosstuck if the envelope is added BEFORE queueing in the LazyQueueMiddleware than the router can be unaffected, thus the whole implementation can be in one place.

[rosstuck]

@sagikazarmark I suspect the root is who should decide whether a command is queued, the controller layer or the service layer.

How about I just send a PR on Thursday or Friday to see if we can compare and/or ship two implementations here? Because it seems we have two completely different takes on this issue and we'll talk in circles otherwise.

[sagikazarmark]

@rosstuck agreed, we have different priorities here and I think this question is already solved that there should be a solution for the mentioned problem, we are only arguing about the "default" behaviour.

Looking forward to see the PR.

Till then here is my solution mentioned earlier:

LazyQueueMiddleware:

    /**
     * {@inheritdoc}
     */
    public function execute(Command $command, callable $next)
    {
        if ($command instanceof Message) {
            if ($command instanceof QueuedCommand) {
                $command = $command->getCommand();
            } else {
                $this->queue->enqueue(new Envelope(new QueuedCommand($command)));

                return;
            }
        }

        return $next($command);
    }

class QueuedCommand implements QueueableCommand
{
    /**
     * @var Command
     */
    protected $command;

    /**
     * @param Command $command
     */
    public function __construct(Command $command)
    {
        $this->command = $command;
    }

    /**
     * Returns the wrapped command
     *
     * @return Command
     */
    public function getCommand()
    {
        return $this->command;
    }
}

It makes the queued object graph more complex, but can be in a single class without caring about the router.

[rosstuck]

Finally got some time to think about this a little more.

I like how how your iteration above keeps it limited to one class. The naming is tricky (I don't want to call it Lazy because that usually refers to lazily loaded) but that could very well work.

One potential downside is that if someone ever switches from the wrapped version to the non-wrapped version, their pending, serialized commands would begin to fail because the non-wrapped version won't find a handler for the QueuedCommand. We could ship a third middleware that does nothing except unwrapping but that gets a little weird.

Also, the more I think about it, I feel like the choice about using the same or separate command bus for the worker should be isolated to just a single place: which instance we passed to the Router. That should be the only part of the system that knows or cares, and while this is technically true, we have some mingled concerns around that now.

So, to resolve those, I have a crazy bizarre compromise to propose: what if we use your pre-wrapping version and require that you always put the QueueMiddleware at the front of your Command Bus, whether it's the same or separate bus that originally queued it? In other words, we always wrap the command.

Doing so has the following properties:

• The user doesn't have to think about which middleware to use.
• The user doesn't have to worry about migration strategies, like draining the queue before switching from wrapped->unwrapped.
• You can use a single or separate command bus without changing any other code.
• There's only way place you have to worry about which command bus to use (the router)
• The bootstrap weight of instantiating the QueueMiddleware for the worker approaches zero, the Bernard infrastructure is already there and running.
• We have a place to add any extra metadata we need/want in the future.

Downsides:

• We should clearly document that you need the QueueMiddleware first on both sides
• An extra layer of wrapping makes the job "opaque" to non Tactician consumers but Bernard's envelope structure kinda does that already and it wouldn't be hard to parse around it for those who really need to.

I know that sounds really unintuitive at first but if we're trying to eliminate the number of branching possibilities here, I think this might be the best compromise for everyone, especially the end users. Give it a once over and let me know what you think. :smiley:

[sagikazarmark]

Do you say that WE should care about whether a developer is stupid enough to exchange the middleware to something that should clearly not work? What other possible reasons can you mention for doing that?

The point of having two, separate implementations is that you can choose the behaviour you expect from the command bus. So far your argument was that end users want to use the same codebase. Why would anyone change it then?

I don't like the always wrap idea, because I don't like the wrap idea at all. I think it is a workaround for those who want to use "the same codebase". So I think this idea eases to use it like you would like to, but makes it harder how I would like to. What I proposed solves the problem as two equally important one. The fact that exchanging them ruins everything...well, choose a path and stick to it. It doesn't worth making one bad to make the other the ultimate one.

Regarding the router: it is just an interaction point with the consumer, it shouldn't be involved in any of these actions as it will never be aware of the command bus context.

As I said, the two problems should not be mixed up.

[sagikazarmark]

Either this or I don't completely get your idea.

[rosstuck]

Yes, we should care about making things easy for our users. Also, let's not call anyone stupid. :disappointed:

Here's the best reason I can give. Let's say we split this into two different middleware: QueueToSameBusMiddleware and QueueToDifferentBusMiddleware. If we split this into two middleware, say, we need to write this in the docs:

Separate Middleware Docs

• You need to choose between two different middleware (with accompanying explanation, a decision most likely don't care about).
• If you choose the QueueToDifferentBusMiddleware, make sure you configure and pass a separate bus to the Bernard Router as well.
• We recommend always having your QueueMiddleware as the first middleware in your stack, unless if you're using the QueueToDifferentBus and configuring the CommandBus for
• If you ever change from one to the other, drain your queues first, otherwise expect infinite loops or missing handler exceptions

If we choose a single middleware approach, we need to write this in the docs:

Single Middleware Docs

• Always having QueueMiddleware as the first middleware in your stack, whether the bus is sending or receiving from the queue.

There's a different burden of knowledge here on the end user and a lot fewer variations to support. You and I know a lot about the internals of how this stuff works, but it's not intuitive to folks who just want to use this and not worry about what's happening internally.

Does this place an extra burden on the QueueToDifferentBus folks? Yes. We are adding the requirement to have one extra middleware in their stack up front. But I think it's a pretty low burden, honestly, and it will be easier for us to support in the future since there's fewer variations possible here (both in code and queue state).

Finally, to gain market share, we need to be really easy to use. The framework integration packages are going to lean towards a single command bus solution, so we should lower the barrier for that, offering users a plug'n'play solution without worrying internally how things are implemented.

[rosstuck]

As a separate or alternate though, if this really boils down to the wrap, we could consider a base class/interface/trait setup that allows setting a mutable flag (isAlreadyQueued) that eliminates that need for the wrap and middleware on the other side. This also allows other middleware to see the true command without having to peel the wrapper first, so it doesn't always need to be first in the list.

It does require tampering with the interface on the class though and a little more setup per command.

Still, that could work too and might solve everything.

[rosstuck]

Also, gonna ping @andrewcairns as well because he's actually dealing with this use case in practice and I'd like to hear his thoughts on either of these solutions. :)

[sagikazarmark]

Your argument is that we have to document more features if we have two separate things. This is true, but you are proposing to have only one feature and call the other a workaround. Also, I don't think having to document more things is a valid argument, because the need of documentation should not have effect on features.

I don't really understand the easy to use and market share arguments. You have one "easy" way to use and there is another. Whichever you choose, your call. What is so difficult about it?

If you remember, I wanted to have a way to control from the code whether a command should be sent to a queue or not. Your point then was to have an interface or something to make this decision. Now we changed places. ;)

I am gonna try to get something out of this, because we are not going to move from one to two otherwise. Let me think about a possible solution.

[sagikazarmark]

I can imagine an always wrapping logic, but in this case, this wrapper also has to be the decision if the command should be queued or not. In other words: having a QueueableCommand interface and a QueuedCommand is redundant. We should choose one of them.

In this case, we should either make it necessary to pass a wrapped command to the queue, or if the QueueMiddleware is in the bus, we have to catch all commands and if it is not from a queue it should be sent to one. In other words: either wrap by the user, or send all commands to the queue by default.

[sagikazarmark]

I made a PR which probably solves all you issues. However I make one last objections: this is WRONG. We should support both options, because the same codebase and all other things are one possible way to do things. The way I do things is an other. None of them is better than the other, simply different thus we should support both way. This idea basically forces me to do things how I wouldn't like to and I have bad feelings about it.

So I will probably remove the PR and let you do what you want.

[sagikazarmark]

Take your time, study them all and choose the last one. ;) I don't even care if you document only that one.

[rosstuck]

Hey, sorry for the late responses, I've been onsite with a remote client all week.

Crazy awesome work on the PRs. :smiley_cat:

• I agree #12 is meh.

• 13 doesn't look that bad. I actually like that and we keep the single middleware. If we tweak the name of the routers, I'm down for it actually. :)

• 14 is also okay but we get into the split middleware thing again. So, it works but I like #13 better at this point. :)

[rosstuck]

To clarify, I'm okay with either of the last two so pick your fave and we'll go from there

[sagikazarmark]

Actually I realized that if we support BOTH in one middleware, it might not be a problem. So my vote goes to #13.

[sagikazarmark]

However I am not sure if it makes any difference, since we have to document two routers instead of one.

[rosstuck]

While admitting that neither of us really liked #12, in light of discussions on #15, what would you think about an interface like this?

interface QueueableCommand / CanBeQueued
{
    public function getQueueStatus();
    public function changeQueueStatus();

    public function getQueuePriority();
    public function getQueueName();
}

We could also supply a trait that has all the necessary code and returns Priority/Name default values as well. I also dislike the mutability of the QueueStatus but if there are going to be other options, this might be the simplest way to resolve all of that in one clean pass.

Just throwing it out there. :smile:

[sagikazarmark]

I have multiple reasons against it:

• It adds mutability to the command, which really shouldn't be there.
• It adds an overhead, since the Serializer has to deal with this data as well. Serialization is currently out of scope (from the point of Tactician), but it is not so easy. Symfony Serializer component (which is currently used by Bernard) contains a Property serializer (not sure about the name) which serializes objects using Reflection (which is bad). So the proper usage is to create serializers for all kind of commands. (I know, sounds insane). If this is the case, adding such properties to the command is weird. @henrikbjorn anything to add here?
• The QueueableCommand is just a helper interface, the middleware actually relies on Command and Bernard's Message directly. In other words: if you implement those interfaces in your command, you can still use it. And it is nicely aligned with SRP.
• Regardless the fact that Message interface requires a method to be implemented on the command, commands should be kept as small and clean as possible.

From the other side, rather this than wrapping the command, because the middleware architecture could really be messed up with that solution. (Actually I am thinking about a general command envelope pattern used all over in tactician, which could generalize the need for envelops and ease the usage of them, but that is just an idea).