Split to separate distribution AnyEvent::MQTT5

[nuclight]

Thanks for the great work!

But, could you please split MQTT client and broker to separate distribution? There are currently no MQTT 5.0 modules on CPAN, except of yours, and for one-machine / no Web/JSONRPC setup - using entire Beekeper for just toy broker seems a significant overhead.

[jmico]

I've always had in mind splitting the distribution into submodules, but it is not a trivial thing to do.

ToyBroker, as its name hints, was never intended to be a full fledged broker. It is part of the test suite of Beekeeper and implements just the bare minimum functionality needed to run Beekeeper applications. Eventually it might be part of a Beekeeper::Test distribution.

I also planned to put the MQTT client into its own distribution, inside the AnyEvent::MQTT namespace. In order to do that I need to create a comprehensive test suite, implement the version 3 of the protocol, and handle the differences in protocol implementations between popular brokers. It is doable, but as i said, it is not a trivial thing to do.

What are you trying to do? Beekeeper is a framework for building distributed applications. If you are only interested in ToyBroker, I strongly suggest to use Mosquitto instead, which is a pretty good broker, well documented and maintained, and easy to install.

If you are interested only in the client you can use it stand-alone, as Beekeeper::MQTT is already a self contained module (see the documentation):

use Beekeeper::MQTT;

my $mqtt = Beekeeper::MQTT->new(
    host     => 'localhost',
    username => 'guest',
    password => 'guest',
);

$mqtt->connect( 
    blocking => 1,
);

$mqtt->publish(
    topic   => 'foo/bar',
    payload => 'Hello',
);

[nuclight]

I've forgot to pull explanations from our e-mail conversation to this ticket, so I'll do this now (better late than never) for ability of public references, and I'll also add reference to https://github.com/nuclight/tkxgram/blob/master/docs/db.md which don't (yet) explicitly describes this idea of MQTT5 using (but does so implicitly):

Subject: Re: Split to separate distribution AnyEvent::MQTT5
Date: Fri, 24 Dec 2021 03:30:31 +0300
X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; amd64-portbld-freebsd11.4)

At Thu, 23 Dec 2021 05:21:39 -0300
José Micó <jose.mico@gmail.com> wrote:

> I am not sure that Beekeeper is the proper tool to solve your use
> case.  

Yes, I know, 

> Let me know what are you trying to do, and I will be glad to help.  

Well, this is a long story, take a cup of tea or coffee...

There is a messenger called Telegram, https://telegram.org (recently reached 2^31 threshold for their user IDs). And it is internally very, VERY weird. They formally have open-source clients and even documentation, but in reality it is hard to even understand, not even to say "implement", without deep diving to code of their clients, which is even published with several months after release...

So in fact most alternative clients just use their code (published lib), making only cosmetic changes (GUI, etc.).

About three years ago, Telegram started to deploy changes which regain control from users over the received messages. As this is a "cloud" messenger, where everything, like in Web, stored on their servers, it's easy. First it was ability to delete messages in peer's inbox for him, not only for self. From the very last changes - ability to forbid saving media, copypaste and even screenshots.

While from some points of view this is viewable as censorship-like, consider following situation: someone threatens to kill you in private chat, you want to go to police, but then they wipe it, and you have no evidence anymore. I've heard of real such cases, with other messengers, though - when lost phone meant lost correspondence.

And as you probably know, that in just decade or two ago Internet was different - nobody could revoke from you logs (mail, chats, etc.) which you received! This was absolutely normal, but now they want to take away our freedom.

So I with my former teamlead wrote a library in Perl for Telegram, from scratch. It works, I have a logger running on my machine, putting machine-readable dumps (my fork is at https://github.com/nuclight/teleperl).

However, it's not very handy to dig by Data::Dumper, also when you have been banned from some chats, etc. So I want to write a GUI client for this, which - important! - would have also work on casual user's machine. No administration, no configuring brokers or database schema's upgrades... BTW, this means the only DB suitable is SQLite (deployed in every iron and toaster, very well tested, 100% no DBA needed).

So this is a very challenging and complex task. Apart from DB side, where there is need to invent something to keep historical data forever while avoiding schema upgrades/downgrades, challenging by itself (could see at https://sqlite.org/forum/forumpost/053105c7ae96fb47 if curious, no one was able to point to ready solution for such tasks) - there are other problems. For example, our decoder from their binary serialization format produced classes in Telegram::* namespace. Then, suddenly, their server started to produce undeserializable data. We found that these were blobs from different schema version, in violation of negotiated version. And one can't have different but same named classes like Telegram::Message in same Perl process - so I developed a big crutch with several processes (AnyEvent::Fork::RPC) for unmarshalling - whichever process with it's version who can, then it responds.

This year this server bug disappeared, but who knows...

"Well", I thought some time later, "if I'm forced to several processes, I can make benfits from it!". First, it reminded me of Quassel IRC client which is split to server part (core) and GUI part, and possibly several GUIs, e.g. from mobile (were connectivity loss is often so it's not good fro IRC), can connect to same core.

Then I thought about plugins, and may be support of other protocols, or other code/libraries, which are not always support async or even could be not very reliable, but crash of separate process will not take down entire messenger (you know, both major browsers, Chrome and Firefox, are multiprocess for this reason, too).

Then in multiprocess environment one need to some kind of protocol to communicate them. I've heard about MQTT earlier, but recently discovered that version 5.0 ideally fits for my task - it has user properties and response topics! For example, I could put a user property like "next-response-topic" in addition of simply response-topic, receiver will process and shift from this array to response-topic, thus allowing for plugins to form a chain of processing like pipes in Unix! And to deal with Telegram brainfarts: for example, when a GUI will make an "RPC" call, the DB process could subscribe to subtree of response topics for GUI - thus intercepting answer, too, processing it for info about users, chats, etc. to save - transparently for GUI.

Now remember that client should be simple enough for casual user, running on Windows machine. This means self-contained app, no frightening Mosquitto downloadin, install, configuration, etc. May be write several simple lines for plugins of messenger itself (if advanced user), but no other software.

And then I've found Beekeper. Obviously, it is solution for another class of tasks. But toy broker perfectly fits for one-machine (say, 3-4 processes) unattended installation, QoS>1 is also not needed inside one host. Of course, at the end i could just copy paste to modules and patch for working without Beekeper, but keeping forks is generally not a good thing in Open Source world.

I've looked to about other parts of Beekeper, but it also seems to be targeted to Linux only, so again not...

As for version 3/testing. Note that on CPAN there are several other MQTT (and some are even on AnyEvent) modules, which just put a disclaimer about unstable API subject to change Work In Progress etc. Thus, I think, putting just two modules, toy broker and client, with such disclaimer - are just fine to start with.

So, that's answer - why.

-- 
WBR, tg://@nuclight