Evaluate EMQTT

[drasko]

EMQTT (http://emqtt.io/) looks like very interesting potential replacement for NATS.

Reasons are following:

• We need at least one MQTT server - why then no use it as a main broker also
• Massive scalability (Erlang shines here)
• Multiple protocol support - WS is actually MQTT-over-WS but this should be more than enough for the start. Good news is CoAP support: https://github.com/emqtt/emqttd/issues/510
• With small HTTP API interface in front (from mainflux-http-server) and mainflux-core-server in the back this set-up should practically work out-of-the-box

We need to examine:

• Auth - how do we handle security. We have to decide what of emqtt plugins to use (HTTP connection with Hydra?)
• Should we eventually add all code as Erlang plugins

[KrishnaPG]

+1 for eMQTT

However, it would be better to restrict it to be just MQTT server / interface than making it central piece (currently held by NATS).

The central piece (which is currently NATs) need to be revisited too. IIRC, NATS does not have persistence (sacrifices durability for speed).

Going over some of the code of mainflux roughly, I observed that NATS is being used as sort of 'queue' for managing workloads (for example, reply http responses)

If that is the case, in a production environment I would expect Kafka or Darner to take care of the durability scenarios for HA.

One good thing is eMQTT has plugin support for Kafka (and MongoDB, ofcourse). The MongoDB plugin has Auth support, which is a bonus.

For the CEP side, I would suggest RethinkDB change feeds is a good starting point. Influx has a long way to go before it can offer such 'live watch/updates'. Influx may serve as historical storage for visualization meanwhile (though influx's storage space requirement is not that optimal). For realtime analytics of time-series, one other option worth considering is Cassandra (or its latest c++ high-performance equivalent ScyllaDB ), though they are hard to tame.

Combining Rethink with DeepStream is a good option for live dashboards.

[drasko]

@KrishnaPG thanks a lot for comments! We are on the same wavelength.

eMQTT needs some investigation, and I really admire their work. However, I would like that we stay on this Gopher path for now (let people master Go throughout the project and harness the community).

NATS can now be used as what they call "Streaming Server": http://nats.io/documentation/streaming/nats-streaming-intro/, and we really have to switch to this mode to guarantee the persistence and delivery. I will have to ask Brian from Apcera, but I think they have been pushed from the pople who really needed these features (even though Derek did not want this arch in the begining AFAIR).

Yes, NATS is currently just a dumb message broker, but it will be probably turned into message queue later, as I doubt that backend will be capable to handle the requests quickly.

I am trying to avoid Kafka because of complexity (Zookeper). I think it is of crucial importance that Mainflux be as simple as possible - even now this all is overkill for deployment, and I am working currently only on Maiflux Lite (https://github.com/Mainflux/mainflux-lite) which is HTTP API and backend bundled in one binary (so we can skip NATS for now).

Darner looks interesting (I was not aware of it) - but last code update 4 years ago? BTW, if we go C/C++ path I would rather look at what @antirez is doing: https://github.com/antirez/disque

Regarding RethinkDB - I think this is a way to go. We have been recently evaluating Hydra (https://github.com/ory-am/hydra), we will try to integrate it ASAP - @nmarcetic already started working on this.

Cassandra is very nice, but somehow I see it complicated (maybe I am wrong). And then again, we are going in that Kafka+Cassandra that I would like to avoid (at least for now until we prove we can not do it simpler).

Fot CEP I was thinking Kapacitor - https://influxdata.com/time-series-platform/kapacitor/. I never used it but it looks promising, and I thought it will sit very nicely with Influx.

Did you maybe had some experience with Kapcitor+Influx?

As for the RethinkChange feeds - I guess that these messagess would have to go back though the NATS, as apps will be wither connected to their API servers (MQTT, WS, CoAP), and data can not be pushed directly from RethinkDB (sensor knows only MQTT protocol for example).

I have been looking in DeepStream for some time now, but I am not sure how we can leverage it and how it can be easily integrated... But I am completely open to this suggestion, I like what this guys are doing.

[drasko]

@mijicd related to the problem we were seeing with double Auth/ACL call in superuser case: https://github.com/emqtt/emqttd/issues/696#issuecomment-250963711

[drasko]

I think we can close this one for now. As a conclusion we will be using EMQTT as the central broker.