MQTT suport for sending data

[Havelock-Vetinari]

This PR adds support for MQTT broker: Supports

• custom topic prefix
• custom client id
• login and password for broker connection

Each sensor is exported to separate topic. Example:

tele/namf-01/sensors/SDS_P1
{"value":6.87,"unit":"µg/m³","timestamp":"Sat Oct  3 10:24:54 2020"}

tele/namf-01/sensors/BME280_temperature
{"value":24.5,"unit":"°C","timestamp":"Sat Oct  3 10:24:54 2020"}

Messages are not retained. If auth error occures, reconnecting while executing mqtt::loop is postponed for 1024 loops or until sending data is required.

[gpatryk]

Any plans to merge the PR?

[netmaniac]

Sorry for delay. We don't use MQTT by our self, so had to get through new topic for us. We'd like but there are some caveats.

• Don't have final results, but from first tests - adding code from this PR uses from 512 up to 1kB of RAM. While this is not showstopper yet, it is noticeable amount, especially MQTT won't be used in most sensor instances. So, I could move MQTT support to new scheduler and use dynamic memory assignment but first there are other things to fix with this PR
• problematic re-connection strategy. If there is problem with connection to MQTT server (I did mistake and entered hostname with space added) NAM becomes unresponsive. ESP is busy trying reconnect with each parameter being sent not handling anything else. Maybe skipping whole send on first failed reconnect would be better
• additional info about units - while it is nice to have units also, but this also have negative memory impact. JSON used to store values is verbose, adding units to JSON uses additional 14-18 bytes for each reading. Seems not much, but for some NAM having SPS30 or other additional sensors reports 10 or 15 physical values.

Why memory impact is so important? Right now NAMF has 24-26 kB of free RAM. While it looks a lot, it is not :) For example config page can have 16 kB (depending on number of enabled sensors) and it is being build in memory. Add memory fragmentation and some required overhead for temporary variables and then each 200-500 bytes are important :)

So, we can merge MQTT support, but after two changes:

• better reconnection strategy
• no unit information added to JSON

With this changes I can merge PR and move code under new scheduler. With change to use dynamic memory for PubSubClient I think memory impact will be acceptable.

Is this possible @Havelock-Vetinari ?

[netmaniac]

@Havelock-Vetinari BTW - do missing ifdef in language .h files have been a issue? In build process they are included only once, and don't see need for use ifdefs?

[Havelock-Vetinari]

Hi! I will try to find some time this or maybe next. Removing units should not be a problem. @netmaniac can you provide me with some kind of documentation or hint for using scheduler? I can also try to add conditional compiling of MQTT support, so it may be an opt-in feature.

While working on this change, one thing catched my attention. Json structure is serialised to string after being gathered from sensor and then parsed back to json structure in sending functions. Is there any particular reason for this aproach?

[netmaniac]

@netmaniac can you provide me with some kind of documentation or hint for using scheduler? This will be described more with SDS moved to new scheduler, work is in progress.

While working on this change, one thing catched my attention. Json structure is serialised to string after being gathered from sensor and then parsed back to json structure in sending functions. Is there any particular reason for this aproach?

This is legacy from Lufdtaten firmware. Will look later for some more robust solution

[Mchl]

Hi @Havelock-Vetinari @netmaniac I would love to see MQTT support in NAMF. Actually I've just been browsing the repo to see if it would be easy to add it myself. So, just some words of encouragement :)