Implementation of MQTT at Edinburgh GIN

[SimonFlower]

This issue originated in issue #6

Architecture for implementation of MQTT.

The MQTT broker is off-the-shelf open source software. The clients are bespoke programs, using an MQTT client library to gain access to the MQTT broker. Questions:

• Is this right?
• Are more MQTT servers involved, for example at the institutes?
• Is server-to-server communication possible and/or desirable?):

Things that need to be decided for implementation at the Edinburgh GIN

To load data into the GIN's data store the following metadata is needed:

• Observatory IAGA code
• Cadence (minute or second)
• Data type (variation, provisional, quasi-definitive, ...)
• Component orientation (HDZ, XYZ, F, G, ...)

For the data, the following is required:

• Time stamp (only the start time is needed if we specify the cadence)
• An array of data samples

Questions:

• Which implementation of the MQTT server to use? Ideally the chosen solution should be available as a Docker image, ready for deployment into Kubernetes. The Mosquito server is available as a Docker image.
• Which MQTT client library to use to build the subscriber client that will load Geomagnetic data into the Edinburgh GIN data store (see recommendations from Roman and Stephan below). The client will be written in C.
• Which parts of the metadata should go in the topic, which in the message body
• How to structure the MQTT topic, so that messages can be filtered as required by clients (thinking here not just of getting data into the Edinburgh GIN, but of wider use of MQTT in the community).
• How to format the payload in the message body (e.g. CSV, JSON, binary, ...)?

Notes from Roman

Required installations for full MQTT support (and python support)

• mosquitto (MQTT client - broker) sudo apt-get install mosquitto mosquitto-clients
• paho-mqtt (MQTT python) sudo pip install paho-mqtt

Broker installation: https://github.com/geomagpy/MARTAS#5-setup-of-a-broker

Authentication and ssl: https://github.com/geomagpy/MARTAS#23-install-mqtt

Quality-of-service (or how to verify the identity between send and received): https://github.com/geomagpy/MARTAS#33-understanding-quality-of-service-qos

Notes from Stephan

You can find the client/server libraries here https://mqtt.org/getting-started/

I also found it very useful to go through the explanations written down by the people from HiveMQ https://www.hivemq.com/mqtt-essentials/ It is well written and complete info on what is mqtt and how to use it.

As you can see the broker used by me and Roman is mosquitto.(https://mosquitto.org/) which is easy to set up.

As client library people often tend to use paho-mqtt because it is one of the first packages available and supports major programming languages (https://www.eclipse.org/paho/index.php?page=downloads.php). I used lots of different clients and all were easy to setup and straight forward to use.

[stephanbracke]

Some comments

First of all is this right : Yes. Are more mqtt servers involved ? Not necessarily. In the institute you can use your own mqtt server and bridge just the topics that are needed towards the MQTT broker at BGS DMZ. What are the advantages :

• your mqtt server (for example mosquitto) will be the one that maintains the connection towards the remote mqtt broker.
• it assures reconnections when failures occur, queuing of messages when connection is broken, persistency can be configured etc
• the client-software on the instruments can be local to this broker and the connection is not easily broken
• just configuration no code involved

disadvantage :

• you need to configure and setup this broker locally (not so difficult)

Personnally I use a broker in Antarctica that is bridged towards a broker in Dourbes ( for the one in Dourbes it is like a normal client no bridge config on the broker in Dourbes) When You don't use a local broker you need to provide a sort of caching/resend/reconnect mechanism in your client code because when client disconnect in general the messages are just lost. In the end it is a decision on the institute level and will not have big influences on the way you implement it on BGS side. I have to admit that I only have experience with bridging mosquitto with mosquitto brokers so I don't know what happens if you use different brokers. The reason why I used mosquitto is simple : it does the job for what I need and I had no big issues with it.Naturally I don't have big load.

[SimonFlower]

Thanks for all the information everyone. It looks to me as though one of the next tasks is for us to agree a payload format for Geomag data in Intermagnet (which I will need to know in order to start writing the software to receive MQTT data into the Edinburgh GIN). Are there any existing candidates for this format from people who are already using MQTT or do we need to design an Intermagnet MQTT payload format from scratch? I think these are the things we need to resolve (please tell me if I've missed anything):

• Which parts of the necessary Geomag metadata should go in the topic, which in the message body. I think required Metadata is:
  • Observatory IAGA code
  • Type of data (variation, provisional, quasi-definitive, definitive)
  • Cadence (minute or second)
  • Component(s) in payload: X, Y, Z, ... - perhaps this should also indicate whether "F" is from an independent instrument?
• How to structure the MQTT topic, so that messages can be filtered as required by clients (thinking here not just of getting data into the Edinburgh GIN, but of wider use of MQTT in the community). Should the topic be a composite of the metadata listed above (e.g. IAGA-CODE/DataType/Cadence) or should we define a Topic name for Intermagnet data (the Edinburgh GIN will want to receive all Intermagnet data, so this would be a simple filter for me, but might not work so well for others)?
• How to format the payload in the message body (e.g. CSV, JSON, binary, ...)?

[SimonFlower]

I've put together a draft service description for an MQTT service to be implemented at the Edinburgh GIN, including a suggestion for the payload format. Please can you have a look https://github.com/INTERMAGNET/wg-www-gins-data-formats/tree/master/ImagMQTT and send me any comments / alterations by the end of December? Please do let me know if this deadline is too short and I'll extend it, but I would like to get started on implementation early in the New Year. Please can people with MQTT expertise check what I've said about the MQTT configuration - my knowledge of this comes only from reading documentation, not from experience!