search

nebkat / esp32-xbee

ESP32 XBee UART Interface
GNU General Public License v3.0
89 stars 46 forks source link

MQTT Protocol #7

Open hagre opened 4 years ago

hagre commented 4 years ago

Hello.

I am working (experimenting) on transmitting the RTCM messages with MQTT frames via Mosquito Brocker and TLS connection. I have seen this idea on: https://github.com/GeoscienceAustralia/gnss-mqtt http://www.ignss2018.unsw.edu.au/sites/ignss2018/files/u80/Slides/D2-S3-ThC-Wang.pdf https://esprtk.wordpress.com

I know it is not the actual standard of transmitting RTCM data, but there are some advanages for DIY projects. e.g Building a secure "caster" / broker was very easy.

Please consider to include a MQTT feature in your software. thx

hagre commented 4 years ago

I just published an very simple MQTT - RTCM bridge project i am working on. https://github.com/hagre/RVMT-RTCM-VIA-MQTT-TRANSMITTER

It is a blocking code, so direct integarion is not an option (and code is not written very nice) and still under development status. Hopefully you are still interrested in including MQTT as transport protocoll for the correction data. What i would like to have is: Your software connecting to an MQTT broker (TLS) and spreading the RTCM data on a local NTRIP server for all standard equipmet (what your software is already doing).

The other way arround, at the basestation, RTCM to MQTT is not so important for me, because my software can do this for me already.

DavidKelleySCSC commented 4 years ago

hagre

Here is a challenge if you are willing. If you can find two other parties who also think adding MQTT support is a good idea for NTRIP, I can arrange to have such a feature added to the SNIP NTRIP Caster (the software found on RTK2go.com) in a couple of weeks of effort (at least doing v3.1.1). I am attracted to adding other secure transport methods to NTRIP in general, and this one seems fairly trivial to support. I also feel that one of the public papers claiming that MQTT is more efficient than NTRIP is nonsense (they are more or less the same IMO, just TCP/IP transport), but apart from that point, this looks like a classic simple publish/subscribe system that is easily mapped to existing NTRIP concepts. Any interest?

hagre commented 4 years ago

DavidKelleySCSC

Basicly i am very interessted in this feature, but i am not quite sure if i am the right person to find other parties for this project. I have just posted this idea on an agricultural forum with "medium" resonance. I will link your offer to my post in the forum and we will see ... . For me personaly, it was relatively easy to get a simple MQTT relay for the RTCM data, but my attempt to rebuild a regular NTRIP V2 caster was not so lucky (with a big lack of protocol details). This was the reason to ask nebkat to add MQTT to his nice looking GUI featured firmware.

I just started to think about rebuilding a caster like python solution on a MQTT broker and reproduce some of the required features, to give this projet a better chance to be accepted and improved.

Even, for me as a diy-self-hoster, i can see good reasons and a lot of possible usecases and users looking forward to use a well maintained stabil MQTT server/broker with a fixed IP to supply RTCM and possibly other nice fetures (wether/soil/..) data from their farms.

One of the bigest reasons for me to not use RTK2go.com was the permanent data stream (even if this are just some bytes) when they are not used 99% of time. I woud like to see a client subscribing to a caster/broker - the broker is swiching the datastream on with a simple MQTT ON msg to the corresponding base - and OFF if no client requests this basestation anymore.

thanks for your offer

hagre commented 4 years ago

If you are interrested, you can follow the discussion on: https://agopengps.discourse.group/t/ntrip-caster-alternative/2167

nebkat commented 4 years ago

I am currently rewriting parts of this project to allow better customization of the available protocols. Instead of being limited to a single instance of each protocol (e.g. single NTRIP server), it will be possible to set up multiple such connections. The protocols will also no longer be hardcoded to UART1, but instead drag/drop connections can be made between various data sources:

@DavidKelleySCSC I also can't see how MQTT would be more efficient, but indeed it is probably no harm to add it. I will support it here, and it's nice to have it as a more open protocol than NTRIP.

gdt commented 4 years ago

It would be nice if there were a published protocol specification for RTCM over MQTT, selecting mountpoints, and so on. The MQTT protocol is defined, but each "do foo over MQTT" really needs a written protocol specification.

DavidKelleySCSC commented 4 years ago

Does gdt + nebkat + hagre now make three? Just asking, no pressure.

It seems to me (speaking only for myself and not for RTCM SC104 in any way, or for NTRIP) a draft protocol spec could be developed in a few hours of discussion if the initial scope and goal was simply to codify a means to select existing mountPts as streams. If the next logical step was to be able to select 'parts' of the stream in a true publish/subscribe method where new content was only sent out only when there were changes, that is a much tougher problem for another day IMO. If there is a group that wanted to start this, I would be willing to take it the RTCM NTRIP working group for further work and consideration as a formal standard.

gdt commented 4 years ago

No, at the moment I do not really see the case for MQTT to replace NTRIP. As I see it, the major problem with NTRIP is the lack of an available-in-practice specification. (For example, the gpsd developer group, of which I am part, doesn't have a copy of the spec due to cost.) But, if there were an MQTT protocol with widespread support, that might be nice.

My real point in writing was to make the point that "let's do this over MQTT" is not a protocol specfication, and the modern world has too much of "the way this program implemented it is the spec but it's not really written down".

hagre commented 4 years ago

Thank you for your participation! Some month ago, i did a short brainstorming regarding the protocol. Working to implement it in my project made me think to do some changes to that draft, but i had no time to revise the repository. https://github.com/hagre/RVMP_RTCM-VIA-MQTT-PROTOCOL Maybe it is some kind of starting point for an discussion.

umeat commented 4 years ago

I'm currently putting together some information on what I see as the benefits of using MQTT for GNSS data. If I publish something with more detail I'll link to it here. Was happy to come across this thread discussing it. Here's a summary of my thoughts:

The "efficiency" benefits that I see are strictly around the way which you can use MQTT. It was mentioned here that NTRIP and MQTT both use TCP, and so they are both equally "efficient" once data is on the wire. If you publish all of your RTCM messages to a single MQTT topic in the same fashion as you would to an NTRIP mountpoint, then there is no functional difference. But you could imagine a couple efficiencies in leveraging MQTT functionality:

Both of these save bandwidth for network constrained devices, or processes which are pulling large subsets of data from a caster.

There are also some potential application specific benefits around things like shared subscriptions in MQTT.

The other main benefit (which is probably more relevant for those people operating the broker) is the reusability of MQTT broker infrastructure - particularly as more and more applications use MQTT (see IoT). There is much more variety in available MQTT broker implementations than NTRIP casters (particularly scalable open source brokers - see https://mqtt.org/software/).

I don't think MQTT replaces NTRIP, I just think it suits some GNSS data use-cases really well.

DavidKelleySCSC commented 4 years ago

Speaking as a provider of professional NTRIP Casters, the real appeal of MQTT is mostly the use case you have mentioned where an individual subscriber device can select what RTCM message types they want (and by omission what they do not want). Caster operators today have to run multiple different streams to address each variation. This most often occurs with different frames of reference, and when the deployment has a mixture of devices that want/need legacy and/or MSM message from the same base.

Also as a good example; the popular stream RTCM3EPH (which contains current orbital data for every satellite in every GNSS system worldwide) is good example of bandwidth bloat that would greatly benefit from the MQTT ability to only send when message content has changed.

And in the spirit of more discussion, a couple of passing observations. First, NTRIP carries much more than RTCM Rev3 content these days. Second, It is not a new idea that the Caster of tomorrow sends mixed content out but also collects mixed content in from the rover devices to store/forward to other places as required. There are a great many "smart Tractor, smart Car, smart observations platform" schemes developing out there that play on this. And this is where MQTT would really make a big difference. IMO.

Have we reached 'three' now?

fisart commented 4 years ago

Excellent points indeed. Very much in favour

On Oct 7, 2020, at 20:43, DC Kelley notifications@github.com wrote:

 Speaking as a provider of professional NTRIP Casters, the real appeal of MQTT is mostly the use case you have mentioned where an individual subscriber device can select what RTCM message types they want (and by omission what they do not want). Caster operators today have to run multiple different streams to address each variation. This most often occurs with different frames of reference, and when the deployment has a mixture of devices that want/need legacy and/or MSM message from the same base.

Also as a good example; the popular stream RTCM3EPH (which contains current orbital data for every satellite in every GNSS system worldwide) is good example of bandwidth bloat that would greatly benefit from the MQTT ability to only send when message content has changed.

And in the spirit of more discussion, a couple of passing observations. First, NTRIP carries much more than RTCM Rev3 content these days. Second, It is not a new idea that the Caster of tomorrow sends mixed content out but also collects mixed content in from the rover devices to store/forward to other places as required. There are a great many "smart Tractor, smart Car, smart observations platform" schemes developing out there that play on this. And this is where MQTT would really make a big difference. IMO.

Have we reached 'three' now?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.