APRS-IS duplicate packet filtering broken by sondegate inserting unique data

[hessu]

The APRS-IS servers are capable of automatically filtering duplicate copies of the same packet, when the same packet is received by multiple receivers / gateways (igates).

This only works if the packet is forwarded intact, and the receiver does not insert unique data in the packet.

This only works correctly within a 30-second time window. This means that packets must not be delayed - they must be forwarded immediately, not delayed for up to 30 seconds.

The sondegate software inserts data in the packet (timestamp, software version), data which is not sent by the sonde, but rather data that is specific to the gateway. This breaks duplicate packet filtering, and when multiple sondegates hear the same packet, it will be forwarded many times over the APRS-IS. This adds congestion on both APRS-IS and at services getting data from it.

Sondegate needs to be fixed so that when a single packet sent by a sonde is sent to APRS-IS, the following fields are exactly the same:

• The source callsign (unique to the sonde)
• The destination callsign (same for all sondes / sonde gateways - not specific to gateway software)
• The packet contents, beginning from the ':' all the way to the end of the line

[jtmnt]

For starters, maybe remove auto_rx version number from description https://github.com/jtmnt/radiosonde_auto_rx/commit/d58fec1f78981e0b71f4dcd15dfffea0aaa5ac36

[darksidelemm]

This will work for auto_rx stations, but there are many other stations uploading to APRS, and they don't all do the same thing. In fact, many user go customize what is uploaded..

Note that the timestamp is from the radiosonde telemetry.

[hessu]

It'd be great if there would be one gateway which works right, which can then be used as a reference implementation. I can then filter out the badly working / customized variants here, and point their authors to do the Right Thing.

If the timestamp within the packet is truely coming from the radiosonde telemetry, and not derived in any way from the local clock of the gateway, that's great! If all gateways forward it the same way, the packet will look the same and duplicates & delayed packets can be suppressed.

One thing that needs to be fixed is the APRS destination callsign. Now it seems to be allocated per gateway software, which will result in different variants of the same original packet. There should be just one dst callsign for all sondes, and a per-gateway-software tocall which can be used for the beacon indicating the location of the gateway itself, to convey its software variant.

[TheSkorm]

hmmm. Would would the recommend dst call be?

[TheSkorm]

Thinking about this more, due to how many sondes work and differences in decoding software, or even differences in what frames have been successfully decoded, some of the sensor values are going to be slightly different.

SondeMonitor, radiosonde_auto_rx and rdz_ttgo all report different values :|

https://github.com/dl9rdz/rdz_ttgo_sonde/issues/112

[hessu]

For the destination callsign, we'll have to assign a common dstcall at https://github.com/aprsorg/aprs-deviceid and have everyone use it.

I'd guess everyone would want to have the decoding work correctly in all gateways, with the same sensor values coming out. That'll just need to be fixed. I can filter all gateways at aprs.fi and then start opening them up once they are consistent and fixed.

[TheSkorm]

Let me know the tocall and I'll get a PR raised in this repo :)

[TheSkorm]

From Mark:

"The issue here is that the values from the telemetry are reverse engineered, and there are still parts of the telemetry that are not completely understood (e.g. the use of all the correction coefficients used to calculate sensor values). There is no 'common' library to produce these values from the telemetry, nor is there even an open forum to discuss these calculations. Expecting every radiosonde decoding software to end up converging on a common format to feed into APRS-IS is wishful thinking, and based on observations over the years since I added APRS support to auto_rx, will never happen. There will always be stations running old versions, there will always be stations who want to add their own stuff into the comment field ('look, I received this telemetry!'), with the end result that effective de-duping will be impossible. Personally, I think that radiosonde telemetry should not end up in the wider APRS-IS network at all. It should stay confined to specialized databases (e.g. Sondehub and http://radiosondy.info) where the data can be handled in a better manner. If you want to show radisondes on aprs-is, then you could take a feed from these systems - e.g. Sondehub has a websockets interface which could be used.

[hessu]

Here's my position on this: https://blog.aprs.fi/2022/03/radiosonde-igates-are-quite-mess-and.html

I'm perfectly happy if it goes away from APRS-IS, that is a valid solution and matches the initial intention of the APRS-IS network.

One option is to take it to aprs.fi from such a system, without injecting it to the APRS-IS network. aprs.fi software itself is not based on the APRS protocol, APRS is just one data feed that goes in.

[hessu]

I should also add that the APRS iGates have somehow managed to keep data going through the gateways mostly intact, without inserting custom comments. It should be possible to encourage radiosonde receivers to do the same, for example by providing them other ways to express themselves, and by simply filtering them out if they do it anyway.

[hessu]

Also, if there is no common forum for the radiosonde folks to discuss this: how about setting one up, and trying to work this out?

It may be wishful thinking, but I think it's worth trying.

[TheSkorm]

Once we have a tocall we can update that in autorx.

Other radiosonde receivers can try to match our format, and the coefficients that we use but I think its unlikely it'll ever converge - and it's not like autorx will be "correct". We'll happily review issues or PRs for correcting the decoders if provided. We've tried our best to get rdz_ttgo project to match our data and even that is still slightly out. Not all decoders support or can support decoding all the telemetry as well - with many just reporting temperature or GPS. Even things like generating callsigns for iMets and DFMs is debated on the approach across systems and apps.

Autorx already has a min time of 30 seconds and the recent PR to testing brings that up to 60 seconds - so I don't think there is anything else we can do here.

[ZL1LAC]

Ok, I'm happy to disable APRS, just need to know how.

[darksidelemm]

You don't need to do anything apart from update.

If you're feeding data into radiosondy.info (as is the default), then your APRS packets won't go out to the wider APRS-IS network anyway, as radiosondy.info has disabled that.

[ZL1LAC]

Thanks, I figured it out and updated it to 1.5.10.

[darksidelemm]

Given we're now no longer allowing uploads to the rest of the APRS-IS network, I'm closing this issue.

I honestly don't see the APRS comment format issues being resolved successfully. If using APRS-IS requires that all packet formats be exactly the same, down the last character, then APRS-IS is the wrong solution for this data.