Receiver beacon format is undocumented

[kerel-fs]

The format of OGN-specific beacons is documented in the wiki. Unfortunately the preliminary ogn-client version 0.2.6 sends modified receiver beacons, breaking existing parsers (according to glidernet/ogn-live#48).

Possible fix

• [ ] Update the documentation in the wiki to describe the changes in v0.2.6.
• [ ] Formalise the beacon format to prevent future breaks of different parsers.

[pyrog]

I started a documentation of the APRS format: issue 48# comment-262274816 😄

[svoop]

I'm maintaining the Ruby parser which logs a warning and ignores beacons with unknown versions.

"Update the documentation in the wiki" alone won't cut it. This would force parser maintainers to immediately implement changes since they might already be out in the wild. As for me, I won't be able to free time every time such a change pops up.

"Formalise the beacon format" certainly helps, but the real issue here are active receivers using unreleased versions. IMO, OGN servers should reject beacons with not yet released (master) versions, which would force receiver admins to only use released versions as of:

http://download.glidernet.org/arm/

Finally, a "heads-up" for parser maintainers say two weeks prior to a new release would be great. This would give the parser maintainers enough time to bring their code up to speed.

How about setting up two channels on Slack or Gitter, one for #general OGN discussion and another one for #announcements?

[svoop]

@pyrog I've already documented "OGN flavoured APRS" in order to write the Ruby client: https://github.com/svoop/ogn_client-ruby/wiki/OGN-flavoured-APRS

[pyrog]

I've already documented the "OGN flavoured APRS"

Yes, I saw it. See https://github.com/glidernet/ogn-live/issues/48#issuecomment-262254864 😉

Someone could update it with the Receiver status beacon format 🤔

This would force parser maintainers to immediately implement changes since they might already be out in the wild.

I'am not the author of any receiver software, but I think that the APRS status format didn't change. The separator between fields is still the space character 😉

[kerel-fs]

For compatibility-breaking updates I second @svoop's proposal to reject beacons by OGN (in the production network), since it generates complaints about broken components by users. For non-breaking changes, IMO, beacons from clients with an unreleased version can be accepted by the OGN servers.

@svoop: For (development-related) discussion I propose #glidernet on freenode. It's low-traffic and already in use. For announcements I would prefer an announcement or dev-only maling-list, since [OGN] is rather high-traffic. @snip Would you like to maintain such a second ML via google?

edit: mixed the mentions, sry.

[svoop]

@pyrog

Someone could update it with the Receiver status beacon format

I'd be happy to, however, I've got a question. Since here's not the right place to discuss this, could you ping me on #glidernet? Thanks!

[svoop]

@kerel-fs Couldn't we use semantic versioning à la major.minor.patch for this? As per convention, major breaks while minor and patch don't. But some projects also define this differently e.g. major and minor break, patch does not.

[snip]

@kerel-fs: i'm not sure creating a new ml is required.

I haven't double checked but the new format should only add new information to a new APRS message. Existing one is not changed. This is a parallel run before removing information from standard APRS receiver beacon.

For APRS parsing, i think a good practice should be to match known fields and ignore others. (This is what i do in my code).

[svoop]

For APRS parsing, i think a good practice should be to match known fields and ignore others. (This is what i do in my code).

I'm parsing with regexes for speed. Where possible, unknown fields are ignored, but there are limits. Not all possible extensions can be handled gracefully without a really expensive crystal ball.

[svoop]

i'm not sure creating a new ml is required.

@snip What would you suggest for announcements e.g. of upcoming releases to give parser maintainers a chance to adapt their code?

[pyrog]

@svoop 💡I suggest to use GitHub watch on a special project or on special a issue.

[svoop]

Nope, @pyrog, subscribing on GitHub is not a "heads-up". The moment the version string is counted up on ogn-rt/Master, receivers can use the code in production. And I can't just drop everything to adapt the parser whenever this happens.

[pyrog]

So the problem is not how to announce a new version?

The issue is that all developers should/must send a announce before releasing a new code (with significants changes in the protocol).

[svoop]

We're not talking about the same thing: I don't know how the maintainers of ogn-rtlsdr and ogn-decode organize their work and I'd never dare to tell them how to. I would, however, appreciate a timely announcement of upcoming releases on a well defined channel – for the parser maintainers (e.g. ogn_client-ruby) and other consumers.

[pyrog]

I think this is the same thing. A kind of netiquette 😉

[svoop]

@pyrog @snip

I've started upping the Ruby client. To get the migration path right, could you please comment on the following observations taken from live beacons?

A full receiver beacon v0.2.5 looks as follows:

LKHS>APRS,TCPIP*,qAC,GLIDERN2:/211635h4902.45NI01429.51E&000/000/A=001689 v0.2.5.ARM CPU:0.2 RAM:777.7/972.2MB NTP:3.1ms/-3.8ppm 4.902V 0.583A +33.6C 14/16Acfts[1h] RF:+62-0.8ppm/+33.66dB/+19.4dB@10km[112619]/+25.0dB@10km[8/15]

And here a typical beacon v0.2.6:

LFLE>APRS,TCPIP*,qAC,GLIDERN3:>114914h v0.2.6.x86 CPU:0.6 RAM:237.7/517.5MB NTP:18.7ms/-28.4ppm +64.0C 0/0Acfts[1h] RF:+49+1.8ppm/+4.80dB/+9.9dB@10km[49]/+9.9dB@10km[1/1]

❓ Comparing the two, is it correct to assume the new v0.2.6 beacons to be receiver status beacons which differ from v0.2.5 by no longer containing the location?

However, some less frequent beacons labeled with v0.2.6 have the "old" structure:

EPZR>APRS,TCPIP*,qAC,GLIDERN3:/112509h4946.47NI01913.38E&/A=001446 v0.2.6.x64 CPU:0.0 RAM:476.1/913.6MB NTP:0.9ms/+7.5ppm +34.0C RF:+94+3.7ppm/-0.85dB

❓ Why are there "old" structure beacons with v0.2.6?

The newly introduced beacons look as follows:

EPLU>APRS,TCPIP*,qAC,GLIDERN2:/114109h5125.65NI01611.65E&/A=000525 Antenna: coax-collinear, on the airfield tower

❓ Is it correct to assume these newly introduced beacons to be receiver description beacons which as of now carry the location and the free form antenna description?

❓ Roughly how often are receiver status and receiver description beacons being sent by an individual receiver?

Thanks for your clarifications!

[pyrog]

Comparing the two, is it correct to assume the new v0.2.6 beacons to be receiver status beacons which differ from v0.2.5 by no longer containing the location?

1. The data type of the first packet is a Position with timestamp (DTI /). The "status" follow the position as a "comment" (See below)
2. The second is a Status (DTI >)

APRS packets are text encoding of AX.25 UI frames. See https://github.com/glidernet/ogn-live/issues/48#issuecomment-262274816

Two forms:

• SOURCE>DESTIN:payload
• SOURCE>DESTIN,VIA,VIA:payload

Each APRS "payload" contain the following informations :

• APRS Data Type Identifier (DTI
  See "5 APRS data in the AX.25 information field" in APRS101.PDF
• APRS Data
• APRS Data Extension (optional)
• Comment (optional)

For more details, read "Chapter 17: Network Tunneling and Third-Party Digipeating" in APRS101.PDF

[pyrog]

@pjalocha Could you, please, give some answers to @svoop ?

For the content of each status fields, I don't know if it is normalized in APRS ?

But I guess that the order is not significant, and some are optionals. i.e. CPU temperature that could not be measured on some boards.

[svoop]

@pyrog I'm not asking about the APRS specification (which is well documented) but how it is implemented by ogn_rtlsdr.

Most notably: Sniffing the beacons, it looks as if as of v0.2.6 the "one receiver beacon contains all payload" approach has been given up in favor of two different receiver beacons, one with status payload and one with description payload. Hence my question above primarily addressed at those working on the actual ogn_rtlsdr code.

[snip]

0.2.6 is still in developpement. So subject to change at any time.

Why are there "old" structure beacons with v0.2.6?

To have APRS clients consuming this data still compatible when introducing also a new format. Data is duplicated in 2 different formats.

[svoop]

@snip

0.2.6 is still in developpement. So subject to change at any time.

Okay, so I just ignore the beacons which fail to parse at this point. Fills the logs, but should work otherwise. It would be very cool if we found a solution for release announcements though so we could clarify questions like the ones above and fix the clients prior to the release of v0.2.6. It would give me personally the feeling that my work is appreciated as much as I do appreciate yours!

I don't get one thing: The current ogn-rtlscr code as being worked on is not in this repo, not even open source in parts. How come v0.2.6 is out in the wild already?

[snip]

0.2.6 is not out but only running on some receivers @pjalocha (the main magic developer) has access to.

[svoop]

Alrighty, so "in development" version receivers are somewhat part of development.

However, undocumented beacons from unreleased versions are not cool and will cause trouble on the consuming end. What might seem non-breaking could still break things, there's no way to tell.

@snip @pjalocha To prevent this, how about adding a development flag to all beacons (aircraft, receivers etc)? When the development flag is set, all "normal" consumers must silently ignore the beacon while you can still consume it for development.

My apologies for being a nag :smile: I really care for OGN and highly acknowledge your work. For the overall project, however, it would be a win to clearly separate development from production IMHO.

[pyrog]

To prevent this, how about adding a development flag to all beacons ?

👍

v0.2.6.x64.dev 😉

[snip]

But we don't want to lose data coming from receivers running a development version.

Yes ideally we should separate clearly dev & prod ... but we are far to be a (big) company, so maybe we can live with a light process?

So now going back to my day work in big company with lot of process ...

[pjalocha]

Yes, I am testing the new code on some receivers, I have access too. I do my best not to loose data although some short periods of time can be lost. For the 0.2.6 I actually gain some data as i am now able to correct more errors in the packets.

For the beacon and status, I have first copied the measured data about a receiver to the "status" message, as I think this is the right place for them. Ideally a client code should read both beacon and status and get the data wherever they are. I rather don't change the data but add new measured values, which I believe are useful. So please keep reading all the know data and tolerate (don't crash the clients) when new data appears.

[svoop]

So please keep reading all the know data and tolerate (don't crash the clients) when new data appears.

Okidoke, I get it.

So now the Ruby parser raises an error when unknown messages appear, but the README encourages parser errors to be rescued and buffered/logged – for debugging and possibly later replay. Errors from your development receivers fill the logs of my test suite sniffer and renders it useless. However, I'll mark offending origins to prevent duplicate logging. This should make the logs great again :wink:

I'll pick up the ball once you give word of the protocol enhancements being stable enough.

So now going back to my day work in big company with lot of process ...

Yep, same here!