M10 - APRS Callsign should match with that produced by dxlAPRS

[oe3jtb]

Hi is it being considered to decode M10 sondes with new names like its done in radiosondy image and OE5DXL chain? Here they are using sondemon > 1.33 or RS image, which is based on sondemod https://radiosondy.info/sonde_archive.php?sondenumber=ME962264A

and here Erich is using Auto_RX https://radiosondy.info/sonde_archive.php?sondenumber=ME994022

So two names are on air :-), which are one sonde

The new names were nesseccary due to dupe names between Vienna and Lubljana, because raw serial number is too long for APRS and the shortend version caused dupes. The new nomenklatura has only one disadvantage, APRS prediction OK1COM-10 could not be made in realtime but thats a more litte issue than the dupes. Tnx kind reggards Alex

[darksidelemm]

Can you produce documentation or a reference to where the serial number is calculated please?

[oe3jtb]

Hi tnx for superfast reaction, yes I believe its done here https://github.com/oe5hpm/dxlAPRS/blob/master/src/sondemod.c#L3446

[darksidelemm]

... I'm going to need to get someone else to decipher what's going on there. A decent description of how the APRS object ID is calculated from the bytes in the M10 frame is going to needed to make this work. @rs1729 might know more.

If you're looking for live landing predictions, then I think https://tracker.sondehub.org might be better suited, and it doesn't have the restrictions on callsign length that APRS does...

[oe3jtb]

yeah nice, yes sondehub is also very precise. The only one reason, why I dont use auto_rx all day is, that 1 stick is dedicated to one probe. That is very fine if there are not so much at the same time as here in EU :-) I believe you can also contact Chris, the author, to exchange, because on his software some other are based on . And if its stored in a database, as Michal SQ6KYX, (radiosondy operator) there are dupes and confusion :-) Same with DFM17 and right recognition of DFM09, was done in sondemod 1.35 kind regards Alex

[darksidelemm]

@rs1729 might have more of an idea of what to do with the DFM09/DFM17 identification stuff. The next release of auto_rx will be updating the dfm demodulators to his latest version, so maybe its fixed there.

As for multiple-sondes-per-rx, well auto_rx is open source, and I accept (well tested) pull requests...

[rs1729]

There was a discussion about the M10 serial number here https://github.com/rs1729/RS/issues/7 and https://github.com/projecthorus/radiosonde_auto_rx/issues/151#issuecomment-502967828

Don't know how sondemod is hashing/cutting the M10 serial number. Parts of their code look familiar, only with magic numbers... strange. There is no "issues" anymore, not very transparent afterall. There have been requests base on @rs1729 code, then some code appeared, but you cannot see the issue history anymore.

DFM we discussed here https://github.com/rs1729/RS/issues/16#issuecomment-513150710 Apparently on radiosondy there have been different formats used by different programs. For DFM-09 the serial number is put together from 2 16bit packets. To this date I have seen only serial numbers with 8 base-10 digits, i.e. 7 hex-digits are used. I don't know much about aprs, but if there are only 9 characters possible, DF12345678 would be possible, maybe the type should not be so important, if it is DFM-06, DFM-09 oder something else, so these would not have to be coded in the aprs-id. But aprs-format is not for me to decide. (DFM-17 pictures show extended serial numbers, though in the data the shorter familiar format was found. Maybe it will change some day.) DFM-06 had its serial number in 1 packet, and although the packets have hex nibbles, the 6 digits of the serial number were the digits on the envelop. I think some did mix that up. Thus for DFM-06 one could also have DF6-123456, and it would distinguish from DFM-09-like sonde types. The problem is when you have different version of aprs-id's over time and let all the different programs upload to the same data base. If you would have a version-identifier, maybe you could convert on the server, but for M10 it seems to be difficult since there first hash came a bit short.

(the latest dfm-fix was only to prevent reading the wrong channel in the first cycle (in particular for DFMxC), it happened to read the id in ch 5 and produced the wrong (too long) id. but this is not aprs-related.)

For decoding multiple radiosondes in on iq-baseband, in principle rs_multi can do it and produce json-output for several signals. Though still not sure which way to go, how to communicate or exchange scanning results. Right now it is static, for automated decoding it would be good add/launch new decoder-threads or close old ones when the signal is gone. @darkside suggested a tcp-communication, maybe reading a named pipe would be even simple, will see.

[darksidelemm]

Hi @rs1729 - My comment about DFM was more about identification of the sonde type, not the serial number generation for APRS (I think we have that part sorted now). It sounds like the dxlAPRS author has made a decision as to what ID number equals DFM17, an what ID number equals DFM09. As to how accurate that decision is I have no idea. Any ideas what the 0xC DFMs are? We seem to be seeing more of them in the US now.

For the multi-sonde decoder stuff, I think TCP would be better - that reduces cross-platform issues. It sounds like @miweber67 is onto something with the spyserver approach - with an Airspy that would allow monitoring of the whole radiosonde band (with CPU limitations of course), and is a nicer 'generic' solution that could be used in other places.

[rs1729]

Ah, ok, then I didn't read carefully enough, the topic came up several times...

To my knowledge, DFMxA (i.e. serial number in packet/channel "0xA") is DFM-09 (polarization opposite to DFM-06). DFMx7 pilotsonde (polarization like DFM-06). DFM-06 has an older and different serial number format. There have been early DFM-09 with DFM-06 chipset, so they transmitted in the old (inverted) DFM-06 format and DFM-06-like serial numbers.

DFMxC, DFMxD: These are new. The one from Nuernberg were suspected to be DFM-17 tests, the DFMxD sample (that I know) had opposite polarization, like the old DFM-06.

In Canada they use DFM-09 with pressure sensor (you don't see this in Europe, only maybe from Graw testing in Nuernberg), and the serial number is in "0xC", so DFMxC. A picture of the latest recovery: https://github.com/projecthorus/radiosonde_auto_rx/issues/152#issuecomment-532838618

I have got a sample from (the source in) Canada what was probably a DFM-17 test, the spectrum looked a little different, frequency more stable, and it had DFMxC as ID, however the polarization was inverted. So maybe the new DFM-17 will come with inverted polarization again, now that DFM-06 is outdated. (Though detected polarization also depends on receiver settings, not so good for identification on an unknown system...) But this is speculation. The serial number comes in the last packet/channel, thus it depends on how many different data packets there are, depending on radiosonde/sensor configuration. So it could be that different sonde-type/config combinations have the serial number in the same packet/channel. The example from Egbert earlier this year with inverted polarization: <-> [136] 2019-02-24 22:38:50.0 (0,0,0) lat: 45.20310 (0) lon: -77.68670 (0) alt: 1376.9 (0) vH: 20.46 D: 47.0 vV: -3.63 <-> [137] 2019-02-24 22:38:51.0 (0,0,0) lat: 45.20321 (0) lon: -77.68651 (0) alt: 1374.1 (0) vH: 19.03 D: 51.0 vV: -2.96 (ID-C:18085772) <-> [138] 2019-02-24 22:38:52.0 (0,0,0) lat: 45.20331 (0) lon: -77.68632 (0) alt: 1369.9 (0) vH: 18.57 D: 58.9 vV: -3.14

EDIT: I didn't check all the new packets, so maybe there is a packet saying "I'm a DFM-17", but there is not much constant data apart from the serial number packets. So maybe the polarization is another clue, but until the DFM-17 is flying more regularly or someone has more information, I would not speculate. I think the channel/packet number is some additional information to the serial number.

[rs1729]

@oe3jtb

is it being considered to decode M10 sondes with new names like its done in radiosondy image and OE5DXL chain?

The idea is to decode/reproduce the serial number as it is on the radiosonde box/envelope. Thus the decoder json-output will have these serial numbers, why should it be cut or mangled!? And if there is no good mangling-algorithm for it, you run into problems after a while and have to change the format again...

And sondehub supports longer serial numbers, only APRS has these limits. Thus the auto_rx-script has to generate the APRS-serial-number, but only for APRS I guess. I would not recommend doing it for sondehub-upload, better to have serial numbers that can be found on the actual radiosonde.

Hi tnx for superfast reaction, yes I believe its done here https://github.com/oe5hpm/dxlAPRS/blob/master/src/sondemod.c#L3446

This is the checksum-calculation (they changed the shift-operator), but kept some variable names like b, t, s ... their rs41-code has the same similarities, a reference would be fine.

[oe3jtb]

Ah ok, I hope I got it right, it would be better to decode the real serial# and change this , if possible, when sending to systems, which cant handle the original number. I belive thats due to the fact, that some systems where stand alone at the beginning, some did the radiosonde decoding as an add on (so DXLAPRS did I belive) at first only for APRS , because the database system Radiosondy.info was not up and running at that time. And than there where done forks, as SPDXL, main for closed user group, but also with the possibility sending data to radiosondy, same as autorx did . And as a new system for searching near the landing place, Hansi, DL9RDZ did a very nice work with LoRa TTGO modules, and as I can see, sending packets received at "the front" are released in the next versions. I hope I had understood this right. I will try to communicate with Chris, OE5DXL, because in my opinion there are 2 facts. First Auto_RX and DXL chain today are the leading systems for monitoring and publishing for many users, and second a consistent name resolution would be preferable over all thosse platforms, to get also a consitent database, as Radiosondy.info. Much effoert for all protagonists, who are playing a significant role in providing us users a nice and easy surface kind regards Alex

[rs1729]

I cannot speak for auto_rx/sondehub, only for the decoder output. But for me it makes no sense to cripple the serial number such that it fits aprs. This can be done when reporting to aprs after decoding, and for that it would probably be good to find consensus. (But don't ask me, I cannot say much about aprs.) Though the information in the databases should not be reduced only because of aprs. For software that focuses on aprs, this might be ok, but this should not affect other databases. I think sondehub/auto_rx have the advantage that they can reject older client-versions on the server, so if there are significant changes, it does not get mixed up. On radiosondy you see different sources uploading data, some use outdated program versions and some programs seem to confuse the serial number and sonde type.

[oe3jtb]

ok I understand, for a future to avoid dupes because of a mathematical conversion on the one hand, and to avoid forks of the decrypting algorythm, which could be more difficult with each new sonde type , the recommendation is: First decoding is always the real serial number and that should be also the primary "key" of a database storage, second a conversion, if neccessary for APRS from a central point or a voluntary commitment to use decentralized the same code for conversion. Puh that woulld be I believe a big effort for all who participate. I also informed Chris and Michal from that thread, perhaps we will get a feedback I will hope so :-) tnx for allowing the discussion here and sri for my two cents....

[darksidelemm]

@rs1729 yep, the 'shortened' callsign is purely just to get over the APRS 9-character object limits. We'll always use the full callsign on sondehub.

I know that @Viproz had a go at implementing the dxlAPRS method of generating a 9-character serial number here: https://github.com/projecthorus/radiosonde_auto_rx/blob/master/m10/M10TrimbleParser.cpp#L349 It outputs this as part of the JSON blob as a 'dxlid' field, which makes my life a lot easier. I'm still currently using his M10 decoder for this reason, but I would like to switch to your m10mod decoder. Thing is, it looks like the dxlAPRS way of calculating the shortened call is based on a lot of mangling of the raw serial number bytes, so I'm not sure if it'll be possible to calculate this from the full numerical serial number you provide.

[rs1729]

Ok, their new aprs serial number here https://github.com/oe5hpm/dxlAPRS/blob/master/src/sondeudp.c#L3914 and it follows the (raw) bytes that make up the serial number, more like https://github.com/projecthorus/radiosonde_auto_rx/issues/151#issuecomment-502967828 but it follows the raw bytes. That's ok, the serial number on the envelope has a strange relation to the raw bytes anyway. (As commented, their is enough space for the sn-bits.)

dxl[0..1] = "ME" dxl[2..3] = ("%02X", b[95]) dxl[4] = ("%1X", b[93]) dxl[5..6] = ("%02X", b[97]) dxl[7..8] = ("%02X", b[96]) dxl[9] = 0

And the year/month is now in the first 2 nibbles. For you it would be easier to have the raw bytes, better than calculating backwards.

The serial number as I reconstruct it and fits my data https://github.com/rs1729/RS/blob/master/demod/mod/m10mod.c#L424 pos_SN=0x5D=93 sn[i] = b[93+i] sn[0], sn[2], sn[3], sn[4] are used. All the samples I know confirmed that.

I would suggest either the plain serial number as json-ID and the 4 relevant raw bytes as comment or aprs-ID, or the other way around, raw json-ID and plain/envelope-ID as comment. I think it is good to have the plain serial number as it can be found on the envelope, as well. @darksidelemm How would you name an additional json-field? 7 hex-nibbles for aprs (dxl[2..8])?

[darksidelemm]

'aprsid' would work well. Definitely keep the regular 'id' as the full serial number as we have it now, that works well.

[rs1729]

ok https://github.com/rs1729/RS/commit/354c58da2e24be84ad39721c39f4d02e24cd7144 including "ME". Actually it is the full serial number data, raw bytes, so this is ok to include. Though the plain serial number is not so obvious, so it is good to keep that as well.

[darksidelemm]

APRS callsign looks good!

The only thing it is missing now is a 'frame' field. I believe the M10 doesn't have a packet counter like the other sondes. @Viproz got around this by converting the date/time to a timestamp and using that as the frame ID - you can see that code here: https://github.com/projecthorus/radiosonde_auto_rx/blob/master/m10/M10TrimbleParser.cpp#L429

I'd prefer not to have to do this in the python code if at all possible...

[rs1729]

Sorry, I don't want external system timestamps. But there is a small counter just before the checksum, counting 0x81 .. 0xe4, i.e. 100 frames. (The double frames every 10 sec are counted but not shown in the printout, every 10 sec it jumps +2) This would be similar to DFM that also has only a 1 byte frame counter 0..255.

[darksidelemm]

Oh this isn't constructing the frame number from an external timestamp - it's converting the time reported by the M10 to a unix timestamp and reporting that as a frame ID which should be unique throughout the flight.

I'm not entirely sure how sondehub/habhub handles the frame ID. It clearly doesn't seem to mind the DFM ID resetting, so using a shorter ID is probably OK. It will break compatibility with older versions, but we've already done that by removing the -T- and -G- indications.

[rs1729]

Oh, ok, getXXX() are defined above, no parameter, global variables. I thought these were (c++)library functions getting the system time... Didn't expect that; it was not needed for DFM... you have the exact gps or utc time.

Don't want another tm-library just to convert back to unix time... I mean there is time_of_week and gps_week already in the data. Not considering gps week rollover, just do gps_week*sec_per_week+tow_sec. It is almost 32bit though (gpsweek (now > 2048) could be stripped). So this I can do and would rather do: "GPS-seconds since week0 or wekk1000 or whatever, ok? It is the same, only without detour (if this is the right word).

[darksidelemm]

Yep, sounds good to me! As long as it's unique (ideally entirely unique over a flight) that will work well.

[rs1729]

If 32bit numbers are ok, I put in the whole "gps_seconds since week 0", Januar 6th 1980, I believe. Will round the milliseconds, in case it is just below integer seconds. (And you don't need that small counter then?!)

Example: (W 2050) Tue 2019-04-23 06:15:14.000 lat: 46.46518 lon: 14.44184 alt: 12121.08 vH: 18.4 D: 16.6 vV: -11.4 SN: 810 2 13082 APRS: ME8A22C0A [0x85=133] [1240035314] # [OK] { "frame": 1240035314 ,"id": "M10-810-2-13082", "datetime": "2019-04-23T06:14:56.000Z", "lat": 46.46518, "lon": 14.44184, "alt": 12121.08000, "vel_h": 18.38551, "heading": 16.59179, "vel_v": -11.36000, "sats": 9, "aprsid": "ME8A22C0A" }

(W 2050) Tue 2019-04-23 06:15:15.000 lat: 46.46533 lon: 14.44190 alt: 12109.84 vH: 17.3 D: 16.2 vV: -11.2 SN: 810 2 13082 APRS: ME8A22C0A [0x86=134] [1240035315] # [OK] { "frame": 1240035315 ,"id": "M10-810-2-13082", "datetime": "2019-04-23T06:14:57.000Z", "lat": 46.46533, "lon": 14.44190, "alt": 12109.84000, "vel_h": 17.29801, "heading": 16.21383, "vel_v": -11.22000, "sats": 9, "aprsid": "ME8A22C0A" }

EDIT: 31bit is fine until week 3550, 32bit until week 7101.

[darksidelemm]

Yep that works just fine - that's pretty close to just calculating the unix timestamp. Also I think the full ID is only calculated if you are in -vvv mode, not just in --json mode.

[rs1729]

It sounds like the dxlAPRS author has made a decision as to what ID number equals DFM17, an what ID number equals DFM09. As to how accurate that decision is I have no idea.

Do you mean this https://github.com/oe5hpm/dxlAPRS/blob/d6cdfc110a28de06ca7af61d7c6d055c9985ac48/dxlAPRS_common/NEW.txt#L428

I have never recoverd a PS-15 or DFM-17, but the signals I received with DFMx7 were reported to me as pilotsonde PS-15, and the few channels would suggest a pilotsonde as well. And DFMxC or DFMxD were reported as DFM-17, though there are DFM-09 w/pressure sensor and DFMxC that have been recovered. Don't know if that statement above is accurate. Perhaps this "-L" option is to set the type if you are certain what it is. But then I don't see the advantage.

Also about the unreliable "pruefsumme", that is not what I tried to explain before. If you require all Hamming codewords error-free, since there are many codewords in one frame, it would be reliable as a CRC (the representation here is not cyclic, but that is not important here). But then you could not make use of the error correction property. On the other hand if you have many codewords with errors in one frame, it is not so certain that the corrected codewords are the originally transmitted codewords. I think Hamming (8,4) is not the best choice for the error probabilities encountered in radiosonde operations (though it is easy to implement). Reed-Solomon with 24 to 32 parity bytes together with CRC, that works much better for radiosonde data rates.

Afterall it is a DFM, the spectrum can tell a bit more (for who is really interested), and maybe the polarization. I don't know of regular DFM-17 operations yet, as long it is in testing, it is even possible that the firmware will change.

[darksidelemm]

I've switched to m10mod in 1.2.0-beta5 (in the testing branch), so this should be resolved when I release 1.2.0 (which will be after I get some feedback from testers).

[bazjo]

Back to the DFM related issues: indeed both PS-15 and DFM-17 have a polarization like the DFM-06 due to the implementation in the now used Si4063 radio ic. All the DFM-17s I have available are late prototypes, both have the pressure sensor populated and are 0xC serial identifier. That would indicate no change here between DFM-09 and -17. However, I noticed a change in the 0x9 bytes between DFM-09 and DFM-17 but can't fully quantify this yet. But that might be best discussed over at your repo @rs1729.

If anyone is more interested in Graws production process, take a look at https://sondehunt.de/language/en/archive/1189. The Graw protocol in general is (in my opinion) long overdue of an update as it is about as old as myself and ther is no need in sticking to it.

Regarding all the trouble with APRS, I proposed a new format to replace it in our usecase and am keen in getting your feedback on this. https://sondehunt.de/language/en/archive/1203

[darksidelemm]

Interesting idea on the interchange format - I think that could work, though there is still going to need to be translation between that format and whatever is required for use on mapping systems like APRS and Habitat.

Currently internally in auto_rx we pass around essentially all of the telemetry data received from the decoder as a dictionary, which then gets converted for upload to Habitat / APRS, or sent out into the network essentially as-is (converted to JSON) for use in chasemapper. The fields used are not too dis-similar from yours - named a little bit differently, but the same data is there (along with required and optional fields).

[oe3jtb]

@rs1729 and @bazjo would you like to contact me via email, I made a group for that, where developers are in to make suggestion, how its possible to make the name problem fit for the future. If you are intrested write to oe3jtb(at)gmail.com tnx