DFM17 decoded as DFM09P sometimes

[AstroTaka]

DFM17 decoded as DFM09P sometimes

When I receive DFM17, I saw DFM09P in Main screen sometimes. But the Sonde List display still DFM17. So I downloaded the log it and check it. Usual case, subtype is DFM17. But DFM09P exists sometimes.

I uploaded sonde.log, debug systen.log and IQ decode.bin. https://drive.google.com/drive/folders/1rt4RhzZZlHYzlIvQ5RHB5a8qOSfxPbzI?usp=share_link

Please check it.

[darksidelemm]

It's using this list of 'guesses', in how to interpret the DFM code IDs: https://github.com/projecthorus/radiosonde_auto_rx/blob/master/auto_rx/autorx/sonde_specific.py#L92

I'm wondering if that code might be changing during decoding?

[AstroTaka]

I searched "decode_dfm_subtype" from whole source code. It seems to be nobody call it but the similar code exists in the following source. https://github.com/projecthorus/radiosonde_auto_rx/blob/testing/demod/mod/dfm09mod.c#L880

[darksidelemm]

Sorry yep, you are right. I take the subtype guess from the decoder itself.

               elif self.sonde_type == "DFM":
                    # As of 2021-2, the decoder provides a guess of the DFM subtype, provided as
                    # a subtype field of "0xX:GUESS", e.g. "0xD:DFM17P"
                    if ":" in _telemetry["subtype"]:
                        _subtype = _telemetry["subtype"].split(":")[1]
                        _telemetry["dfmcode"] = _telemetry["subtype"].split(":")[0]
                        _telemetry["type"] = _subtype
                        _telemetry["subtype"] = _subtype
                    else:
                        _telemetry["type"] = "DFM"
                        _telemetry["subtype"] = "DFM"

[rs1729]

I observed this, too. First I thought it is because of different versions of the decoder. But sometimes the decoder makes a wrong guess and outputs wrong temperature for a short while. In principle these two subtypes have different polarity, but it can depend on the receiver setup, which one is which. Thus the decoder looks also in the raw PTU data to determine what channel is raw temperature data etc. Sometimes it can be wrong, in particular for weak signals if there are decoding errors. In the comments is an additional condition that could reduce false detection, but I don't have enough test data, so I don't know if this could produce false positives. If the subtype is wrong, the temperature will be wrong as well. But the majority of frames should be ok.

Another questions is which subtype remains on sondehub as the "final" subtype.

@AstroTaka sensortyp == 'P'means there are additional channels for pressure data such that the temperature data is not at the "normal" place.

It's always the same for DFM, if you would have an additional CRC over the whole frame, you would know how reliable the decoding is, or you have to use the Hamming code as strict CRC, but then you could not use the error correction. Using the Hamming code for error correction is not reliable for radiosonde transmissions. (I guess what I'm trying to say is, the DFM has a data integrity issue.)

remark: If you launch and track a DFM-09P or DFM-17, you wouldn't have the problem, obviously you would know what to expect.

[AstroTaka]

@rs1729 I do not launch a sonde :-) I only receive a sonde which somebody launched.

If you hope to get another kind of log, please let me know. My 1st tnterest is that it is BUG or unsolvable some kind of limitation. If you can solve it, I hope to help you as debugger. Or I understand that it is a limitation.

[rs1729]

@AstroTaka ;) Most of us don't. But the manufacture and the weather stations don't necessarily have the problems we have.

Sure there could be still bugs (most likely), but the main problem is somewhere else, the old backwards compatible frame structure is problematic together with insufficient error detection code. Compare to RS41, much easier to get reliable results. The DFM decoder makes many compromises. And not all specifications are known to me, so probably it is possible to get better results, though I don't know if it is worth the effort. Often it depends on the situation, which solution works better. And if you make more restrictions (higher SNR, receiver type), you can also get better results. But e.g. if there are two DFM signals at the same frequency, the packets can be mixed up and it might be difficult sometimes to separate them, see the remarks above. Maybe if you count the channels, though the DFMs don't have all the same channel order. I think the older DFM-06 repeated channel 0 every other subframe, not all do that. The decoder tries to keep track if there are gaps though (option --dist).

You can try modifying the following and test if it gives better results: https://github.com/projecthorus/radiosonde_auto_rx/blob/e063903b43300c8dfc7c3ada1f2cad94e84516c4/demod/mod/dfm09mod.c#L718 try

        if (gpx->ptu_out >= 0xD || (gpx->ptu_out >= 0xC && gpx->meas24[0] > 1e5)) {
            gpx->sensortyp0xC = 'P';
        }

or

        if (gpx->ptu_out >= 0xD || (gpx->ptu_out >= 0xC && gpx->meas24[6] < 220e3 && gpx->meas24[0] > 1e5)) {
            gpx->sensortyp0xC = 'P';
        }

or maybe adjusting the thresholds for these channels. But in the end, if a false decoding gets through, the decoded values might suggest, it is a different type.

Perhaps I could add that if the PTU type changes (similar to SN differences), hold back the PTU/SN/subtype output and confirm the old or new state next round, reset everything if necessary. Would require last state and maybe last confirmed state. If it is a different DFM on the same frequency (or someone changed the receiver frequency), the conf-data would have to be reset anyway.

The decoder outputs data if the last GPS packet 8 is received. The PTU packets (channels) run in parallel with a different cycle length (depending on DFM type), that's not making the situation better...

When I run the decoder, it does not matter to me if there are suspicious frames from time to time, if I see the error count for the packets and previous frame. For JSON output it is different, a reliable frame would be desirable, so different receivers send the same data. The Meisei radiosondes are actually not so different to DFM, not only with regard to the serial number.

EDIT: Looking at your recordings, I get 3/2366 identified as "DFM09P" (in you log 5 more), doesn't sound so bad. With invalid temperature, PTU and subtype could be reset, will check the debugger. Although I observed reasonable but wrong temperatures in the past in such cases, so comparing to previous frames might be good idea. I wonder what would happen if the sensor is broken.

EDIT2: I believe it should be rather gpx->meas24[0] > 1e5, have to see if I have more DFM-P data. (One sample shows 4e5 at 10hPa.) Not sure if it could overlap with temperature for non-P DFMs. They don't have all the same scaling. meas24[0]=1e5 for non-P would give -96C and -108C for 4e5, that's to close to plausible values... Anyways, have to check other plausibility checks.

[rs1729]

https://github.com/rs1729/RS/commit/cf91b3966337c3f3c7f26e0b2f0f7af16dec9934 Added a restriction to read only error-free sensor data. Data acquisition might take a little longer, and sometimes the temperature might be calculated based on the previous frame, but that's better than potentially having completely wrong data.

Will look into more plausibility checks later, but for now this should mitigate false temperature and subtype output. At least it cleaned up the output when testing the data above.

[darksidelemm]

This change now in testing, along with the mts01 battery voltage addition

[AstroTaka]

Thank you very much! I started to use the latest beta21 which dfm09mod.c modified.

According to NOTAM RJTD 2734/22, I can receive DFM17 until Dec. 5th for every 3 hours. https://www.skyinfo.jp/

[AstroTaka]

This is a current result of beta21. It seems to be OK. I continue to check it.

$ ls  *DFM*.log | xargs grep -c DFM09
----------------------------------------------------
beta20
----------------------------------------------------
20221122-180426_DFM-21063336_DFM17_404405_sonde.log:0
20221122-211403_DFM-21063320_DFM17_404403_sonde.log:1
20221123-000839_DFM-21062767_DFM17_404404_sonde.log:7
20221123-002146_DFM-21062767_DFM17_404404_sonde.log:7
20221123-030754_DFM-21062799_DFM17_404404_sonde.log:2
20221123-061007_DFM-21063293_DFM17_404404_sonde.log:9
20221123-090942_DFM-21062581_DFM17_404404_sonde.log:2
20221123-120932_DFM-21062667_DFM17_404404_sonde.log:3
20221123-151051_DFM-21063310_DFM17_404403_sonde.log:1
20221123-180927_DFM-21062654_DFM17_404404_sonde.log:3
20221123-210358_DFM-21062755_DFM17_404405_sonde.log:5
20221124-000932_DFM-21062768_DFM17_404405_sonde.log:1
20221124-030234_DFM-21063260_DFM17_404404_sonde.log:15
20221124-060335_DFM-21063297_DFM17_404405_sonde.log:2
20221124-090628_DFM-21062781_DFM17_404404_sonde.log:0
20221124-120308_DFM-21063159_DFM17_404404_sonde.log:13
20221124-150401_DFM-21062814_DFM17_404404_sonde.log:12
20221124-180302_DFM-21065745_DFM17_404404_sonde.log:1
20221124-210308_DFM-21062811_DFM17_404403_sonde.log:5
20221125-000304_DFM-21062739_DFM17_404405_sonde.log:0
20221125-030826_DFM-21062551_DFM17_404405_sonde.log:5
20221125-060538_DFM-21065757_DFM17_404402_sonde.log:9
20221125-090357_DFM-21065738_DFM17_404405_sonde.log:3
20221125-123110_DFM-21070040_DFM17_404406_sonde.log:1
20221125-150846_DFM-21062759_DFM17_404404_sonde.log:0
20221125-180241_DFM-21070598_DFM17_404404_sonde.log:3
20221125-210744_DFM-21062740_DFM17_404404_sonde.log:0
20221126-000641_DFM-21063148_DFM17_404404_sonde.log:3
----------------------------------------------------
beta21
----------------------------------------------------
20221126-031035_DFM-21069994_DFM17_404404_sonde.log:0
20221126-060755_DFM-21070919_DFM17_404404_sonde.log:0
20221126-090458_DFM-21069970_DFM17_404404_sonde.log:0
20221126-120449_DFM-21070036_DFM17_404402_sonde.log:0
20221126-150254_DFM-21070144_DFM17_404404_sonde.log:0

[AstroTaka]

This is a additional result. This is big better than beta20.

beta21

20221126-180259_DFM-21070633_DFM17_404403_sonde.log:0 20221126-210310_DFM-21070545_DFM17_404404_sonde.log:0 20221127-000253_DFM-21069961_DFM17_404405_sonde.log:0 20221127-030238_DFM-21063815_DFM17_404403_sonde.log:0 20221127-060238_DFM-21069925_DFM17_404404_sonde.log:0 20221127-090752_DFM-21070620_DFM17_404404_sonde.log:0 20221127-120340_DFM-21070896_DFM17_404404_sonde.log:0 20221127-150650_DFM-21070122_DFM17_404404_sonde.log:0 20221127-180329_DFM-21070170_DFM17_404404_sonde.log:0 20221127-211800_DFM-21070604_DFM17_404405_sonde.log:0 20221128-000422_DFM-21069977_DFM17_404404_sonde.log:0 20221128-030358_DFM-21070565_DFM17_404404_sonde.log:0 20221128-060426_DFM-21070617_DFM17_404404_sonde.log:0 20221128-090244_DFM-21070110_DFM17_404404_sonde.log:0 20221128-120813_DFM-21070117_DFM17_404405_sonde.log:0 20221128-150615_DFM-21063155_DFM17_404405_sonde.log:0 20221128-180906_DFM-21065555_DFM17_404404_sonde.log:0 20221128-210706_DFM-21070907_DFM17_404405_sonde.log:0 20221129-000653_DFM-21070566_DFM17_404403_sonde.log:0 20221129-030749_DFM-21062791_DFM17_404403_sonde.log:0 20221129-060613_DFM-21063191_DFM17_404403_sonde.log:0 20221129-090740_DFM-21063241_DFM17_404405_sonde.log:3 20221129-120738_DFM-21062709_DFM17_404404_sonde.log:4 20221129-150513_DFM-21063183_DFM17_404404_sonde.log:0

I attached 20221129-090740_DFM-21063241_DFM17_404405_sonde.log and 20221129-120738_DFM-21062709_DFM17_404404_sonde.log for your reference.

https://drive.google.com/file/d/1huF4wYFKEItRTPV_Vqfvmzh6al7iIbmL/view?usp=share_link

I think that we can close this issue...

[rs1729]

As long as it shows temperature -273C (invalid) for misidentified "DFM09P" its ok. And it does not seem to happen that often.

[darksidelemm]

On a related note, the ID that we're getting (e.g. 21062709) is a truncated form of what's on the DFM17 label. (e.g. it's 22xxx-062709). @rs1729 do you know if any more of the ID is transmitted? It would be nice to get the complete ID out of it.

[rs1729]

The SN is transmitted the same way for DFM-09 and DFM-17, in two parts in the last "channel" (high/low 16bit word). I don't remember any other constant value that would add to the middle part. I think on the box it is just a more detailed version of the SN. 8 character SN seems to be enough for transmission.

[rs1729]

Added a check, if temperature T is invalid, reset DFM09P/DFM17 subtype, i.e. if "subtype": "0xC:DFM17"or "subtype": "0xC:DFM09P", it will be set to "subtype": "0xC:DFMxX" until the temperature is valid again.

Didn't find any test data where the wrong subtype was set with the last version, it should not happened often anyway. Don't know @AstroTaka , if you also have recordings for two

20221129-090740_DFM-21063241_DFM17_404405_sonde.log:3
20221129-120738_DFM-21062709_DFM17_404404_sonde.log:4

EDIT: There is a chance that the detected subtype is corrected and the temperature is invalid, but I think it is better to reset. E.g.

<-> [232] 2022-08-16 13:51:47.0 (0,0,0)   lat: 52.24825 (0)   lon: 14.42914 (0)   alt: 12535.7 (0)   vH: 12.69  D: 109.2  vV:  4.10   T=-50.2C  (IDxC:21052237:DFM17) 
{ "type": "DFM", "frame": 1344693107, "id": "DFM-21052237", "datetime": "2022-08-16T13:51:47.000Z", "lat": 52.24825, "lon": 14.42914, "alt": 12535.70000, "vel_h": 12.69000, "heading": 109.19000, "vel_v": 4.10000, "sats": 15, "batt": 4.92, "temp": -50.2, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [233] 2022-08-16 13:51:48.0 (0,0,0)   lat: 52.24821 (0)   lon: 14.42933 (0)   alt: 12540.2 (0)   vH: 14.20  D: 111.3  vV:  4.58   T=-50.2C 
{ "type": "DFM", "frame": 1344693108, "id": "DFM-21052237", "datetime": "2022-08-16T13:51:48.000Z", "lat": 52.24821, "lon": 14.42933, "alt": 12540.20000, "vel_h": 14.20000, "heading": 111.33000, "vel_v": 4.58000, "sats": 15, "batt": 4.92, "temp": -50.2, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [246] 2022-08-16 13:52:01.0 (0,0,0)   lat: 52.24783 (0)   lon: 14.43190 (0)   alt: 12597.3 (0)   vH: 13.65  D: 104.9  vV:  4.83  (IDxC:21052237:DFMxX) 
{ "type": "DFM", "frame": 1344693121, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:01.000Z", "lat": 52.24783, "lon": 14.43190, "alt": 12597.30000, "vel_h": 13.65000, "heading": 104.92000, "vel_v": 4.83000, "sats": 15, "batt": 4.90, "subtype": "0xC:DFMxX", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [255] 2022-08-16 13:52:10.0 (0,0,0)   lat: 52.24752 (0)   lon: 14.43365 (0)   alt: 12633.5 (0)   vH: 13.73  D:  93.6  vV:  3.79 
{ "type": "DFM", "frame": 1344693130, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:10.000Z", "lat": 52.24752, "lon": 14.43365, "alt": 12633.50000, "vel_h": 13.73000, "heading": 93.57000, "vel_v": 3.79000, "sats": 15, "batt": 4.90, "subtype": "0xC:DFMxX", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [  1] 2022-08-16 13:52:12.0 (0,0,0)   lat: 52.24750 (0)   lon: 14.43405 (0)   alt: 12642.3 (0)   vH: 13.90  D: 100.5  vV:  4.81 
{ "type": "DFM", "frame": 1344693132, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:12.000Z", "lat": 52.24750, "lon": 14.43405, "alt": 12642.30000, "vel_h": 13.90000, "heading": 100.48000, "vel_v": 4.81000, "sats": 15, "batt": 4.90, "subtype": "0xC:DFMxX", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [  9] 2022-08-16 13:52:20.0 (0,0,0)   lat: 52.24723 (0)   lon: 14.43564 (0)   alt: 12674.3 (0)   vH: 13.05  D: 100.0  vV:  3.85 
{ "type": "DFM", "frame": 1344693140, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:20.000Z", "lat": 52.24723, "lon": 14.43564, "alt": 12674.30000, "vel_h": 13.05000, "heading": 99.96000, "vel_v": 3.85000, "sats": 15, "batt": 0.41, "subtype": "0xC:DFMxX", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [ 20] 2022-08-16 13:52:31.0 (0,0,0)   lat: 52.24696 (0)   lon: 14.43760 (0)   alt: 12719.9 (0)   vH: 12.57  D: 104.9  vV:  4.51   T=-51.0C 
{ "type": "DFM", "frame": 1344693151, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:31.000Z", "lat": 52.24696, "lon": 14.43760, "alt": 12719.90000, "vel_h": 12.57000, "heading": 104.94000, "vel_v": 4.51000, "sats": 15, "batt": 4.89, "temp": -51.0, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [ 21] 2022-08-16 13:52:32.0 (0,0,0)   lat: 52.24693 (0)   lon: 14.43778 (0)   alt: 12724.6 (0)   vH: 12.80  D: 106.9  vV:  4.48   T=-51.0C 
{ "type": "DFM", "frame": 1344693152, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:32.000Z", "lat": 52.24693, "lon": 14.43778, "alt": 12724.60000, "vel_h": 12.80000, "heading": 106.91000, "vel_v": 4.48000, "sats": 15, "batt": 4.89, "temp": -51.0, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [ 22] 2022-08-16 13:52:33.0 (0,0,0)   lat: 52.24689 (0)   lon: 14.43796 (0)   alt: 12729.1 (0)   vH: 12.95  D: 108.3  vV:  4.16   T=-51.0C 
{ "type": "DFM", "frame": 1344693153, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:33.000Z", "lat": 52.24689, "lon": 14.43796, "alt": 12729.10000, "vel_h": 12.95000, "heading": 108.34000, "vel_v": 4.16000, "sats": 15, "batt": 4.89, "temp": -51.0, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [ 29] 2022-08-16 13:52:40.0 (0,0,0)   lat: 52.24679 (0)   lon: 14.43925 (0)   alt: 12767.6 (0)   vH: 11.31  D:  89.3  vV:  7.55   T=-51.2C 
{ "type": "DFM", "frame": 1344693160, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:40.000Z", "lat": 52.24679, "lon": 14.43925, "alt": 12767.60000, "vel_h": 11.31000, "heading": 89.35000, "vel_v": 7.55000, "sats": 15, "batt": 4.89, "temp": -51.2, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

<-> [ 31] 2022-08-16 13:52:42.0 (0,0,0)   lat: 52.24678 (0)   lon: 14.43956 (0)   alt: 12783.0 (0)   vH:  9.95  D: 103.6  vV:  7.30   T=-51.0C  (IDxC:21052237:DFM17) 
{ "type": "DFM", "frame": 1344693162, "id": "DFM-21052237", "datetime": "2022-08-16T13:52:42.000Z", "lat": 52.24678, "lon": 14.43956, "alt": 12783.00000, "vel_h": 9.95000, "heading": 103.61000, "vel_v": 7.30000, "sats": 15, "batt": 4.89, "temp": -51.0, "subtype": "0xC:DFM17", "ref_datetime": "UTC", "ref_position": "GPS", "diff_GPS_MSL": 41.40 }

The only case I found in my test data that would reset the subtype. Before it would continue to show "DFM17", however in other situations it might switch to "DFM09P", so I think it is better to reset as long as there is something wrong with the temperature. I guess this could also happen if the sensor is broken, but that's ok.

[AstroTaka]

Thank yo very much.

Recently, there is no DFM receive. I will continue to check DFM sonde.