Improving accuracy of frequency reported to Sondehub

[eben80]

Some of the auto_rx receivers in scanning mode that are reporting to Sondehub seems not to have done frequency calibration and as such they are reporting the wrong frequency back to the Sondehub api.

This means that when we use the sondehub api to request the frequencies for sondes in the area, we are getting also these which show the incorrect frequencies. One such example where the frequency was actually 400.6MHz:

"R5030861": {
    "software_name": "radiosonde_auto_rx",
    "software_version": "1.5.5",
    "uploader_callsign": "XXXX",
    "uploader_position": "XXX",
    "uploader_antenna": "1/4 wave monopole",
    "time_received": "2021-10-10T13:21:00.166403Z",
    "datetime": "2021-10-10T13:21:16.001000Z",
    "manufacturer": "Vaisala",
    "type": "RS41",
    "serial": "R5030861",
    "subtype": "RS41-SGP",
    "frame": 8552,
    "lat": 49.24005,
    "lon": 20.9442,
    "alt": 4700.08129,
    "temp": -12.7,
    "humidity": 80.4,
    "pressure": 573.62,
    "vel_v": -16.9317,
    "vel_h": 4.71026,
    "heading": 343.25841,
    "sats": 10,
    "batt": 2.6,
    "frequency": 400.577,
    "burst_timer": 29909,
    "snr": 15.8,
    "user-agent": "Amazon CloudFront",
    "position": "49.24005,20.9442",
    "upload_time_delta": -1.35,
    "uploader_alt": XXX
  },

Is there anything that could be done to rectify this apart from just having the request to calibrate in the installation docs?

[darksidelemm]

It's kind of annoying that this keeps happening, as the note about using a TCXO RTLSDR is there up the top of the installation guide, in bold. When I see these kind of offsets, I usually make the point of trying to contact the station and let them know.

As for what can be done about it... well the RS41 at least does send information about its transmit frequency, so that could possibly be used to at least warn the user. I'm a bit hesitant to go trying to automatically calibrate the RTLSDR based on that, as if the user is using a non-TCXO RTLSDR then it'll drift all over the place.

[eben80]

Yes I saw the instructions and I'm not even a user. Would it be an idea to use the frequency from telemetry for the rs41?

On Sun, 10 Oct 2021, 22:37 Mark Jessop, @.***> wrote:

It's kind of annoying that this keeps happening, as the note about using a TCXO RTLSDR is there up the top of the installation guide, in bold. When I see these kind of offsets, I usually make the point of trying to contact the station and let them know.

As for what can be done about it... well the RS41 at least does send information about its transmit frequency, so that could possibly be used to at least warn the user. I'm a bit hesitant to go trying to automatically calibrate the RTLSDR based on that, as if the user is using a non-TCXO RTLSDR then it'll drift all over the place.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/projecthorus/radiosonde_auto_rx/issues/577#issuecomment-939548723, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATNONF6IZOI4VGU76V5ASTUGH2PLANCNFSM5FWSMWJA .

[darksidelemm]

Part of the point of reporting the observed frequency is so we have some idea of the drift in the transmission. For the RS41 we've learned that this isn't really that much (maybe +/- a few kHz), so yes, we could probably pass the RS41 frequency along... However its worth noting that this information isn't sent with every frame (kind of like the calibration data for PTU), so we don't have it immediately.

What is your application for looking at the frequency data? If more than one station is receiving then hopefully its possible to figure out the frequency using the mean across the entire dataset.

[eben80]

Oh yes then it will not be available from the start. It's a functionality in rdzsonde where we query the SondeHub /sondes api with the receiver coordinates, max range and max age, in return we receive current reported sondes which are then automatically added to the scan list. So if the frequency reported to SondeHub is incorrect, you sit with duplicate entries for the same sonde potentially, with one having the wrong frequency.

On Sun, 10 Oct 2021, 22:55 Mark Jessop, @.***> wrote:

Part of the point of reporting the observed frequency is so we have some idea of the drift in the transmission. For the RS41 we've learned that this isn't really that much (maybe +/- a few kHz), so yes, we could probably pass the RS41 frequency along... However its worth noting that this information isn't sent with every frame (kind of like the calibration data for PTU), so we don't have it immediately.

What is your application for looking at the frequency data? If more than one station is receiving then hopefully its possible to figure out the frequency using the mean across the entire dataset.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/projecthorus/radiosonde_auto_rx/issues/577#issuecomment-939551175, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATNONBJEURIIP7U5VXJFDDUGH4T7ANCNFSM5FWSMWJA .

[darksidelemm]

Well, it's certainly something I can look into. It should be possible to add the frequency reported by the sonde into the JSON output from the decoder once it is available. This may be true for other radiosonde types too.

[eben80]

Great. Thank you very much for having a look.

On Sun, 10 Oct 2021, 23:07 Mark Jessop, @.***> wrote:

Well, it's certainly something I can look into. It should be possible to add the frequency reported by the sonde into the JSON output from the decoder once it is available. This may be true for other radiosonde types too.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/projecthorus/radiosonde_auto_rx/issues/577#issuecomment-939552980, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATNONEOKZLYAGNMEFGJS7TUGH57NANCNFSM5FWSMWJA .

[mwheeler-ep]

🤔 we might be able to make an ttgo specific endpoint on sondehub that reports the frequency which has the most reports rather than the current method

[eben80]

That would work well for areas with sufficient receivers yes. Anything that can be done closer to the source of the problem will of course help the areas with less receivers.

On Sun, 10 Oct 2021, 23:22 Michaela Wheeler, @.***> wrote:

🤔 we might be able to make an ttgo specific endpoint on sondehub that reports the frequency which has the most reports rather than the current method

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/projecthorus/radiosonde_auto_rx/issues/577#issuecomment-939555109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATNONEGE6AI3NMY2TH4P33UGH7ZRANCNFSM5FWSMWJA .

[LukePrior]

You could add an option to the auto rx configuration file to disable uploading the frequency so that users with these bad receivers could continue to upload telemetry but not introduce these errors.

[darksidelemm]

The problem is that the users who are uploading the bad information don't know they are doing it, so this won't really help.

[LukePrior]

Oh yeah absolutely more just so when contacting stations you can offer a simple solution instead of needing to purchase a new dongle or taking the station offline.

You could also make frequency uploading disabled by default until the user enables it with a message about needing a correct dongle for future versions.

[darksidelemm]

Again, i'd rather not disabling it by default. I think the solution of passing through the frequency information from the sonde telemetry (if it provides it) is a better approach. This is not going to work for all decoders however - some sonde definitely do not provide frequency information.

I'm not going to be able to work on any of this much over the next month. Hopefully i'll find some time to poke around in the decoders and see if we can add in the telemetry-provided frequency information in as many sondes as we can.

[darksidelemm]

So I've modified @rs1729 's rs41mod and rs92mod to output a tx_frequency field if that information has been extracted from the telemetry. This is currently just uploaded to SondeHub, and not used elsewhere.

I'm unsure if any other sonde types include frequency information in their telemetry data.

[rs1729]

RS41, RS92, and iMS-100/RS-11G transmit frequency.

rs41mod and rs92mod could already output the frequency in the telemetry with option --jsn_cfq <fq>: https://github.com/projecthorus/radiosonde_auto_rx/blob/65ba75a0a1923667033d65394a9a08648193105d/demod/mod/rs41mod.c#L1846-L1855 The only difference is that as long as the frequency from the telemetry is not received, <fq> is used in "freq". But I see that you only want the frequency from the data.

EDIT: I can include "tx_frequency" in ims10mod.

[darksidelemm]

Yes I noted that it would be output if jsn_cfq was used, but I'm not normally using that option when calling the demods in auto_rx. I figured keeping that existing functionality would be useful when people start up the demods externally and feed data in via UDP, with the telemetry-derived information available separately for clarity.

It'll be interesting to plot out the observed vs expected frequencies for a few sondes, once these updates are a bit further out into the wild.

Adding tx_frequency into the ims demod would be much appreciated! I'm wondering if similar information is available in other sonde telemetry, but just hasn't been found yet...

[rs1729]

OK, added "tx_frequency"also to meisei100mod.c in test-branch https://github.com/rs1729/RS/tree/test/demod/mod The Meisei radiosondes don't transmit serial number in every frame, there could be some "DFM" issues, though the iMS-100 is not flying so frequently, I guess it's ok for now.

[darksidelemm]

Thank for that! I've merged this change into the testing branch.

[eben80]

Thanks a lot for this 👍

[darksidelemm]

So the 'frequency' field in the telemetry from sondehub will still be the 'reported' frequency from the receiver. You will need to look for the presence of a 'tx_frequency' field - if this is present, that's frequency information from the sonde telemetry itself.

[darksidelemm]

This has been added for RS41, RS92 and iMS-100 sondes in auto_rx 1.5.7. I'm not sure if any other radiosonde types report frequency information in their telemetry, so I'm going to close this issue. If there is any other information available for other sonde types, please re-open this issue.

[eben80]

All good. Thank you for this.

On Thu, 11 Nov 2021, 01:08 Mark Jessop, @.***> wrote:

Closed #577 https://github.com/projecthorus/radiosonde_auto_rx/issues/577.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/projecthorus/radiosonde_auto_rx/issues/577#event-5600705957, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATNONDBGHKU5NND4BTK6VDULMCQXANCNFSM5FWSMWJA .