search

dl9rdz / rdz_ttgo_sonde

266 stars 93 forks source link

SondeHub Sonde Support #112

Closed LukePrior closed 2 years ago

LukePrior commented 3 years ago

I am opening this issue to document the compatibility of SondeHub uploaders for various sondes.

General

RS41

This sonde is uploaded with the correct time.

RS92

Might work, I think the ephemeris is broken and probably some other issues. I'm not sure if you want to continue supporting this considering they are almost all gone. It might be an idea to remove support from rdzTTGOsonde completely to get some extra space.

M20

This sonde seems to be a frame off auto_rx sometimes not sure why.

M10

This sonde is uploaded with the correct time.

DFM

This sometimes uploads correctly

MRZ

This has never been tested and I don't know anyone with one who can test.

dl9rdz commented 3 years ago

M10 should be fixed now. I guess heading calculation had x/y swapped, and the frame number got affected by switching to real UTC time stamps in sondehub data

LukePrior commented 3 years ago

Ok I made a PR to fix the data precision, do you know how we can get system milliseconds as struct tm seems to only go down to seconds. I'm also not sure if we have the milliseconds from sondes and if so we might as well send that too.

Also did you have any ideas on the caching of data for uploading?

dl9rdz commented 3 years ago

For uploaded, if I see it right the goal would be to do an API request with a multi-element array as payload instead of the single element we see right now? This would basically be rather straight forward to do, as long as a continuous stream of packets is received. After the last correct packet, some logic would be needed, i.e. wait for a few seconds and if no new data arrives, upload what there is.

Not sure about compression. There is a in-ROM miniz library on the ESP32 that should support deflate compression that might be compatible with gzip, but not sure if computational performance will be sufficient to handle that.

LukePrior commented 3 years ago

For uploaded, if I see it right the goal would be to do an API request with a multi-element array as payload instead of the single element we see right now? This would basically be rather straight forward to do, as long as a continuous stream of packets is received. After the last correct packet, some logic would be needed, i.e. wait for a few seconds and if no new data arrives, upload what there is.

Yep this is exactly right we upload data every 15s with auto rx but something as low as 5s would still be a big improvement.

We just need to send an array containing multiple of the telemetry objects we currently send e.g. [{telemetry},{telemetry}].

Not sure about compression.

It would be much more beneficial to have the multi element payload over compression. I wouldn't worry about it unless quite easy to implement.

LukePrior commented 3 years ago

Ok so now that packet packaging and M10/M20 have been fixed only a few minor things are left.

Do you know if every sonde frame will always include an RSSI value, heading, and vel_v/h @dl9rdz?

Getting milliseconds doesn't seem possible with the existing time solution so that idea may need to be scrapped.

dl9rdz commented 3 years ago

RSSI, heading, vel_v/h should always be available for all decoders.

Currently all time values are processed internally with seconds granularity. Not sure if all radiosondes transmit gps time in ms, but certainly some of them do, so it is not impossible to decode that. But would be some work to do. Regarding local time, there is some internal delay in asynchronous messaging between the radio part and the decoder part in the software, and also NTP time sync is never perfect, so for that adding a ms part to the time stamp would not be much better than adding a random value :)

LukePrior commented 3 years ago

RSSI, heading, vel_v/h should always be available for all decoders.

Ok cool, we won't need any checks for them then.

Currently all time values are processed internally with seconds granularity. Not sure if all radiosondes transmit gps time in ms, but certainly some of them do, so it is not impossible to decode that. But would be some work to do.

If we don't already have times in ms it probably isn't worth the development cost.

Regarding local time, there is some internal delay in asynchronous messaging between the radio part and the decoder part in the software, and also NTP time sync is never perfect, so for that adding a ms part to the time stamp would not be much better than adding a random value :)

No worries let's leave it as is.

I think that solves all the issues I found, we may get some server side-filtering implemented which can help with the distance issues.

LukePrior commented 3 years ago

I also see you tested MRZ and that seems to work.

LukePrior commented 3 years ago

The RS41 seems to have a different humidity value detected than what auto_rx produces, is the decoder different from what auto_rx uses?

I have attached an image of the data for T1910880 where a device running devel20210728 produces different values.

image

I believe this is the humidity decoder code used in auto rx: https://github.com/projecthorus/radiosonde_auto_rx/blob/master/demod/mod/rs41mod.c#L584

LukePrior commented 3 years ago

Yeah it looks like different code is used to perform the RS41 humidity calculation in this software just having a quick look at: https://github.com/projecthorus/radiosonde_auto_rx/blob/master/demod/mod/rs41mod.c#L584 against https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/libraries/SondeLib/RS41.cpp#L609

dl9rdz commented 3 years ago

Yes, I used the version from the ra source code (which since then has been extended slightly), and my version seems to be identical to oe5dxl's version. zilog (and autorx) seems to be use a less accurate humidity calculation, not making use of available calibration data...

darksidelemm commented 3 years ago

I don't want to annoy zilog too much given all the other awesome work they have been doing, so we'll probably accept the differences for now :-)

I might try and get some high resolution 'truth' data from the local bureau of meteorology to see how close the TTGO code is to truth.

mycarda commented 3 years ago

Yeah it looks like different code is used to perform the RS41 humidity calculation in this software just having a quick look at: https://github.com/projecthorus/radiosonde_auto_rx/blob/master/demod/mod/rs41mod.c#L584 against https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/libraries/SondeLib/RS41.cpp#L609

I am not an expert on this stuff but this is what I remember when I looked at adding the temperature and humidity code.

The humidity depends on the external temperature, the temperature of the humidity sensor heater and the water vapour saturation pressure (which depends on the temperature). The code is different for more than just the humidity.

Temperature rdz_ttgo_sonde - https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/libraries/SondeLib/RS41.cpp#L581 radiosonde_auto_rx - https://github.com/projecthorus/radiosonde_auto_rx/blob/master/demod/mod/rs41mod.c#L569 Both use the reference resistors and the resistance of the platinum sensor along with a calibration temperature to determine the actual temperature. As @dl9rdz says, radiosonde_auto_rx uses fewer of the calibration values than rdz_ttgo_sonde. rdz_ttgo_sonde also applies a polynomial correction to the temperature. I think this is something to do with the fact that at very low temperatures the platinum resistor might not have a linear response or it might be fact that to measure the resistance you need to pass current through the sensor and doing that causes slight sensor heating. What I found surprising was that, even though the code is different, the temperature value produced by both rdz_ttgo_sonde and radiosonde_auto_rx is remarkably similar. I suspect that some of the calibration values (especially for the higher order polynomial values) are zero which will make both pieces of code produce the same temperature value.

Water vapour saturation pressure rdz_ttgo_sonde - https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/libraries/SondeLib/RS41.cpp#L554 radiosonde_auto_rx - https://github.com/projecthorus/radiosonde_auto_rx/blob/master/demod/mod/rs41mod.c#L603 Both the rdz_ttgo_sonde code and radiosonde_auto_rx code are almost identical and use the Hyland and Wexler equation (https://www.eas.ualberta.ca/jdwilson/EAS372_13/Vomel_CIRES_satvpformulae.html scroll down to equation [13]). One difference is the "correction magic" in rdz_ttgo_sonde (https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/libraries/SondeLib/RS41.cpp#L559) which is commented out in radiosonde_auto_rx. I am not sure how much this affects the final relative humidity value.

Relative humidity rdz_ttgo_sonde - https://github.com/dl9rdz/rdz_ttgo_sonde/blob/devel/libraries/SondeLib/RS41.cpp#L609 radiosonde_auto_rx - https://github.com/projecthorus/radiosonde_auto_rx/blob/master/demod/mod/rs41mod.c#L584 The rdz_ttgo_sonde uses reference capacitors and calibration values to obtain an initial value for the humidity, then applies a 42 value calibration matrix, then adjusts for the water vapour saturation pressure. According to this reference (https://www.vaisala.com/sites/default/files/documents/White%20paper%20RS41%20Performance%20B211356EN-A.pdf) the Vaisala RS41 uses the ITS-90 compatible form of Wexler’s formula by Hardy. What I have failed to find is a reference paper with an equation that looks just like the rdz_ttgo_sonde code. The radiosonde_auto_rx code has a few hard-coded values with the comment "empirical" along with a reference to actual radiosonde German observations. Maybe this code has been optimised to fit the values to real-world observations. If this is the case, it is probably just as accurate as the long version of an equation but probably runs faster :-). Also note in the radiosonde_auto_rx code is a function called get_RH2adv https://www.vaisala.com/sites/default/files/documents/White%20paper%20RS41%20Performance%20B211356EN-A.pdf and this code is very similar to the code used in rdz_ttgo_sonde.

Next steps I am going to log all the calibration values for the next sonde I receive. I will use these to calculate both the rdz_ttgo_sonde and radiosonde_auto_rx temperatures, water vapour saturation pressures and relative humidity. Then I will see how much they differ from each other. I might even include the get_RH2adv calculation from radiosonde_auto_rx as well if I have time. Then, like @darksidelemm, I just need to find an "official" source of corresponding data to see the truth. That might be the Met Office here in the UK. There are also many open source RS41 decoders on GitHub so I might look through those to see if anyone has referenced the exact relative humidity equation they are using.

Wouldn't it be good if Vaisala open sourced the code they use in their RS41 calculations.

dl9rdz commented 3 years ago

For Germany, you find the official measurements as open data here https://opendata.dwd.de/climate_environment/CDC/observations_germany/radiosondes/high_resolution/recent/

The code in rdz_ttgo_sonde is just copied from other places, I never had a closer look at those calculations myself.

mycarda commented 3 years ago

As the humidity calculation depends on the temperature, I have started by looking at the temperature code in detail. In summary, I think the calculation used by rdz_ttgo_sonde and radiosonde_auto_rx are almost identical and agree to better than 0.001 degrees C.

If you want to know why I think that, here is what I did, else you can skip the rest of this.

I dumped all the calibration values from a few sonde launches I can receive from Larkhill (UK). I noticed that although rdz_ttgo_sonde uses more of the calibration values, many of these values are set to zero in the calibration frames. After eliminating all the parts of the calculation that used zero calibration values, I worked out (long hand) the difference. Although the first part of rdz_ttgo_sonde and radiosonde_auto_rx looks different in code, when you expand it out mathematically, it's the same. The second part of the calculation when the polynomial correction is applied is mathematically different but the the product of polyT[0] and polyT[1] (these variables are called calT[1] and calT[2] in radiosonde_auto_rx). Typically, these calibration values are small (-0.11625 and 0.00696) so when multiplied, they amount to a tiny difference (-0.00081 deg C).

IMG_20210810_092120

Where there might be a significant difference is if an RS41 sonde was set up with fewer non-zero calibration values.

The radiosonde_auto_rx will calculate a temperature slightly faster (because the zero calibration values are not in the calculation) but given the speed of an ESP32, I think the difference will be microseconds. On a personal note, I find the rdz_ttgo_sonde calculation easier to understand and future-proof in case the calibration values are non-zero.

Next steps. I have found a bug. The code tries to work out the humidity before we have all the calibration frames required. It's a problem with my original code so I will fix, test, and push an update. Also, I think there might be another bug because I cannot see where the the variable used to check we have enough frames (s->valid) is initialised to zero. I intermittently see the humidity calculation attempted before all the frames are received and this might be due to random values in the validation variable if it is not initialised to zero. I will set this to zero when it is first created, test a few times, then add to the same pull update. Then I will start looking for some soundings data for radiosondes I can receive.

LukePrior commented 3 years ago

Hi @mycarda

Great work on all this you seem to have a strong understanding of the equations.

You might be interested in the Comparison dashboard on the SondeHub Elastic Search database it let's you compare the temperature and humidity values uploaded to SondeHub for a specific radiosonde.

You need to login to view the site&_a=(description:'',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'8fec99f0-6222-11eb-b862-2f0a42a5ce55',key:serial,negate:!f,params:(query:S4610683),type:phrase),query:(match_phrase:(serial:S4610683)))),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!f,title:Comparison,viewMode:view)).

That link should take you to the comparison for the last radiosonde launch from Adelaide.

The page should look like this:

image

You can zoom in and compare on a per-second basis:

image

mycarda commented 3 years ago

Hi @LukePrior

Brilliant! That's the missing part I needed to try to get a little nearer to "truth". I have data from https://opendata.dwd.de/climate_environment/CDC/observations_germany/radiosondes/high_resolution/recent/ suggested by @dl9rdz and I assume this is "official" data so can be considered the "truth". However, although I run both radiosonde_auto_rx and rdzTTGOsonde here, I am too far away to receive any German radiosondes. The prevailing wind blows them all away from my location in the UK.

Here is what I did.

I used https://opendata.dwd.de/climate_environment/CDC/observations_germany/radiosondes/high_resolution/recent/ to obtain the data for Altenstadt (DE) for 2021-08-06 for radiosonde R3320859. I used a query from https://es.v2.sondehub.org/ to extract data from sondehub.org for R3320859 for the same date for radiosonde_auto_rx using rdzTTGOsonde.

The data in https://opendata.dwd.de/climate_environment/CDC/observations_germany/radiosondes/high_resolution/recent/ is for every 2m altitude rather than frames from a radiosonde and I could not save my query so could not download the data easily so I had to match up the data manually based on altitude. I checked every 500m but sometimes I could not match the altitude exactly but it's all less than a 5m variation. I had to start at 9000m because there was no data from es.v2.sondehub.org for R3320859 lower than this height.

I also had to make the assumption that AE_RF is the German for relative humidity (Feuchtigkeit?). Sorry, humidity was not one of the words on my German vocabulary list when I did German at school in 1978 :-).

Here are the results with the differences between the German observation data, rdzTTGOsonde and radiosonde_auto_rx. I had to use a picture as I was not able to format a table easily.

observationdata

In trying to answer "which code is correct" what I see is sometimes rdzTTGOsonde is more accurate and sometimes radiosonde_auto_rx is more accurate. Sometimes (when the observation shows 80% humidity), both radiosonde_auto_rx and rdzTTGOsonde are way off.

As I don't seem to be nearer the "truth", I am opening this up for peer review in case my assumptions are not valid or I have made a mistake somewhere. Don't worry, I will not be upset if you find I have made a mistake, @everybody I will be glad you have seen my error (if any) and let me know.

Next steps Try this again for some other German radiosondes for other dates to see if that gets any nearer the "truth".

LukePrior commented 3 years ago

We now have checks on the server side for data validity so some users running older versions may receive error messages when uploading M10 sondes with sats as 0. This has been fixed in recent versions.

These server side checks also mean we no longer have to implement serial regex and distance checks on device.

LukePrior commented 3 years ago

Next steps. I have found a bug. The code tries to work out the humidity before we have all the calibration frames required. It's a problem with my original code so I will fix, test, and push an update. Also, I think there might be another bug because I cannot see where the the variable used to check we have enough frames (s->valid) is initialised to zero. I intermittently see the humidity calculation attempted before all the frames are received and this might be due to random values in the validation variable if it is not initialised to zero. I will set this to zero when it is first created, test a few times, then add to the same pull update. Then I will start looking for some soundings data for radiosondes I can receive.

Did you have any luck getting this implemented?

I should also note that @eben80 has also compiled a comparison table here: https://github.com/dl9rdz/rdz_ttgo_sonde/issues/152

I wonder if anyone has managed to make a decoder that perfectly matches Viasala's.

LukePrior commented 2 years ago

I will close this issue so discussion can continue in #152.