Mismatch of rain units?

[nicholaswaton]

Hello!

Thanks so much for this wonderful driver! It is so useful.

We've been using it for our Sailing Club on this page here: https://status.cal-sailing.org/ One thing we've noticed is that the rain and rainrate is getting grossly exaggerated in inches (we are located in the US) but it would make sense if it was in mm. In fact our Tempest page shows the right amount of rain in inches if the numbers on WeeWx were in fact mm. https://tempestwx.com/station/127241/grid

I've been trying to figure out if there is a bug somewhere, or a misconfiguration on my end.

[nicholaswaton]

[livysdad27]

Thanks for the report @nicholaswaton . Also sharing your site and the kind words. I did a quick scan of the code. The driver is hard coded now to use metric as a default. The 'usUnits' value (strangely implies imperial but it isn't) is set to weewx.METRICWX. That inherited value comes from weewx itself and is tied to the device and API. My recollection is that all the weatherflow Tempest data comes through in metric but I can double check. It's pretty much the 'rain' value that's collected so I don't think it does any aggregate measurements which means it's weewx that's doing the cumulative. I'll re-scan the documents from Tempest again in the next couple days and get back to you!

        if "type" in resp:
            if resp['type'] == 'obs_st':
                mqtt_data = resp['obs'][0]
                loop_packet['dateTime'] = mqtt_data[0]
                loop_packet['usUnits'] = weewx.METRICWX
                loop_packet['outTemp'] = mqtt_data[7]
                loop_packet['outHumidity'] = mqtt_data[8]
                loop_packet['pressure'] = mqtt_data[6]
                loop_packet['supplyVoltage'] = mqtt_data[16]
                loop_packet['radiation'] = mqtt_data[11]
                loop_packet['rain'] = mqtt_data[19]
                loop_packet['UV'] = mqtt_data[10]
                loop_packet['lightening_distance'] = mqtt_data[14]
                loop_packet['lightening_strike_count'] = mqtt_data[15]
                loop_packet['windDir'] = mqtt_data[4]
                loop_packet['windGust'] = mqtt_data[3]
                loop_packet['windSpeed'] = mqtt_data[1]

[nicholaswaton]

Thanks for looking into it.

Indeed, the Cumulative graph is something that the skin I use does (WeeWX WDC). I added a rain graph just below which just uses the rain observations and it is too high compared to all the reliable weather stations nearby. For instance, another weather not too far away from us shows 0.38 in: https://www.pwsweather.com/station/mid/av737 When ours shows 12.10 inches. In fact it was mm than it would be closer to being correct, 12.10mm is 0.48 inches. For reference, our station is this one: https://www.pwsweather.com/station/pws/calsailingclub (WeeWx publishes the data to pwsweather.com)

I am trying to figure out where does the mismatch happen, and how to correct it.

[nicholaswaton]

I checked what does the API return for my station and here's the output: { "elevation": 2.977462768554688, "is_public": true, "latitude": 37.86297, "longitude": -122.31367, "obs": [ { "air_density": 1.22039, "air_temperature": 12.7, "barometric_pressure": 1001.4, "brightness": 2, "delta_t": 1.2, "dew_point": 10.6, "feels_like": 12.7, "heat_index": 12.7, "lightning_strike_count": 0, "lightning_strike_count_last_1hr": 0, "lightning_strike_count_last_3hr": 0, "precip": 1.400992, "precip_accum_last_1hr": 23.990288, "precip_accum_local_day": 325.356182, "precip_accum_local_day_final": 30.245059, "precip_accum_local_yesterday": 0.015309, "precip_accum_local_yesterday_final": 0.000321, "precip_analysis_type_yesterday": 1, "precip_minutes_local_day": 683, "precip_minutes_local_yesterday": 3, "precip_minutes_local_yesterday_final": 3, "pressure_trend": "falling", "relative_humidity": 87, "sea_level_pressure": 1002.3, "solar_radiation": 0, "station_pressure": 1001.4, "timestamp": 1706752670, "uv": 0, "wet_bulb_globe_temperature": 11.9, "wet_bulb_temperature": 11.5, "wind_avg": 9.4, "wind_chill": 12.7, "wind_direction": 125, "wind_gust": 11.7, "wind_lull": 4.9 } ], "outdoor_keys": [ "timestamp", "air_temperature", "barometric_pressure", "station_pressure", "pressure_trend", "sea_level_pressure", "relative_humidity", "precip", "precip_accum_last_1hr", "precip_accum_local_day", "precip_accum_local_day_final", "precip_accum_local_yesterday_final", "precip_minutes_local_day", "precip_minutes_local_yesterday_final", "wind_avg", "wind_direction", "wind_gust", "wind_lull", "solar_radiation", "uv", "brightness", "lightning_strike_count", "lightning_strike_count_last_1hr", "lightning_strike_count_last_3hr", "feels_like", "heat_index", "wind_chill", "dew_point", "wet_bulb_temperature", "wet_bulb_globe_temperature", "delta_t", "air_density" ], "public_name": "Cal Sailing Club", "station_id": 127241, "station_name": "Cal Sailing Club", "station_units": { "units_direction": "cardinal", "units_distance": "km", "units_other": "imperial", "units_precip": "in", "units_pressure": "inhg", "units_temp": "f", "units_wind": "kts" }, "status": { "status_code": 0, "status_message": "SUCCESS" }, "timezone": "America/Los_Angeles" }

That precip_accum_local_day is awfully high and not realistic if it's mm. Reading this, I am assuming that precip_accum_local_day_final is their Rain NearCast value and it is really different.

[livysdad27]

That's fascinating. I think I may have borked this from the beginning. https://weatherflow.github.io/Tempest/api/ws.html suggests I'm using a field that's likely accumulation already. That said, there's a separate set of documentation at https://apidocs.tempestwx.com/reference/websocket-reference. In both cases though I suspect the field in use is an aggregate and not "interval" field so that would explain the highly inflated values. Can you confirm, for the sake of my minimal sanity, that the API response you're displaying above is from the weatherflow REST API and not another source such as weewx or elsewhere? I can start working on the "correct" field for this however the default units for both REST, websocket and UDP. One link shows field 19 (what I use for rain) as Nearcast accumulated and the other shows.... Rain Accumulated Final (Rain Check) so either way it's likely the wrong data. I'll get some clarification from weatherflow and we'll see if we can sort this.

That said, if the REST data above is from weatherflow's API and it's totally borked even in mm then you might need to have them look at your device or log a ticket.

This is a fun mystery for sure....

[nicholaswaton]

I confirm that this is from the REST API (https://weatherflow.github.io/Tempest/api/swagger/#!/observations/getObservationsByDeviceId)

[livysdad27]

Awesome, thank you. My code isn't solid, the field chosen was likely wrong from the start but I need to open a ticket with weatherflow to double check which field is right and which documentation source is correct. Your output from the API SHOULD according to both sources, be in mm though so you may have a device or other issue as well. I'll keep you posted on what I learn. Thanks for all the help in troubleshooting this.

[nicholaswaton]

One theory is that the device is "vibrating" because it is very windy, and it is messing up the haptic sensor for rain and that's why it is inflated. I just find it interesting that it would be actually pretty accurate if it was just in the wrong unit. I'll send a message as well to the WeatherFlow support to get their thoughts.

[livysdad27]

That's a good theory if you could correlate bad readings to wind speed. My current theory is that I'm using the wrong field. In the obs_st payload field 19 shows in the documentation as aggregate. Field 12 shows as "since last observation" which I think would be five minute slices or so and Weewx would use the timestamp to generate it correctly. An EASY test of this would be to update line 184 of the tempestWS.py and change the 19 to a 12. I either got it wrong to begin with or accidentally adjusted it during an update but I'm not good enough at git to know for sure :^)

                loop_packet['rain'] = mqtt_data[19] 

If you're not comfortable doing this then I can make a branch and give you instructions for installing/running that also. Your choice. I'd run the test on my installation but my battery pack got borked and I'm in a bit of shadow from the north facing trees. https://weather.boggyhollowfarm.com

Lemmieknow

[nicholaswaton]

I've been able to make this change. Let's see how the rain totals look now. Good thing is that we are in the middle of a storm here in the San Francisco Bay Area.

[livysdad27]

Right on!

[nicholaswaton]

After running it for several hours it does look bette and it does fall in line with some nearby weather stations: https://status.cal-sailing.org/index.html

It is still more than what I see on my tempestwx dashboard (I guess Weatherflow applies even more correction there): https://tempestwx.com/station/127241/graph/314522/rain/3

Did Weatherflow saying anything about the API?

[livysdad27]

That's good news and hopefull. I also see deltas between the rest API and raw UDP or Websocket data. My understanding is that they do massage the data in their platform. I haven't done a direct compare of the REST/Websocket data though so that would be a good test.

They haven't responded yet beyond letting me know they assigned they triaged and assigned the ticket. I'm going to replace my battery pack batteries today and make the change on my station as well and do some comparisons but all up this is likely a good start. Thanks for the test! Will keep you posted.

[livysdad27]

Tempest did get back to me and indicate that https://apidocs.tempestwx.com/reference/quick-start should be considered the source of truth for their API documentation. I'll drop a pull request to update the readme. Let me know if the update you did seems to synch correctly with other weather monitors. If so I'll include the field change and close the bug. I do have one other question to Weatherflow but with these two things confirmed I think the resolution seems correct.

[nicholaswaton]

It is still overestimating the amount of rain. It has measured almost 1 inch today when all the nearby station show around 0.25/0.30 inches. https://www.pwsweather.com/station/pws/calsailingclub https://www.pwsweather.com/station/mid/av737

not sure if it's the code, or because of the Tempest's rain sensor though.

[nicholaswaton]

I am curious, if Field 12 is "since last observation" and it would be 5 minute slices, if I have configured WeeWx to have an archive_interval of 60 seconds, would it lead to duplicate amounts? Or does WeeWx figure that out when it receives the data?

[livysdad27]

That's a good question. I'm looking into that very thing. Also, in the "correct" documentation per Weatherflow field 19 is the "nearcast" rain so it's adjusted.

I have my station updated and back up and running now but we haven't had much rain for me to do a test against. I'll have to check that later in the week when we're supposed to get a bit more.

Thank you for the updates and the partnership in solving this. I like your interval hypothesis and am researching that tonight.

[livysdad27]

I believe the accumulator should be handling all of the loop packets (generated once/minute by the websocket endpoint) correctly. I'm going to post something in the weewx developer group to double check my thinking though. If they indicate I'm correct then it MIGHT be a hardware calibration issue. Which would be wacky.

Basically the archive interval and speed by which packets are generated should be independent.

[livysdad27]

OK, so after consulting with the weewx developer forum and the dev documentation it really looks like this is probably an issue with your device. The weatherflow forum is a good source of info also. There is a link to this...

Calibration Page

hiding in one of the threads there. I suspect this, and dampening the vibrations might be a good way to go.

The forum has been super helpful for dev and getting in touch with support sometimes also. My guess though is that we're dealing with the goofiness of the haptic rain sensor.

Any chance you could tell me if the patch change or the original setup had a bigger delta? I.e. di the discrepancy go down after you switched to field 12 or did it go up?

Let me know and thanks for logging this....

[nicholaswaton]

No in the end it didn't work, it detected almost 50 inches of rain yesterday during the epic storm we had in California: https://status.cal-sailing.org/day-archive/day-2024-02-04.html The storm was bad but not that insane. The official count is around 4 inches. The Tempest also had a problem showing the wind accurately.

I think that either the device is faulty, or it is really because the pole it is mounted is vibrating too much, especially in heavy rain and wind and it messes up the readings completely.

So I don't think I can be very helpful.

[livysdad27]

No sweat, I'm going to close this issue and annotate any of my other findings. From this I learned there are two potential sources of truth and I started an automation on my personal server to see if there's a meaningful difference between 12 and 19. I also think there are some lightening related messages that I could start listening to. Thanks for all your help here, would be super curious to find out if a new mounting makes a realistic difference.

On Mon, Feb 5, 2024 at 1:19 PM nicholaswaton @.***> wrote:

No in the end it didn't work, it detected almost 50 inches of rain yesterday during the epic storm we had in California: https://status.cal-sailing.org/day-archive/day-2024-02-04.html The storm was bad but not that insane. The official count is around 4 inches. The Tempest also had a problem showing the wind accurately.

I think that either the device is faulty, or it is really because the pole it is mounted is vibrating too much, especially in heavy rain and wind and it messes up the readings completely.

So I don't think I can be very helpful.

— Reply to this email directly, view it on GitHub https://github.com/livysdad27/tempestWS/issues/12#issuecomment-1928112814, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNPU3BN5JPTH7BU42E3LN3YSFEGRAVCNFSM6AAAAABCT6MBEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRYGEYTEOBRGQ . You are receiving this because you commented.Message ID: @.***>

-- Billy Jackson @.***

[livysdad27]

Finally got some rain today. I added a line of code to log the delta between the 12 field and the 19 field. Then I drop the values for each as well.... Feb 05 19:40:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.001355 - 0.001355 Feb 05 19:41:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.000828 - 0.000828 Feb 05 19:42:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.002082 - 0.002082 Feb 05 19:43:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.000828 - 0.000828 Feb 05 19:44:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.000369 - 0.000369 Feb 05 19:45:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.000904 - 0.000904 Feb 05 19:46:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.003203 - 0.003203 Feb 05 19:47:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.010559 - 0.010559 Feb 05 19:48:42 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.004073 - 0.004073 Feb 05 19:49:43 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.003047 - 0.003047 Feb 05 19:50:42 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.003292 - 0.003292 Feb 05 19:51:42 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.004906 - 0.004906 Feb 05 19:52:42 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.002323 - 0.002323 Feb 05 19:53:42 ip-172-26-7-9 python3[28838]: weewx[28838] INFO user.tempestWS: DELTA:0.0:0.002963 - 0.002963 ubuntu@ip-172-26-7-9:~$ ls

Not easy to read but the DELTA 0.0 means there's no difference so at least in my case the values are the same. Wil post if weatherflow shares a good answer for why but right now they appear, on my installation, to be the same.