Weather Underground Precip Rate = Precip Accum

[GeekyTim]

Since about 11th August I have noticed that on my Weather Underground results the Precip Rate = Precip Accum. I had to reinstall pywws due to an issue with my SD card, but managed to rescue the data, but installed the latest pywws.

I must admit, I don't know if it is a pywws issue or Weather Underground. My MQTT messages are showing that the station is reporting different values, so that appears to be correct.

Weather station is at: https://www.wunderground.com/personal-weather-station/dashboard?ID=IPOTTON3#history/s20180811/e20180811/mdaily

[jim-easterbrook]

The 10mm jump at 1am is very odd as well. I haven't noticed anything odd with mine. There's actually some rain today, 23/08/2018. https://www.wunderground.com/personal-weather-station/dashboard?ID=ISURREYE4

Running pywws at high verbosity should show what data is actually being uploaded to Weather Underground. (And all your other services.)

The underground service module uses rain_hour and rain_day to calculate rate and accum. These are defined in pywws.template. You could try some manual calculations on your calib data for comparison.

[GeekyTim]

Turning verbosity to full, I see:

18:51:05:pywws.service.mqtt:publishing on topic "weather/now" with retain=False, data="{u'rainin': u'0', u'rain': u'0', u'wind_ave': u'0.67', u'rel_pressure': u'29.8962', u'hum_in': u'48', u'temp_out_c': u'20.1', u'wind_dir': u'315', u'temp_in_c': u'23.6', u'hum_out': u'55', u'dailyrain': u'1.8', u'wind_gust': u'2.24', u'idx': u'2018-08-23 17:48:57', u'dailyrainin': u'0.0708661'}"

and 18:51:05:requests.packages.urllib3.connectionpool:https://weatherstation.wunderground.com:443 "GET /weatherstation/updateweatherstation.php?windgustmph=2.24&tempf=68.2&indoorhumidity=48&rainin=0&winddir=315&dewptf=51.4&humidity=55&ID=IPOTTON3&dateutc=2018-08-23+17%3A48%3A57&softwaretype=pywws&windspeedmph=0.67&baromin=29.8962&dailyrainin=0.0708661&action=updateraw&indoortempf=74.5&PASSWORD=xxxxxxxHTTP/1.1

So it looks like it's passing the right info to WeatherUnderground, which means it's WU that is the issue :-(

[GeekyTim]

Just gone back to WU after restarting the service, and it looks to be 'back to normal'!

[GeekyTim]

And it's broken again! Convinced it's WU

[GeekyTim]

Ok, changed my mind.

I stopped pywws and ran it manually with verbose logging. pywws is sending 'reainin=0.141732' as well as 'dailyrainin=0.141732'. It hasn't rained today yet. However, a few minutes later it sent the correct values of 'reainin=0' and dailyrainin=0. This was also being sent by MQTT. Therefore I think there must be something going wrong within pywws.

Where/how do I read the raw data coming from my weather station itself? It's type 1080.

[GeekyTim]

Got it:

sudo pywws-testweatherstation -d 10:44:14:pywws.logger:pywws version 18.8.0, build 1570 (4e3889c) {'abs_pressure': 1008.4, 'alarm': {'abs_pressure': {'hi': 1040, 'lo': 960}, 'dewpoint': {'hi': 10, 'lo': -10}, 'hum_in': {'hi': 65, 'lo': 35}, 'hum_out': {'hi': 70, 'lo': 45}, 'illuminance': 0, 'rain': {'day': 150, 'hour': 3}, 'rel_pressure': {'hi': 1040, 'lo': 960}, 'temp_in': {'hi': 30, 'lo': 0}, 'temp_out': {'hi': 30, 'lo': -10}, 'time': '12:00', 'uv': 0, 'wind_ave': {'bft': 3, 'ms': 12.1}, 'wind_dir': 0, 'wind_gust': {'bft': 5, 'ms': 23.9}, 'windchill': {'hi': 20, 'lo': 0}}, 'alarm_1': {'hum_out_hi': False, 'hum_in_lo': False, 'hum_in_hi': False, 'hum_out_lo': False, 'time': False, 'wind_dir': False}, 'alarm_2': {'wind_ave': False, 'wind_gust': False, 'rain_hour': False, 'pressure_rel_lo': False, 'pressure_abs_hi': False, 'rain_day': False, 'pressure_rel_hi': False, 'pressure_abs_lo': False}, 'alarm_3': {'temp_out_hi': False, 'wind_chill_lo': False, 'dew_point_lo': False, 'temp_in_lo': False, 'wind_chill_hi': False, 'temp_in_hi': False, 'temp_out_lo': False, 'dew_point_hi': False}, 'current_pos': 25520, 'data_changed': 0, 'data_count': 4080, 'date_time': '2018-08-25 10:39:00', 'display_1': {'wind_gust': False, 'show_day_name': False, 'show_year': True, 'time_scale_24': False, 'pressure_rel': False, 'alarm_time': False, 'date_mdy': False, 'clock_12hr': False}, 'display_2': {'temp_out_temp': True, 'rain_hour': True, 'rain_month': False, 'rain_week': False, 'temp_out_chill': False, 'rain_day': False, 'temp_out_dew': False, 'rain_total': False}, 'display_3': {'illuminance_fc': False}, 'lux_wm2_coeff': 0, 'magic_0': 85, 'magic_1': 170, 'max': {'abs_pressure': {'date': '2015-01-01 12:02:00', 'val': 1033.4}, 'dewpoint': {'date': '2018-08-03 08:03:00', 'val': 20.9}, 'hum_in': {'date': '2015-01-01 12:28:00', 'val': 92}, 'hum_out': {'date': '2017-01-30 02:39:00', 'val': 99}, 'illuminance': {'val': 28774.4}, 'rain': {'day': {'date': '2017-05-17 21:10:00', 'val': 98.1}, 'hour': {'date': '2017-07-19 00:00:00', 'val': 31.5}, 'month': {'date': '2017-08-19 08:29:00', 'val': 382.5}, 'total': {'date': '2018-08-24 21:05:00', 'val': 2330.1}, 'week': {'date': '2017-07-24 11:52:00', 'val': 161.1}}, 'rel_pressure': {'date': '2017-04-19 11:10:00', 'val': 1038.8}, 'temp_in': {'date': '2018-07-15 19:53:00', 'val': 36}, 'temp_out': {'date': '2017-06-19 16:08:00', 'val': 37.5}, 'uv': {'val': 0}, 'wind_ave': {'date': '2017-02-23 14:16:00', 'val': 7.1}, 'wind_gust': {'date': '2017-02-23 15:39:00', 'val': 15}, 'windchill': {'date': '2017-06-19 16:08:00', 'val': 37.5}}, 'min': {'abs_pressure': {'date': '2017-12-10 12:34:00', 'val': 966.2}, 'dewpoint': {'date': '2018-02-28 18:39:00', 'val': -13.3}, 'hum_in': {'date': '2018-07-02 19:31:00', 'val': 12}, 'hum_out': {'date': '2015-01-01 12:00:00', 'val': 1}, 'rel_pressure': {'date': '2017-12-10 12:34:00', 'val': 973.6}, 'temp_in': {'date': '2015-01-09 18:42:00', 'val': -2.5}, 'temp_out': {'date': '2015-01-01 12:00:00', 'val': -40}, 'windchill': {'date': '2015-01-01 12:00:00', 'val': -40}}, 'read_period': 5, 'rel_pressure': 1015.8, 'settings_1': {'pressure_hPa': False, 'temp_out_F': False, 'rain_in': False, 'temp_in_F': False, 'pressure_inHg': False, 'pressure_mmHg': False}, 'settings_2': {'wind_knot': False, 'wind_bft': False, 'wind_kmph': False, 'wind_mps': False, 'wind_mph': True}, 'timezone': 0, 'unknown_01': 0, 'unknown_18': 96} min -> temp_out -> {'date': '2015-01-01 12:00:00', 'val': -40} alarm -> hum_out -> {'lo': 45, 'hi': 70} rel_pressure -> 1015.8 abs_pressure -> 1008.4

Although I'm not sure which bit I need to read!

[jim-easterbrook]

Your 'raw' data files should be an almost verbatim copy of what pywws has read from the station. The final number is the "status", the previous one is the total rain since the outside sensors were last reset. This number should only increase. (Note that the data is only flushed to file every hour unless you have "frequent writes" set to True.) The same rain figures should be in your 'calib' data, unless you have a user calibration routine that's doing strange things. You can do manual calculations of hourly and daily rain on these numbers as I suggested above.

[jim-easterbrook]

However, if it's sometimes sending the right numbers and sometimes not then there's something very odd happening. Are you running the current GitHub version of pywws?

[jim-easterbrook]

One other thing to consider. In "live" logging pywws sends live data every 48 seconds and (before my latest updates) logged data every 5 minutes. If you look at what it's sending and compare the time stamps with those in your raw data files you might be able to see if the wrong rain data is associated with live or logged data only.

[GeekyTim]

The only difference between my calib and raw data is the pressure.

Looking at today's data, the ACCUMULATION figures are correct. However, the RATE figures are exactly the same, which they should not be.

I am using the github version from a few days ago - the one I updated for MQTTS.

[GeekyTim]

I've connected up my weather station to the PC and retrieved the data. All looks okay there. The hourly rainfall looks exactly as I would think and matches the pywws raw and calib data.

[GeekyTim]

Just pulled the latest.

[GeekyTim]

I just realised that I am verbose logging (-v) to the systemd journal. In this I am seeing the following every 5 minutes:

Aug 27 10:20:02 PiWeather systemd[1]: Started pywws weather station live logging. Aug 27 10:20:11 PiWeather pywws-livelog[24721]: pywws.logger:pywws version 18.8.0, build 1598 (4f6cb2f) Aug 27 10:20:38 PiWeather pywws-livelog[24721]: pywws.weatherstation:status {'rain_overflow': False, 'lost_connection': False} Aug 27 10:20:40 PiWeather pywws-livelog[24721]: pywws.service.mqtt:OK Aug 27 10:20:40 PiWeather pywws-livelog[24721]: pywws.service.underground:server response "success" Aug 27 10:24:04 PiWeather pywws-livelog[24721]: pywws.weatherstation:live_data log extended Aug 27 10:25:01 PiWeather pywws-livelog[24721]: pywws.livelog:[Errno 13] Permission denied: '/home/pi/weather/data/calib/2018/2018-08/2018-08-26.txt' Aug 27 10:25:01 PiWeather pywws-livelog[24721]: Traceback (most recent call last):\n File "/usr/local/lib/python2.7/dist-packages/pywws-18.8.0-py2.7.egg/pyw Aug 27 10:25:01 PiWeather systemd[1]: pywws.service: Main process exited, code=exited, status=1/FAILURE Aug 27 10:25:01 PiWeather systemd[1]: pywws.service: Unit entered failed state. Aug 27 10:25:01 PiWeather systemd[1]: pywws.service: Failed with result 'exit-code'. Aug 27 10:25:02 PiWeather systemd[1]: pywws.service: Service hold-off time over, scheduling restart. Aug 27 10:25:02 PiWeather systemd[1]: Stopped pywws weather station live logging.

So it looks like pywws is restarting every 5 minutes. I'm sure that doesn't help things!

[GeekyTim]

Permissions change to being owned by pywws. Now not getting those errors

[jim-easterbrook]

I think the permissions problem might be the cause. rain_hour and rain_day both subtract a previous rain value from the current 'live' value. If calib data isn't being stored then the "nearest" record to one hour ago could also be the nearest to one day ago, so rain_hour and rain_day would both return the same value.

[GeekyTim]

I believe this is solved now! Getting different accum and rates!