Closed claudobahn closed 4 years ago
what driver are you using?
it looks like you've got an empty archive record. that has nothing to do with windfinder - any uploader would fail on that.
Using weewx-interceptor. I have a second uploader running and that one has no issues with the same record.
what is the other uploader?
https://github.com/claudobahn/weewx-windguru which is based on this one. It's got the same conversion line, actually, which works fine.
I restarted weewx to try again and same thing:
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG user.interceptor: raw packet: {'dateTime': 1591370004, 'usUnits': 1, 'temperature_in': 78.8, 'humidity_in': 46.0, 'pressure': 29.255, 'temperature_out': -40.0, 'wind_dir': 204.0, 'wind_speed': 3.13, 'wind_gust': 4.92, 'solar_radiation': 468.57, 'uv': 4.0, 'rain_rate': 0.0, 'rain_total': 0.012, 'wh65_battery': 0.0, 'rain': 0.0}
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG user.interceptor: mapped packet: {'dateTime': 1591370004, 'usUnits': 1, 'pressure': 29.255, 'inHumidity': 46.0, 'inTemp': 78.8, 'windSpeed': 3.13, 'windGust': 4.92, 'windDir': 204.0, 'radiation': 468.57, 'rain': 0.0, 'rainRate': 0.0, 'UV': 4.0}
Jun 5 10:13:24 weewx4 weewx[39627] INFO weewx.manager: Added record 2020-06-05 10:13:00 CDT (1591369980) to database 'weewx.sdb'
Jun 5 10:13:24 weewx4 weewx[39627] INFO weewx.manager: Added record 2020-06-05 10:13:00 CDT (1591369980) to daily summary in 'weewx.sdb'
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: WindFinder: Unexpected exception of type <class 'TypeError'>
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** Traceback (most recent call last):
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** File "/usr/share/weewx/weewx/restx.py", line 378, in run_loop
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** self.process_record(_record, dbmanager)
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** File "/usr/share/weewx/weewx/restx.py", line 426, in process_record
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** _url = self.format_url(_full_record)
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** File "/usr/share/weewx/user/windfinder.py", line 176, in format_url
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** record = weewx.units.to_METRICWX(in_record)
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** File "/usr/share/weewx/weewx/units.py", line 1384, in to_METRICWX
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** return to_std_system(datadict, weewx.METRICWX)
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** File "/usr/share/weewx/weewx/units.py", line 1388, in to_std_system
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** if datadict['usUnits'] == unit_system:
Jun 5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** TypeError: 'NoneType' object is not subscriptable
Jun 5 10:13:24 weewx4 weewx[39627] CRITICAL weewx.restx: WindFinder: Thread terminating. Reason: 'NoneType' object is not subscriptable
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Running reports for latest time in the database.
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'SeasonsReport' not enabled. Skipping.
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'MobileReport' not enabled. Skipping.
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'StandardReport' not enabled. Skipping.
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Running report 'SmallImages'
Jun 5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Found configuration file /etc/weewx/skins/Images/skin.conf for report 'SmallImages'
Jun 5 10:13:25 weewx4 /weewxd: translategenerator.py: No language override specified.
Jun 5 10:13:25 weewx4 weewx[39627] DEBUG weewx.manager: Daily summary version is 2.0
Jun 5 10:13:25 weewx4 weewx[39627] INFO weewx.restx: WindGuru: Published record 2020-06-05 10:13:00 CDT (1591369980)
At the very bottom you can see the WindGuru push went through.
the exception indicates that in_record
is None. i do not see how that could happen, so i must be missing something.
could you dump in_record
to log in the function WindFinderThread.format_url
so we can see if it is None when windfinder gets it?
I added a logdbg(in_record)
to both uploaders as the very first thing in format_url. The difference is definitely puzzling.
Jun 5 10:23:24 weewx4 weewx[40329] DEBUG user.windfinder: None
Jun 5 10:23:24 weewx4 weewx[40329] ERROR weewx.restx: WindFinder: Unexpected exception of type <class 'TypeError'>
Jun 5 10:23:24 weewx4 weewx[40329] DEBUG user.windguru: {'dateTime': 1591370580, 'usUnits': 1, 'pressure': 29.267, 'inHumidity': 49.0, 'inTemp': 78.1, 'windSpeed': 3.6950000000000003, 'windDir': 206.84317153625466, 'windGust': 4.92, 'windGustDir': 220.0, 'radiation': 454.645, 'rain': 0.0, 'rainRate': 0.0, 'UV': 4.0, 'altimeter': 29.90933399921402, 'beaufort': 1.0, 'inDewpoint': 57.362256776145216, 'maxSolarRad': None, 'interval': 1.0, 'ET': None, 'windrun': 0.06158333333333334, 'hourRain': 0.0, 'rain24': 0.012, 'dayRain': 0.012}
what else is happening in your data pipeline? could you post the [[Services]]
section of your weewx config file?
also, please do a logdbg(record)
at the beginning of get_record
and logdbg(rec)
at the end in both windfinder and windguru
It appears it's the def get_record(self, record, dbm):
method. I noticed @tkeffer had removed that from my plugin when he submitted the weewx4 PR, but it still seems to be there in yours.
yes. looks like confounded APIs. in the old days you would override get_record
in order to check for essential observations. now tom has pushed that functionality into the restx base. (restx base has become rather convoluted, imho).
k, want me to submit a PR for removing that method or want to just take care of it?
Once that's taken care of, there's a Python3 string issue that needs to be addressed next :) Here's how I addressed it in my derivative uploader: https://github.com/claudobahn/weewx-windguru/commit/c0a9b04f9c24a418430d05a2fd1c52411ae6feac
decode patch applied. i would like to understand why the get_record invocation returns None. that is a pattern that is in use all over the place in other extensions...
oh, i see. get_record does not return a value. that line must have gotten lost in tom's edits.
The b''
is also necessary, because Python3 uses byte strings. Haven't dug into why that's happening, but the msg = ''.join(lines)
will fail with TypeError: argument should be integer or bytes-like object, not 'str'
does that work in python2?
I'm not sure tbh, I've moved to weewx4 / Python3.
we have to maintain python2 compatibility. at least for a few more years :)
the b''
seems to be ok
i cannot keep track of all the python2/python3 idioms. especially not when i keep having to do shell scripting and perl too.
changes applied at commit a98448bcffa9bacf79d3ac0478b7c9cff06954dc
thank you for tracking this down!
Thanks for taking care of this so quickly! I'll update and try.
Sorry about forgetting to return the dictionary from get_record()
!
I could probably do a better job of documenting restx, but it has a lot of advantages over just overriding process_record()
and doing everything there. restx takes care of POST vs GET, application agent, retries, whether to skip a post, queue size, etc.
Plus, it has a (modest) test suite.
Worth learning to use it. I'll work on the docs.
I had a chance to give this a try. Unfortunately the byte string caused another issue a few lines up.
Here's the code that works for me right now:
def check_response(self, response):
"""Override, and check the response for WF errors."""
# this is a very crude way to parse the response, but windfinder does
# not make things easy for us. the status is contained within the
# body tags in an html response. no codes, no http status. sigh.
lines = []
reading = False
for line in response:
line = line.decode('utf-8')
if line.find('<body') >= 0:
reading = True
elif line.find('</body>') >= 0:
reading = False
elif reading:
lines.append(line)
msg = ''.join(lines)
logdbg(msg)
if not msg.startswith('OK'):
raise weewx.restx.FailedPost("Server response: %s" % msg)
Basically, the decode moved further up and the b
in b''
is no longer needed. It was blowing up on doing .find(str)
on a byte string.
Wow. That's a pretty hostile server!
WeeWX 4.1.1 w/ latest weewx-windfinder fails for me while trying to report.
This seems to be happening while trying to do
record = weewx.units.to_METRICWX(in_record)
for presumably aNone
value.