TypeError: 'NoneType' object is not subscriptable

[claudobahn]

WeeWX 4.1.1 w/ latest weewx-windfinder fails for me while trying to report.

This seems to be happening while trying to do record = weewx.units.to_METRICWX(in_record) for presumably a None value.

Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: WindFinder: Unexpected exception of type <class 'TypeError'>
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: *** Traceback (most recent call last):
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 378, in run_loop
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***     self.process_record(_record, dbmanager)
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 426, in process_record
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***     _url = self.format_url(_full_record)
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***   File "/usr/share/weewx/user/windfinder.py", line 176, in format_url
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***     record = weewx.units.to_METRICWX(in_record)
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/units.py", line 1384, in to_METRICWX
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***     return to_std_system(datadict, weewx.METRICWX)
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/units.py", line 1388, in to_std_system
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: ***     if datadict['usUnits'] == unit_system:
Jun  4 22:26:22 weewx4 weewx[7985] ERROR weewx.restx: *** TypeError: 'NoneType' object is not subscriptable
Jun  4 22:26:22 weewx4 weewx[7985] CRITICAL weewx.restx: WindFinder: Thread terminating. Reason: 'NoneType' object is not subscriptable

[matthewwall]

what driver are you using?

it looks like you've got an empty archive record. that has nothing to do with windfinder - any uploader would fail on that.

[claudobahn]

Using weewx-interceptor. I have a second uploader running and that one has no issues with the same record.

[matthewwall]

what is the other uploader?

[claudobahn]

https://github.com/claudobahn/weewx-windguru which is based on this one. It's got the same conversion line, actually, which works fine.

[claudobahn]

I restarted weewx to try again and same thing:

Jun  5 10:13:24 weewx4 weewx[39627] DEBUG user.interceptor: raw packet: {'dateTime': 1591370004, 'usUnits': 1, 'temperature_in': 78.8, 'humidity_in': 46.0, 'pressure': 29.255, 'temperature_out': -40.0, 'wind_dir': 204.0, 'wind_speed': 3.13, 'wind_gust': 4.92, 'solar_radiation': 468.57, 'uv': 4.0, 'rain_rate': 0.0, 'rain_total': 0.012, 'wh65_battery': 0.0, 'rain': 0.0}
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG user.interceptor: mapped packet: {'dateTime': 1591370004, 'usUnits': 1, 'pressure': 29.255, 'inHumidity': 46.0, 'inTemp': 78.8, 'windSpeed': 3.13, 'windGust': 4.92, 'windDir': 204.0, 'radiation': 468.57, 'rain': 0.0, 'rainRate': 0.0, 'UV': 4.0}
Jun  5 10:13:24 weewx4 weewx[39627] INFO weewx.manager: Added record 2020-06-05 10:13:00 CDT (1591369980) to database 'weewx.sdb'
Jun  5 10:13:24 weewx4 weewx[39627] INFO weewx.manager: Added record 2020-06-05 10:13:00 CDT (1591369980) to daily summary in 'weewx.sdb'
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: WindFinder: Unexpected exception of type <class 'TypeError'>
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** Traceback (most recent call last):
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 378, in run_loop
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***     self.process_record(_record, dbmanager)
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 426, in process_record
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***     _url = self.format_url(_full_record)
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***   File "/usr/share/weewx/user/windfinder.py", line 176, in format_url
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***     record = weewx.units.to_METRICWX(in_record)
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/units.py", line 1384, in to_METRICWX
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***     return to_std_system(datadict, weewx.METRICWX)
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/units.py", line 1388, in to_std_system
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: ***     if datadict['usUnits'] == unit_system:
Jun  5 10:13:24 weewx4 weewx[39627] ERROR weewx.restx: *** TypeError: 'NoneType' object is not subscriptable
Jun  5 10:13:24 weewx4 weewx[39627] CRITICAL weewx.restx: WindFinder: Thread terminating. Reason: 'NoneType' object is not subscriptable
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Running reports for latest time in the database.
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'SeasonsReport' not enabled. Skipping.
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'MobileReport' not enabled. Skipping.
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Report 'StandardReport' not enabled. Skipping.
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Running report 'SmallImages'
Jun  5 10:13:24 weewx4 weewx[39627] DEBUG weewx.reportengine: Found configuration file /etc/weewx/skins/Images/skin.conf for report 'SmallImages'
Jun  5 10:13:25 weewx4 /weewxd: translategenerator.py: No language override specified.
Jun  5 10:13:25 weewx4 weewx[39627] DEBUG weewx.manager: Daily summary version is 2.0
Jun  5 10:13:25 weewx4 weewx[39627] INFO weewx.restx: WindGuru: Published record 2020-06-05 10:13:00 CDT (1591369980)

At the very bottom you can see the WindGuru push went through.

[matthewwall]

the exception indicates that in_record is None. i do not see how that could happen, so i must be missing something.

could you dump in_record to log in the function WindFinderThread.format_url so we can see if it is None when windfinder gets it?

[claudobahn]

I added a logdbg(in_record) to both uploaders as the very first thing in format_url. The difference is definitely puzzling.

Jun  5 10:23:24 weewx4 weewx[40329] DEBUG user.windfinder: None
Jun  5 10:23:24 weewx4 weewx[40329] ERROR weewx.restx: WindFinder: Unexpected exception of type <class 'TypeError'>
Jun  5 10:23:24 weewx4 weewx[40329] DEBUG user.windguru: {'dateTime': 1591370580, 'usUnits': 1, 'pressure': 29.267, 'inHumidity': 49.0, 'inTemp': 78.1, 'windSpeed': 3.6950000000000003, 'windDir': 206.84317153625466, 'windGust': 4.92, 'windGustDir': 220.0, 'radiation': 454.645, 'rain': 0.0, 'rainRate': 0.0, 'UV': 4.0, 'altimeter': 29.90933399921402, 'beaufort': 1.0, 'inDewpoint': 57.362256776145216, 'maxSolarRad': None, 'interval': 1.0, 'ET': None, 'windrun': 0.06158333333333334, 'hourRain': 0.0, 'rain24': 0.012, 'dayRain': 0.012}

[matthewwall]

what else is happening in your data pipeline? could you post the [[Services]] section of your weewx config file?

[matthewwall]

also, please do a logdbg(record) at the beginning of get_record and logdbg(rec) at the end in both windfinder and windguru

[claudobahn]

It appears it's the def get_record(self, record, dbm): method. I noticed @tkeffer had removed that from my plugin when he submitted the weewx4 PR, but it still seems to be there in yours.

[matthewwall]

yes. looks like confounded APIs. in the old days you would override get_record in order to check for essential observations. now tom has pushed that functionality into the restx base. (restx base has become rather convoluted, imho).

[claudobahn]

k, want me to submit a PR for removing that method or want to just take care of it?

Once that's taken care of, there's a Python3 string issue that needs to be addressed next :) Here's how I addressed it in my derivative uploader: https://github.com/claudobahn/weewx-windguru/commit/c0a9b04f9c24a418430d05a2fd1c52411ae6feac

[matthewwall]

decode patch applied. i would like to understand why the get_record invocation returns None. that is a pattern that is in use all over the place in other extensions...

[matthewwall]

oh, i see. get_record does not return a value. that line must have gotten lost in tom's edits.

[claudobahn]

The b'' is also necessary, because Python3 uses byte strings. Haven't dug into why that's happening, but the msg = ''.join(lines) will fail with TypeError: argument should be integer or bytes-like object, not 'str'

[matthewwall]

does that work in python2?

[claudobahn]

I'm not sure tbh, I've moved to weewx4 / Python3.

[matthewwall]

we have to maintain python2 compatibility. at least for a few more years :)

[matthewwall]

the b'' seems to be ok

i cannot keep track of all the python2/python3 idioms. especially not when i keep having to do shell scripting and perl too.

[matthewwall]

changes applied at commit a98448bcffa9bacf79d3ac0478b7c9cff06954dc

thank you for tracking this down!

[claudobahn]

Thanks for taking care of this so quickly! I'll update and try.

[tkeffer]

Sorry about forgetting to return the dictionary from get_record()!

I could probably do a better job of documenting restx, but it has a lot of advantages over just overriding process_record() and doing everything there. restx takes care of POST vs GET, application agent, retries, whether to skip a post, queue size, etc.

Plus, it has a (modest) test suite.

Worth learning to use it. I'll work on the docs.

[claudobahn]

I had a chance to give this a try. Unfortunately the byte string caused another issue a few lines up.

Here's the code that works for me right now:

    def check_response(self, response):
        """Override, and check the response for WF errors."""
        # this is a very crude way to parse the response, but windfinder does
        # not make things easy for us.  the status is contained within the
        # body tags in an html response.  no codes, no http status.  sigh.
        lines = []
        reading = False
        for line in response:
            line = line.decode('utf-8')
            if line.find('<body') >= 0:
                reading = True
            elif line.find('</body>') >= 0:
                reading = False
            elif reading:
                lines.append(line)
        msg = ''.join(lines)
        logdbg(msg)
        if not msg.startswith('OK'):
            raise weewx.restx.FailedPost("Server response: %s" % msg)

Basically, the decode moved further up and the b in b'' is no longer needed. It was blowing up on doing .find(str) on a byte string.

[tkeffer]

Wow. That's a pretty hostile server!