Need clarification on data formats and configs

[crichmon762]

I've got this non-weather thingy I'm using with weewx for plotting and the like, and have a driver that works, but isn't the best, so I thought about switching to using MQTT for communications as part of the refactoring to python3 and modularizing a bit. After quite a bit of hair pulling, I think I have it working, but that required an edit in _on_message_keyword.

The issue was this: type <class 'list'>, fieldname ['datetime'], type <class 'float'>, value 1609953130.0

Traceback (most recent call last): File "/home/weewx/bin/user/MQTTSubscribe.py", line 1189, in _on_message_keyword data[fieldname] = value TypeError: unhashable type: 'list'

from: self._log_message(" type %s, fieldname %s, type %s, value %s" % (type(fieldname), fieldname, type(value),

Partly my fault for reading too fast, but the problem seems to be that the fieldname comes back from the parser as a list of one item. It took adding this commented print in the except: block to see the actual error: except Exception as exception: # (want to catch all) pylint: disable=broad-except

print("\n%s" % "".join(traceback.format_exc()))

        self._log_exception('on_message_keyword', exception, msg)

This is the partial list of the config and topics: [MQTTSubscribeDriver]

This section is for the MQTTSubscribe driver.

driver = user.MQTTSubscribe
host = router1
port = 1883
keepalive = 60
username = None
password = None

#console = True
#log = True
#logging_level = trace # trace 5, debug 10, info 20, warn 30,  error 40, critical 50

# Configuration for the message callback.
[[message_callback]]
    type = keyword
    keyword_delimiter = ", "

# The topics to subscribe to.
[[topics]]
    unit_system = US
    [[[cer8000]]]
       [[[[datetime]]]]
          name = 'datetime',      # epoch timestamp
       [[[[furnace]]]]
          name = 'furnace_dc',    # 3cyc running avg int(duty cycle)
       [[[[furnace_hour]]]]
          name = 'furnace_hour',  # int(duty cycle for last hour)
       [[[[furnace_day]]]]
          name = 'furnace_day',   # int(duty cycle for last day)
       [[[[furnace_cyccnt]]]]
          name = 'furnace_CyC',   # cycle count

and this is the data being sent to the broker: SendLoopStats type <class 'str'>, loop_record [datetime=1609965487, upright_freezer=0, upright_freezer_day=0, upright_freezer_cyccnt=1, upright_freezer_cyctime=0, space_heater=0, space_heater_hour=0, space_heater_day=0, space_heater_cyccnt=2, space_heater_cyctime=0, furnace=0, furnace_hour=0, furnace_day=0, furnace_cyccnt=1, furnace_cyctime=0, unused1=0, unused2=0, unused3=0, unused4=0, totalSupply=24.0]

And as received by the MQTT driver: (Driver) MQTTSubscribe MQTT: Received PUBLISH (d0, q0, r0, m0), 'cer8000', ... (359 bytes) (Driver) MessageCallbackProvider data-> incoming topic: cer8000, QOS: 0, retain: 0, payload: b'datetime=1609964886, upright_freezer=0, upright_freezer_day=0, upright_freezer_cyccnt=1, upright_freezer_cyctime=0, space_heater=0, space_heater_hour=0, space_heater_day=0, space_heater_cyccnt=2, space_heater_cyctime=0, furnace=0, furnace_hour=0, furnace_day=0, furnace_cyccnt=1, furnace_cyctime=0, unused1=0, unused2=0, unused3=0, unused4=0, totalSupply=24.0'

What seems to have got this going is the last line of this block (referencing item 0 of the 'list'): if not fields.get(key, {}).get('ignore', fields_ignore_default): (fieldname, value) = self._update_data(fields, key, field[eq_index + 1:].strip(), unit_system)

print("type %s, fieldname %s, type %s, value %s" % (type(fieldname), fieldname, type(value), value))

                data[fieldname[0]] = value

The rest of the keys/values are also getting parsed and stored and the few requested fields to be ignored are ignored as expected. I've got control over all ends of the process, so if there's a better way to go about this, please let me know.

As a side note, I just happened on how to enable debug logging in weewx.loop.conf. It wasn't referenced in the README, nor is how the data is supposed to be formatted based on how the topic config is defined. For almost any application I can think of, these docs would be really helpful. I'm retired and have time if I can be of assistance.

Thx, Chris

[bellrichm]

Hi, Sorry to hear of your struggles. I believe the root cause of you problem is the comma after all of your name values. This causes configObj to treat the value as a list. So, try

[[topics]]
    unit_system = US
    [[[cer8000]]]
       [[[[datetime]]]]
          name = 'datetime'      # epoch timestamp
       [[[[furnace]]]]
          name = 'furnace_dc'    # 3cyc running avg int(duty cycle)
       [[[[furnace_hour]]]]
          name = 'furnace_hour'  # int(duty cycle for last hour)
       [[[[furnace_day]]]]
          name = 'furnace_day'   # int(duty cycle for last day)
       [[[[furnace_cyccnt]]]]
          name = 'furnace_CyC'   # cycle count

Note, the quotes are not needed so this should work name = furnace_CyC # cycle count. You stumbled onto my experimental logging. Using the standard WeeWX debug = 1 should have gotten what you wanted, with the exception of the lack of exception logging. I knew that exception logging needed work, now that you have shown me the traceback module, I can improve that.
Yeah, the documentation is still a work in progress. I continue to round out the wiki. There is a bit on the experimental logging here. I’m not sure I will keep it, so I’m not ready to add to it. But, you got me thinking that a page for each of the supported MQTT message formats, individual, json, and keyword would be useful. Additional thoughts, comments, pull requests are welcome. -rich

[crichmon762]

Hi Rich, The commas certainly seemed to be the problem, so thanks for the hint. The quotes were leftover from other code when there were spaces in the values and didn't seem to cause an issue, but they are gone now anyway. The per-input type format/config wiki pages would be a great help.

As an aside, had I not found the logging code and setup and added the traceback code, I'd have never figured this out, so keep something, please. :^)

Thanks again, Chris

[bellrichm]

You’ve got me thinking that I should revamp the readme to be more of a ‘table of contents’ of wiki pages. One of these would be an introduction to debugging.

[earlthesquirrel]

I'd very MUCH agree that this would be helpful!

Not having to dig thru tickets to find info would be very helpful and help consolidate answers.

On Fri, Jan 8, 2021 at 10:29 AM Rich Bell notifications@github.com wrote:

You’ve got me thinking that I should revamp the readme to be more of a ‘table of contents’ of wiki pages. One of these would be an introduction to debugging.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bellrichm/WeeWX-MQTTSubscribe/issues/107#issuecomment-756816570, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCQJ22QQ6SJVRT23SIP5VTSY4QFJANCNFSM4VYDCJ6Q .

[bellrichm]

@earlthesquirrel Agreed, one should not have to dig thru tickets. Now that the code is fairly stable, I should have a bit more time for documentation. But if you have any questions/comments/feedback, feel free to open an issue, start a discussion in github, or post on WeeWX google group.

[bellrichm]

@crichmon762 Since you are up an running and documenting is going to be an ongoing process, I’m going to close this. If you have any more problems, etc, feel free to open an issue, start a discussion in github, or post on WeeWX google group.