MQTT publishing hangs after a while

[pgalbavy]

Not sure what is happening here, but running on an RPi with Jessie running the base leaf-python-mqtt works fine but after some hours (never measure fully) MQTT publication stops to a remote broker - an emonpi in this case.

The symptoms are that the process continues to log data acquisition from Nissan and publish to MQTT but the broker receives nothing. This is evidenced by local mosquitto_pub/sub working to the same remote broker from the command line and netstat showing a CLOSE_WAIT connection to the broker with one byte in the receive queue.

Not being a python expert but trying to read the paho docs I think there should be some more error checking somewhere and maybe an on_disconnect callback defined. But I am guessing.

[pgalbavy]

Ran this with a subscribe in the background, ended up stopped after 20 hours:

Jul 17 14:16:15 leaf/status Connecting to MQTT host emonpi
Jul 17 14:16:15 leaf/status MQTT connected
Jul 17 14:16:31 leaf/status/last_updated 2018/07/17 13:58
Jul 17 14:16:32 leaf/status/battery_percent 96.25
Jul 17 14:16:33 leaf/status/charging_status NOT_CHARGING
...
Jul 18 10:51:24 leaf/status/last_updated 2018/07/18 09:58
Jul 18 10:51:25 leaf/status/battery_percent 92.0833333333
Jul 18 10:51:26 leaf/status/charging_status NOT_CHARGING
Jul 18 10:51:27 leaf/status/raw { _[removed for brevity]_ }
Jul 18 10:51:28 leaf/status/connected Yes

The socket is then dead:

pi@pizero:~ $ sudo netstat -anp |grep 1883
tcp        0      0 192.168.234.41:33926    192.168.234.18:1883     ESTABLISHED 435/node-red
tcp        1      0 192.168.234.41:57881    192.168.234.18:1883     CLOSE_WAIT  12447/python
tcp        0      0 192.168.234.41:37194    192.168.234.18:1883     ESTABLISHED 12438/mosquitto_sub

I need to go learn how to turn on debug logging now.

Edit: I have speculatively added an on_disconnect callback handler to see if is catches it (and it calls client.reconnect() )

[pgalbavy]

More data: Installing on an i7 based ubuntu server (my media server, at home) and all is well once I changed the BASE_URL in the pywings code. No hangs. Something on the RPi/Debian side I guess.

[glynhudson]

MM interesting. I also ran this on a RasPi, seemed to work ok but the nissan API was very unreliable. Is there any difference in python version between RasPi and ubuntu?

[glynhudson]

I changed the BASE_URL in the pywings code

What did you change it to?

[pgalbavy]

The BASE_URL in pycarwings.py from:

BASE_URL = "https://gdcportalgw.its-mo.com/gworchest_160803A/gdc/"

to

BASE_URL = "https://gdcportalgw.its-mo.com/api_v180117_NE/gdc/"

But I suspect it's regional...

Getting you version numbers but the RPi is actually down at the moment. Which is odd.

[glynhudson]

Ah yes, it looks like since May 18 we have a new API for the UK https://github.com/jdhorne/pycarwings2/issues/35

Best use this version of pycarwings it has the new BASE_URL merged https://github.com/gym22/pycarwings2

This fixed it for me:

pip uninstall pycarwings2
pip install git+https://github.com/gym22/pycarwings2.git

I'll add a note to the readme for for the UK users to use this version.

[pgalbavy]

FWIW The version running on the ubuntu server also hung (but still logging) it just took longer. Feels like a resource leak and there is more of whatever on the bigger server. It's not memory.

[pgalbavy]

One more guess here; It seems to hang every time I try to publish into it, namely to request an "update". It sees it, reacts and publishes the request to Nissan and the response to the broker (which I see on another host) but then the TCP socket to the broker goes CLOSE_WAIT. I wonder if this is what happens on the RPi too, as I think I have a timed injection in Node-Red to query the car... More debug this weekend maybe. Paho MQTT seems fragile is used uncarefully.