search

jziolkowski / tdm

GUI application to discover and monitor devices flashed with https://github.com/arendst/tasmota
GNU General Public License v3.0
652 stars 84 forks source link

Use wildcards in Autodiscovery patterns? #254

Open sammyke007 opened 7 months ago

sammyke007 commented 7 months ago

Is it possible to use wildcards in the Autodiscovery patterns? I've tried *, # or %, but none of them work? My fulltopics are %prefix%/tasmota/*room*/%topic%/. I can configure each room in the pattern match, but using a wildcard could be a lot easier (like %prefix%/tasmota/*/%topic%/).

jziolkowski commented 7 months ago

That kinda makes sense. Let me test something

jziolkowski commented 7 months ago

Ok so tests are very promising! It was easier that I assumed.

Will you be able to test a TDM version from a branch?

jziolkowski commented 7 months ago

here is the branch: https://github.com/jziolkowski/tdm/tree/autodiscovery wildcard character is *, in your case it would be %prefix%/tasmota/*/%topic%/

This code also enables support for native tasmota autodetection, so some of the devices might be discovered anyway irrelevant if you have patterns configured for them.

Next, for some devices the online/offline status is randomly not updating; will be fixed also the current solution makes some topic subscriptions overlap; will be optimized

barbudor commented 7 months ago

Why not using the standard + MQTT wildcard ?

jziolkowski commented 7 months ago

I did consider that, yes. For now it's a proof of concept

jziolkowski commented 7 months ago

I've updated the branch. Now it's expected to use + the same way as you would use in MQTT topic. # is still not allowed.

sammyke007 commented 7 months ago

I'll test tomorrow! What a fast fix, tnx!

barbudor commented 7 months ago

Hi @jziolkowski

So I did some test with one of my MQTT broker where I'm using sonnaz/%topic%/%prefix%/as full topic. Not using yet tasmota Discovery on that broker (but will do soon, may be tomorrow)

I have 4 devices : teleinfo, gateway, doorctr-garage1 and skt-gateway

With the latest version and using pattern to +/%topic%/%prefix%/ I had only 3 devices detected. teleinfo isn't But despite detection, none of the columns on any tab get filled. Module renames on "Fetching module name..." Openning a console and sending a command works fine and response is provided.

image

Logs:

$ python tdmgr.py --debug --config-location /c/Users/Jean-Michel/AppData/Roaming/tdm/flespi/ --log-location  /c/Users/Jean-Michel/AppData/Roaming/tdm/flespi/
2023-12-16 23:43:44 [INFO] ### TDM START ###
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
2023-12-16 23:43:44 [INFO] MQTT: Subscribed to tasmota/discovery/+/config, tele/+/#, stat/+/#, +/tele/#, +/stat/#, +/+/tele/#, +/+/stat/#
2023-12-16 23:43:44 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/teleinfo/tele/LWT
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at cmnd/teleinfo/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/cmnd/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/cmndinfo/cmnd/FullTopic
2023-12-16 23:43:44 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/gateway/tele/LWT
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at cmnd/gateway/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/cmnd/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/gateway/cmnd/FullTopic
2023-12-16 23:43:44 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/doorctr-garage1/tele/LWT
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at cmnd/doorctr-garage1/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/cmnd/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/doorctr-garage1/cmnd/FullTopic
2023-12-16 23:43:44 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/skt-gateway/tele/LWT
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at cmnd/skt-gateway/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/cmnd/tele/FullTopic
2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/skt-gateway/cmnd/FullTopic
2023-12-16 23:43:45 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/doorctr-garage1/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-16 23:43:45 [INFO] DISCOVERY(LEGACY): Discovered topic=doorctr-garage1 with fulltopic=sonnaz/%topic%/%prefix%
2023-12-16 23:43:45 [DEBUG] DISCOVERY: Sending initial query to topic doorctr-garage1
2023-12-16 23:43:45 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/skt-gateway/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-16 23:43:45 [INFO] DISCOVERY(LEGACY): Discovered topic=skt-gateway with fulltopic=sonnaz/%topic%/%prefix%
2023-12-16 23:43:45 [DEBUG] DISCOVERY: Sending initial query to topic skt-gateway
2023-12-16 23:43:45 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/gateway/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-16 23:43:45 [INFO] DISCOVERY(LEGACY): Discovered topic=gateway with fulltopic=sonnaz/%topic%/%prefix%
2023-12-16 23:43:45 [DEBUG] DISCOVERY: Sending initial query to topic gateway

I believe the tele in teleinfo may be creating some issues.

With the version just before the last commit (when hte wildcard was *), the teleinfo device was detected but still no details retreived.

Let me know if you want more tests

barbudor commented 7 months ago

Adding a view from MQTT Explorer (note that blegateway is not a real Tasmota device so it has no LWT topic).

image

jziolkowski commented 7 months ago

Thx a lot, I'll look into it.

As for the tele in teleinfo, it's of not importance. I'm not using partial comparisons with tele, only a full string. As long as tele is a part of a word it's ok.

As for the "fetching..." it means that the incoming command responses are not being parsed correctly. Work continues.

jziolkowski commented 7 months ago

Oh wait, indeed there is a replacement.

2023-12-16 23:43:44 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/cmndinfo/cmnd/FullTopic

Need to have a look again.

jziolkowski commented 7 months ago

Ah got it. I'm replacing this part of the topic using regex matched string, not the location. Therefore it replaces all occurences in the string. Need to use start/end location in the string.

barbudor commented 7 months ago

As usual, regex save the world until they break it 🤣

sammyke007 commented 7 months ago

This noob is meanwhile looking how to compile an exe from source. It's easier to use python I guess?

jziolkowski commented 7 months ago

Yes, get the python code and run it in a virtualenv

jziolkowski commented 7 months ago

Pushed a fix.

As the old adage goes "You have a problem. You try fixing it with regex. Now you have two problems."

barbudor commented 7 months ago

Hi @jziolkowski Just pulled your latest commit Result is worse, only 2 out of 4 devices detected but teleinfois part of it. Still fetching module name...

image

2023-12-17 19:04:30 [INFO] ### TDM START ###
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
2023-12-17 19:04:30 [INFO] MQTT: Subscribed to tasmota/discovery/+/config, tele/+/#, stat/+/#, +/tele/#, +/stat/#, +/+/tele/#, +/+/stat/#
2023-12-17 19:04:30 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/teleinfo/tele/LWT
2023-12-17 19:04:30 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/teleinfo/cmnd/FullTopic
2023-12-17 19:04:30 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/gateway/tele/LWT
2023-12-17 19:04:30 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/gateway/cmnd/FullTopic
2023-12-17 19:04:30 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/doorctr-garage1/tele/LWT
2023-12-17 19:04:30 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/skt-gateway/tele/LWT
2023-12-17 19:04:30 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/teleinfo/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-17 19:04:30 [INFO] DISCOVERY(LEGACY): Discovered topic=teleinfo with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 19:04:30 [DEBUG] DISCOVERY: Sending initial query to topic teleinfo
2023-12-17 19:04:31 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/gateway/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-17 19:04:31 [INFO] DISCOVERY(LEGACY): Discovered topic=gateway with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 19:04:31 [DEBUG] DISCOVERY: Sending initial query to topic gateway
jziolkowski commented 7 months ago

Argh I omitted - in regex 🤦‍♂️

jziolkowski commented 7 months ago

Pushed

jziolkowski commented 7 months ago

another push; I reverted the regex to simply ignore chars that are not valid in topics as per mqtt spec

barbudor commented 7 months ago

Detection o fthe 4 devices is now ok but still nothing more than "fetching module name" Same result wether I use +/%topic%/%prefix%/ or the exact sonnaz/%topic%/%prefix%/ Should I have expected more ?

barbudor commented 7 months ago

I have upgraded the 4 units with a firmware including TASMOTA_DISCOVERY I don't know if it was a good idea or not, with the idea to be sure it would use the discovery and not the LWT, I removed the LWT messages from the broker:

Only 2 devices detected (strangely the ESP32 based ones), module name fetched but only that. Ohter columns remains empty

Logs: Note the [ERROR] Unable to parse Tasmota discovery message messages

2023-12-17 21:40:51 [INFO] ### TDM START ###
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
2023-12-17 21:40:51 [INFO] MQTT: Subscribed to tasmota/discovery/+/config, tele/+/#, stat/+/#, +/tele/#, +/stat/#, sonnaz/+/tele/#, sonnaz/+/stat/#
2023-12-17 21:40:51 [DEBUG] ip=IPv4Address('192.168.1.5') dn='Controleur Porte Garage 1' fn=['Controleur Porte Garage 1', None, None, None, None, None, None, None] hn='doorctr-garage1' mac='3C71BF3ADCC7' md='ESP8266' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='doorctr-garage1' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:40:51 [ERROR] Unable to parse Tasmota discovery message: {"ip":"192.168.1.5","dn":"Controleur Porte Garage 1","fn":["Controleur Porte Garage 1",null,null,null,null,null,null,null],"hn":"doorctr-garage1","mac":"3C71BF3ADCC7","md":"ESP8266","ty":0,"if":0,"ofln":"Offline","onln":"Online","state":["OFF","ON","TOGGLE","HOLD"],"sw":"13.3.0.1","t":"doorctr-garage1","ft":"sonnaz/%topic%/%prefix%","tp":["cmnd","stat","tele"],"rl":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"swc":[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],"swn":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"btn":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"so":{"4":1,"11":0,"13":0,"17":0,"20":0,"30":0,"68":0,"73":0,"82":0,"114":0,"117":0},"lk":0,"lt_st":0,"bat":0,"dslp":0,"sho":[],"sht":[],"ver":1}
2023-12-17 21:40:51 [DEBUG] ip=IPv4Address('192.168.1.6') dn='Prise Gateway' fn=['Prise Gateway', None, None, None, None, None, None, None] hn='skt-gateway' mac='2462AB34DBF0' md='Gosund SP111' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='skt-gateway' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:40:51 [ERROR] Unable to parse Tasmota discovery message: {"ip":"192.168.1.6","dn":"Prise Gateway","fn":["Prise Gateway",null,null,null,null,null,null,null],"hn":"skt-gateway","mac":"2462AB34DBF0","md":"Gosund SP111","ty":0,"if":0,"ofln":"Offline","onln":"Online","state":["OFF","ON","TOGGLE","HOLD"],"sw":"13.3.0.1","t":"skt-gateway","ft":"sonnaz/%topic%/%prefix%","tp":["cmnd","stat","tele"],"rl":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"swc":[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],"swn":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"btn":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"so":{"4":1,"11":0,"13":0,"17":0,"20":0,"30":0,"68":0,"73":0,"82":0,"114":0,"117":0},"lk":0,"lt_st":0,"bat":0,"dslp":0,"sho":[],"sht":[],"ver":1}
2023-12-17 21:40:51 [DEBUG] ip=IPv4Address('192.168.1.3') dn='Gateway (Sonnaz)' fn=['Unconfigured', None, None, None, None, None, None, None] hn='gateway' mac='B8F009CC8C5C' md='ESP32-DevKit' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='gateway' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:40:51 [INFO] DISCOVERY(NATIVE): Discovered topic=gateway with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 21:40:51 [DEBUG] ip=IPv4Address('192.168.1.4') dn='Teleinfo Sonnaz' fn=['Teleinfo Sonnaz', None, None, None, None, None, None, None] hn='teleinfo' mac='78E36D0914D0' md='miniD1-teleinfo' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='teleinfo' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 1, '73': 0, '82': 0, '114': 1, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:40:51 [INFO] DISCOVERY(NATIVE): Discovered topic=teleinfo with fulltopic=sonnaz/%topic%/%prefix%
barbudor commented 7 months ago

Restarting the devices to get the LWT back, still having the +/%topic%/%prefix%/ pattern, I get the 4 devices back but still error messages parsing the discovery payload for the ESP8266 devices

2023-12-17 21:49:30 [INFO] MQTT: Subscribed to tasmota/discovery/+/config, tele/+/#, stat/+/#, +/tele/#, +/stat/#, +/+/tele/#, +/+/stat/#
2023-12-17 21:49:30 [DEBUG] ip=IPv4Address('192.168.1.5') dn='Controleur Porte Garage 1' fn=['Controleur Porte Garage 1', None, None, None, None, None, None, None] hn='doorctr-garage1' mac='3C71BF3ADCC7' md='ESP8266' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='doorctr-garage1' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:49:30 [ERROR] Unable to parse Tasmota discovery message: {"ip":"192.168.1.5","dn":"Controleur Porte Garage 1","fn":["Controleur Porte Garage 1",null,null,null,null,null,null,null],"hn":"doorctr-garage1","mac":"3C71BF3ADCC7","md":"ESP8266","ty":0,"if":0,"ofln":"Offline","onln":"Online","state":["OFF","ON","TOGGLE","HOLD"],"sw":"13.3.0.1","t":"doorctr-garage1","ft":"sonnaz/%topic%/%prefix%","tp":["cmnd","stat","tele"],"rl":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"swc":[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],"swn":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"btn":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"so":{"4":1,"11":0,"13":0,"17":0,"20":0,"30":0,"68":0,"73":0,"82":0,"114":0,"117":0},"lk":0,"lt_st":0,"bat":0,"dslp":0,"sho":[],"sht":[],"ver":1}
2023-12-17 21:49:30 [DEBUG] ip=IPv4Address('192.168.1.6') dn='Prise Gateway' fn=['Prise Gateway', None, None, None, None, None, None, None] hn='skt-gateway' mac='2462AB34DBF0' md='Gosund SP111' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='skt-gateway' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:49:30 [ERROR] Unable to parse Tasmota discovery message: {"ip":"192.168.1.6","dn":"Prise Gateway","fn":["Prise Gateway",null,null,null,null,null,null,null],"hn":"skt-gateway","mac":"2462AB34DBF0","md":"Gosund SP111","ty":0,"if":0,"ofln":"Offline","onln":"Online","state":["OFF","ON","TOGGLE","HOLD"],"sw":"13.3.0.1","t":"skt-gateway","ft":"sonnaz/%topic%/%prefix%","tp":["cmnd","stat","tele"],"rl":[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"swc":[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],"swn":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"btn":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"so":{"4":1,"11":0,"13":0,"17":0,"20":0,"30":0,"68":0,"73":0,"82":0,"114":0,"117":0},"lk":0,"lt_st":0,"bat":0,"dslp":0,"sho":[],"sht":[],"ver":1}
2023-12-17 21:49:30 [DEBUG] ip=IPv4Address('192.168.1.3') dn='Gateway (Sonnaz)' fn=['Unconfigured', None, None, None, None, None, None, None] hn='gateway' mac='B8F009CC8C5C' md='ESP32-DevKit' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='gateway' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:49:30 [INFO] DISCOVERY(NATIVE): Discovered topic=gateway with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 21:49:30 [DEBUG] ip=IPv4Address('192.168.1.4') dn='Teleinfo Sonnaz' fn=['Teleinfo Sonnaz', None, None, None, None, None, None, None] hn='teleinfo' mac='78E36D0914D0' md='miniD1-teleinfo' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='teleinfo' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 1, '73': 0, '82': 0, '114': 1, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
2023-12-17 21:49:30 [INFO] DISCOVERY(NATIVE): Discovered topic=teleinfo with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 21:49:30 [DEBUG] MQTT: LWT message for teleinfo: Online
2023-12-17 21:49:30 [DEBUG] MQTT: LWT message for gateway: Online
2023-12-17 21:49:30 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/doorctr-garage1/tele/LWT
2023-12-17 21:49:30 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/doorctr-garage1/cmnd/FullTopic
2023-12-17 21:49:30 [INFO] DISCOVERY(LEGACY): LWT from an unknown device sonnaz/skt-gateway/tele/LWT
2023-12-17 21:49:30 [DEBUG] DISCOVERY(LEGACY): Asking an unknown device for FullTopic at sonnaz/skt-gateway/cmnd/FullTopic
2023-12-17 21:49:30 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/doorctr-garage1/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-17 21:49:30 [INFO] DISCOVERY(LEGACY): Discovered topic=doorctr-garage1 with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 21:49:30 [DEBUG] DISCOVERY: Sending initial query to topic doorctr-garage1
2023-12-17 21:49:31 [DEBUG] DISCOVERY(LEGACY): topic sonnaz/skt-gateway/stat/FULLTOPIC is matched by fulltopic sonnaz/%topic%/%prefix%
2023-12-17 21:49:31 [INFO] DISCOVERY(LEGACY): Discovered topic=skt-gateway with fulltopic=sonnaz/%topic%/%prefix%
2023-12-17 21:49:31 [DEBUG] DISCOVERY: Sending initial query to topic skt-gateway
jziolkowski commented 7 months ago

Legacy discovery won't work without retained LWT.

As for the fetching: I don't see this issue.

Also I'm surprised you now have the same errors parsing some messages as I do have. I expected it to be somehow related with my oddly behaving broker, but not. Hmm.

jziolkowski commented 7 months ago

From your earlier posts I see that you have SetOption4 enabled (the response is coming from the /<cmnd name> topic, not /RESULT; I'll look into it.

barbudor commented 7 months ago

I don't see any differences in the discovery messages from ESP32 or ESP8266 except the ESP8266 are the only ones wiht a dash - in both the topic and the hostname

barbudor commented 7 months ago

The broker is flespi.io

barbudor commented 7 months ago

you have SetOption4

Indeed but that doesn't create a problem with hte same code on my home setup with mosquitto and a standard full topic

jziolkowski commented 7 months ago

you have SetOption4

Indeed but that doesn't create a problem with hte same code on my home setup with mosquitto and a standard full topic

Indeed it should not. The result is parsed later at this point; the matching of device is done earlier. Still, I don't see why you would have a problem with that.

Either way I don't have spare test devices here, I'll be able to try and mimic your setup tomorrow.

Right now I will just add options to TDM to control which discovery is being used.

barbudor commented 7 months ago

Upgrading home to get the discovery behave strangely After their upgrade and reboot, all devices appeared as Offline Restarting TDM solved that except for 1 device that remains displayed as Offline while it is not And the logs show TDM see the Online status A 2nd restart of TDM cleared the problem

jziolkowski commented 7 months ago

Yeah there's still something off with the fulltopic parsing.

Meanwhile image

sammyke007 commented 7 months ago

Finally got around to testing. Latest pull from autodiscovery (just now) worked fine on initial start, but when I start up TDM the second time, all devices appear offline and log is full of:

2023-12-18 11:23:53 [ERROR] MQTT MESSAGE DECODE ERROR: 'utf-8' codec can't decode byte 0xf6 in position 134: invalid start byte (stat/tasmota/tuin_schuifraam/STATUS2=b...

jziolkowski commented 7 months ago

Damn it. I have a hunch that it might be related to duplicated subscriptions. I need to optimize that. Will push a fix later.

jziolkowski commented 7 months ago

It's not it. It happens randomly :/

jziolkowski commented 7 months ago

I pushed a small rework of how the initial query is performed. Pls check and let me know.

sammyke007 commented 7 months ago 
(myenv) C:\Users\verdo\OneDrive\Documenten\Python\tdm>python tdmgr.py
Traceback (most recent call last):
  File "C:\Users\verdo\OneDrive\Documenten\Python\tdm\tdmgr.py", line 26, in <module>
    from GUI.console import ConsoleWidget
  File "C:\Users\verdo\OneDrive\Documenten\Python\tdm\GUI\console.py", line 31, in <module>
    from util.commands import commands
ModuleNotFoundError: No module named 'util'

Util folder <-> util folder

After changing the util folder to lowercase, I get this in the logs:

(myenv) C:\Users\verdo\OneDrive\Documenten\Python\tdm>python tdmgr.py
2023-12-18 14:05:11 [INFO] ### TDM START ###
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
QWidget::insertAction: Attempt to insert null action
2023-12-18 14:05:12 [INFO] MQTT: Subscribed to tasmota/discovery/+/config, tele/#, stat/#, +/tele/#, +/stat/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#, tele/tasmota/#, stat/tasmota/#
Traceback (most recent call last):
  File "C:\Users\verdo\OneDrive\Documenten\Python\tdm\tdmgr.py", line 373, in mqtt_message
    device.update_property("LWT", lwt)
UnboundLocalError: local variable 'device' referenced before assignment
jziolkowski commented 7 months ago

On it.

jziolkowski commented 7 months ago

I fixed Util/util and fixed indent of the above line

barbudor commented 7 months ago 
Traceback (most recent call last):
  File "tdmgr.py", line 362, in mqtt_message
    self.mqtt.subscribe([(sub_topic, 0)])
  File "D:\Projets\repos\tdm\util\mqtt.py", line 167, in subscribe
    self.m_client.subscribe(path)
  File "C:\Users\Jean-Michel\AppData\Local\Programs\Python\Python310\lib\site-packages\paho\mqtt\client.py", line 1491, in subscribe
    raise ValueError('Invalid subscription filter.')
ValueError: Invalid subscription filter.

image

Missing a /

barbudor commented 7 months ago

The above was without parttern A first pass on that is ok, subscribe tasmota discovery which allows to infere the FT sonnaz/%topic%/%prefix% On a 2nd line it true to subscribve on the infered FT

When adding a pattern `+/%topic%/%prefix/, it include the broken subscription topic on 1st call:

image

barbudor commented 7 months ago

image and image

barbudor commented 7 months ago

But still empty columns

jziolkowski commented 7 months ago

as for the patterns missing trailing /: do they include the / in the dialog? I don't see such shenanigans in my setup, topics are replaced as expected

jziolkowski commented 7 months ago

I've added some simple tests for this parser, you can try some topics and see what happens

barbudor commented 7 months ago

In the 1st case, I didn't add the pattern It was infered by TDM by parsing the discovery topic which doesn't not include the ending / : "ft": "sonnaz/%topic%/%prefix%", And when manually adding the pattern, yes actually I forgot the ending / So indeed better to be prepared for both cases

Did you pushed anything ? The branch doesn't seems to have any new commit since 20:00

leaving for today...

jziolkowski commented 7 months ago

No, I'm back to drawing board. Yes, I will ensure that all patterns have the trailing / added or kept.

I noticed that I'm using two different matching functions in separate places, I will rework the message/topic coming from the broker to have some methods included. Most likely tomorrow.

jziolkowski commented 7 months ago

Ok, there's progress. I refactored quite a lot of how messages and devices interact from TDM point of view. That triggered more refactoring of the general processing of incoming messages.

Also I've "accidentally" fixed #95, so that's nice.

No commit at the moment, some stuff still not working.

jziolkowski commented 7 months ago

I just pushed a commit. Beware: this is a huge WIP. For now all I ask you to test is the initial discovery process and any report errors that might appear. --debug would be helpful. There are still Online/Offline issues and the data received is 99% not represented visually in TDM yet

barbudor commented 7 months ago

On the cloud broker, still some / missing as with no pattern defined, it tries to subscribe to sonnaz/+/tele+ in tdmngr.py:362

2023-12-21 21:27:18 [INFO] MQTT: Subscribed to tasmota/discovery/+/config, tele/#, stat/#, +/tele/#, +/stat/#
2023-12-21 21:27:18 [DEBUG] ip=IPv4Address('192.168.1.5') dn='Porte Garage 1 [Sonnaz]' fn=['Porte Garage 1', None, None, None, None, None, None, None] hn='doorctr-garage1' mac='3C71BF3ADCC7' md='ESP8266' ofln='Offline' onln='Online' state=['OFF', 'ON', 'TOGGLE', 'HOLD'] sw='13.3.0.1' t='doorctr-garage1' ft='sonnaz/%topic%/%prefix%' tp=TopicPrefixes(cmnd='cmnd', stat='stat', tele='tele') rl=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] swc=[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1] btn=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] so={'4': 1, '11': 0, '13': 0, '17': 0, '20': 0, '30': 0, '68': 0, '73': 0, '82': 0, '114': 0, '117': 0} lk=0 lt_st=<LightType.NONE: 0> ver=1
Traceback (most recent call last):
  File "D:\Projets\repos\tdm\tdmgr.py", line 362, in mqtt_message
    self.mqtt.subscribe([(sub_topic, 0)])
  File "D:\Projets\repos\tdm\util\mqtt.py", line 223, in subscribe
    self.m_client.subscribe(path)
  File "C:\Users\Jean-Michel\AppData\Local\Programs\Python\Python310\lib\site-packages\paho\mqtt\client.py", line 1491, in subscribe
    raise ValueError('Invalid subscription filter.')
ValueError: Invalid subscription filter.

Same with pattern defined

On my home broker (standard FT), without patterns, it starts but there is a lot of errors such as:

2023-12-21 21:32:04 [ERROR] PROCESSING: No processor for endpoint GPIOS
2023-12-21 21:32:04 [ERROR] PROCESSING: No processor for endpoint MARGINS
2023-12-21 21:32:05 [ERROR] PROCESSING: No processor for endpoint GPIOS
2023-12-21 21:32:05 [ERROR] PROCESSING: No processor for endpoint COMMAND
2023-12-21 21:32:05 [ERROR] PROCESSING: No processor for endpoint MARGINS

Devices are listed, but as Offline and all columns are empty

jziolkowski commented 7 months ago

Ah I forgot about fixing the missing trailing /

The processing errors are expected for now. Also SetOption4 is not supported yet :P (temporarily)