no data receive in mqtt

[m4dm4rtig4n]

Hi,

I don't receive ny message in mqtt while I have a lot of data going back from my sensors.

Any idea why ?

Thx

Debug log :

2020-04-27 08:12:09,113 INFO: Logging to file: /enoceanmqtt/../enoceanmqtt.log
2020-04-27 08:12:09,113 INFO: Logging debug to console
2020-04-27 08:12:09,113 INFO: Loading config file /config/enoceanmqtt.conf
2020-04-27 08:12:09,114 DEBUG: Created sensor: {'name': 'enocean/temp_hum_chloe', 'address': 27554380, 'rorg': 165, 'func': 4, 'type': 1, 'persistent': 1}
2020-04-27 08:12:09,114 DEBUG: Created sensor: {'name': 'enocean/temp_hum_parent', 'address': 27559341, 'rorg': 165, 'func': 4, 'type': 1, 'persistent': 1}
2020-04-27 08:12:09,114 DEBUG: Created sensor: {'name': 'enocean/temp_hum_simon', 'address': 27550576, 'rorg': 165, 'func': 4, 'type': 1, 'persistent': 1}
2020-04-27 08:12:09,114 DEBUG: Created sensor: {'name': 'enocean/temp_hum_cuisine', 'address': 27563857, 'rorg': 165, 'func': 4, 'type': 1, 'persistent': 1}
2020-04-27 08:12:09,114 DEBUG: Created sensor: {'name': 'enocean/temp_hum_hall', 'address': 27558244, 'rorg': 165, 'func': 4, 'type': 1, 'persistent': 1}
2020-04-27 08:12:09,115 DEBUG: Created sensor: {'name': 'enocean/temp_hum_sejour', 'address': 27557611, 'rorg': 165, 'func': 4, 'type': 1, 'persistent': 1}
2020-04-27 08:12:09,115 DEBUG: Global config: {'enocean_port': '/dev/ttyUSB0', 'log_packets': '1', 'mqtt_host': 'URI.DOMAIN.COM', 'mqtt_port': '1883', 'mqtt_client_id': 'enocean', 'mqtt_keepalive': '0', 'mqtt_prefix': 'enocean/', 'mqtt_user': 'XXXXX', 'mqtt_pwd': 'XXXXX', 'mqtt_ssl': 'false', 'mqtt_ssl_insecure': 'true'}
2020-04-27 08:12:09,115 INFO: Authenticating: mqtt
2020-04-27 08:12:09,115 INFO: Enabling SSL
2020-04-27 08:12:09,141 WARNING: Disabling SSL certificate verification
2020-04-27 08:12:09,156 INFO: SerialCommunicator started
2020-04-27 08:12:09,156 WARNING: Replacing Packet.optional with default value.
2020-04-27 08:12:09,257 INFO: Sending packet
2020-04-27 08:12:09,257 DEBUG: 0x05 ['0x8'] [] OrderedDict()
2020-04-27 08:12:09,357 DEBUG: 0x02 ['0x0', '0xff', '0x8f', '0xdf', '0x80'] ['0xa'] OrderedDict()
2020-04-27 08:12:09,357 INFO: got response packet: OK 
2020-04-27 08:12:16,234 DEBUG: 01:A4:97:51->FF:FF:FF:FF (-65 dBm): 0x01 ['0xa5', '0x0', '0x65', '0x80', '0xa', '0x1', '0xa4', '0x97', '0x51', '0x0'] ['0x0', '0xff', '0xff', '0xff', '0xff', '0x41', '0x0'] OrderedDict()
2020-04-27 08:12:16,234 INFO: received: 01:A4:97:51->FF:FF:FF:FF (-65 dBm): 0x01 ['0xa5', '0x0', '0x65', '0x80', '0xa', '0x1', '0xa4', '0x97', '0x51', '0x0'] ['0x0', '0xff', '0xff', '0xff', '0xff', '0x41', '0x0'] OrderedDict()
2020-04-27 08:12:16,235 DEBUG: enocean/temp_hum_cuisine: HUM (Rel. Humidity (linear))=40.400000000000006 % 
2020-04-27 08:12:16,235 DEBUG: enocean/temp_hum_cuisine: TMP (Temperature (linear))=20.48 °C
2020-04-27 08:12:16,235 DEBUG: enocean/temp_hum_cuisine: TSN (Availability of the Temperature Sensor)=available 
....

My config :

## the general section defines parameter for the mqtt broker and the enocean interface
[CONFIG]
enocean_port    = /dev/ttyUSB0
log_packets     = 1

mqtt_host       = URI.DOMAIN.COM
mqtt_port       = 1883
mqtt_client_id  = enocean
mqtt_keepalive  = 0

## the prefix is used for the mqtt value names; this is extended by the sensor name
mqtt_prefix     = enocean/

## publish received packets as single MQTT message with a JSON payload
# mqtt_publish_json = true

## optionally also set mqtt_user and mqtt_pwd
mqtt_user       = XXXXXX
mqtt_pwd        = "XXXXXX"

## enable SSL on MQTT connection
## Ensure that mqtt_host matches one of the hostnames contained in the broker's
## certificate, otherwise the client will refuse to connect.
##
## mqtt_ssl_ca_certs: CA certificates to be treated as trusted. Required if
##     the MQTT broker is configured with a self-signed certificate.
## mqtt_ssl_certfile, mqtt_ssl_keyfile: Client certificate and private key.
##     Only required if the broker requires clients to present a certificate.
## mqtt_ssl_insecure: Disable verification of the broker's certificate.
##     WARNING: do NOT use on production systems as this is insecure!
##
mqtt_ssl          = false
# mqtt_ssl_ca_certs = /path/CA_files_merged.pem
# mqtt_ssl_certfile = /path/client_cert.pem
# mqtt_ssl_keyfile  = /path/client_key.pem
mqtt_ssl_insecure = true

## all other sections define the sensors to monitor

[temp_hum_chloe]
address         = 0x01A4724C
rorg            = 0xa5
func            = 0x04
type            = 0x01
persistent      = 1

[temp_hum_parent]
address         = 0x01A485AD
rorg            = 0xa5
func            = 0x04
type            = 0x01
persistent      = 1

[temp_hum_simon]
address         = 0x01A46370
rorg            = 0xa5
func            = 0x04
type            = 0x01
persistent      = 1

[temp_hum_cuisine]
address         = 0x01A49751
rorg            = 0xa5
func            = 0x04
type            = 0x01
persistent      = 1

[temp_hum_hall]
address         = 0x01A48164
rorg            = 0xa5
func            = 0x04
type            = 0x01
persistent      = 1

[temp_hum_sejour]
address         = 0x01A47EEB
rorg            = 0xa5
func            = 0x04
type            = 0x01
persistent      = 1

[romor]

It seems like that your enocean interface is running well but the connection to the MQTT broker is not established. On successful connection you will get a log entry like this, which is not present in your log:

2018-10-30 23:55:35,113 INFO: Succesfully connected to MQTT broker.

I suggest testing the mqtt connection first, e.g. using the mosquitto command line tool:

mosquitto_pub -h uri.domain.com -p 1883 -u xxxx -P xxxx -t 'enocean/temp_hum_cuisine/TMP' -m 20

Is this working?

[m4dm4rtig4n]

Hi @romor ,

It's working.

I have install mosquitto client on docker :

# docker run --rm -it --device=/dev/ttyUSB0 -v /opt/docker/data/enocean:/config --entrypoint sh flopon/enocean-mqtt:latest
> apk --no-cache add mosquitto mosquitto-client
> mosquitto_pub -h URI.DOMAIN.FR -p 1883 -u XXXX -P XXXX -t 'enocean/temp_hum_cuisine/TMP' -m 20

For information I override entrypoint :

python /usr/local/bin/enoceanmqtt /config/enoceanmqtt.conf --debug

Ansible deployment configuration :

- docker_container:
    name: "enocean"
    image: flopon/enocean-mqtt:latest
    state: started
    restart: yes
    restart_policy: unless-stopped
    volumes:
     - /opt/docker/data/enocean:/config
    devices:
     - /dev/ttyUSB0:/dev/ttyUSB0
    entrypoint:
     - python
     - /usr/local/bin/enoceanmqtt
     - /config/enoceanmqtt.conf
     - --debug

[crowbarz]

From your debug it looks like SSL is enabled. Try commenting out the line:

mqtt_ssl = false

in your configuration. The way the test is coded right now, if that option is present (irrespective of what the value is set to) then the client will try to connect via SSL, which does not appear to be desired in your case.

[m4dm4rtig4n]

Same :/

2020-04-27 12:41:22,001 INFO: Authenticating: mqtt
2020-04-27 12:41:22,017 INFO: SerialCommunicator started
2020-04-27 12:41:22,017 WARNING: Replacing Packet.optional with default value.
2020-04-27 12:41:22,118 INFO: Sending packet
2020-04-27 12:41:22,118 DEBUG: 0x05 ['0x8'] [] OrderedDict()
2020-04-27 12:41:22,218 DEBUG: 0x02 ['0x0', '0xff', '0x8f', '0xdf', '0x80'] ['0xa'] OrderedDict()
2020-04-27 12:41:22,218 INFO: got response packet: OK
2020-04-27 12:41:42,965 DEBUG: 01:A4:97:51->FF:FF:FF:FF (-64 dBm): 0x01 ['0xa5', '0x0', '0x61', '0x8b', '0xa', '0x1', '0xa4', '0x97', '0x51', '0x0'] ['0x0', '0xff', '0xff', '0xff', '0xff', '0x40', '0x0'] OrderedDict()
2020-04-27 12:41:42,965 INFO: received: 01:A4:97:51->FF:FF:FF:FF (-64 dBm): 0x01 ['0xa5', '0x0', '0x61', '0x8b', '0xa', '0x1', '0xa4', '0x97', '0x51', '0x0'] ['0x0', '0xff', '0xff', '0xff', '0xff', '0x40', '0x0'] OrderedDict()
2020-04-27 12:41:42,966 DEBUG: enocean/temp_hum_cuisine: HUM (Rel. Humidity (linear))=38.800000000000004 %
2020-04-27 12:41:42,966 DEBUG: enocean/temp_hum_cuisine: TMP (Temperature (linear))=22.240000000000002 °C
2020-04-27 12:41:42,966 DEBUG: enocean/temp_hum_cuisine: TSN (Availability of the Temperature Sensor)=available 

I have try with :

• mqtt_ssl = true
• mqtt_ssl = false

• mqtt_ssl = ...

I never have :'( INFO: Succesfully connected to MQTT broker.

[crowbarz]

Do you have the MQTT password surrounded by quotes? Try removing the quotes. I believe ConfigParser returns the value after the = verbatim, including any quotes, which the client then tries to use as a password. You shouldn't need to use quotes even if the password has spaces in it, although I am not sure it will work if the password starts and/or ends with spaces.

[m4dm4rtig4n]

I have try :

• with quotes
• without quote.
• My default password have special charact, i have remove special charact.
• change mqtt_host with hostname to ip address.

And it still doesn't work :(

I have zwave2mqtt config with the same configuration and it's work perfectly :(

[crowbarz]

It's strange that you also don't get the warning message that is printed if the MQTT server disconnects due to either bad protocol or bad password:

Unexpectedly disconnected from MQTT

The lack of that message would normally imply that the MQTT server is not responding to the connection requests due to a firewall and/or a network routing issue.

In order to get more insight into the problem, you could try enabling the Paho MQTT client level debugging. That might give you more clues as to why you MQTT client is not connecting. To enable client logging, add the following line to the installed copy of communicator.py, immediately before line 56:

  self.mqtt.enable_logger()

On my systems running Python 3.7 this file is installed at: /usr/local/lib/enoceanmqtt/lib/python3.7/site-packages/enoceanmqtt-0.1.0-py3.7.egg/enoceanmqtt/communicator.py. I'm not familiar with the docker image you're using as I run it in a venv on my system.

Alternatively you could analyse a packet capture either at the client or the server end.

[m4dm4rtig4n]

I have add line to /enoceanmqtt/communicator.py in docker and now i have :

2020-04-27 14:32:18,777 DEBUG: Connection failed, retrying
2020-04-27 14:32:22,781 DEBUG: Connection failed, retrying
2020-04-27 14:32:30,783 DEBUG: Connection failed, retrying

I can increase verbosity?

[crowbarz]

That's as much verbosity as you will get from the MQTT client. Either it could not resolve the broker name, or could not establish the TCP connection to the broker because the broker rejected the connection (there are only 4 and 8 seconds between connection attempts.) I think it gives you the unexpected disconnect message mentioned above rather than the retry message if you are trying to connect to the broker via SSL, but it is not configured for SSL (and vice versa).

Make sure that you have commented out the mqtt_ssl line as you don't seem to be using SSL with zwave2mqtt, and that Enabling SSL doesn't appear in the logs.

Also check that MQTT and firewall ACLs are allowing connections.

You could also add another line just before the connect_async() line to dump the connection hostname and port to verify that the config is being read correctly:

logging.debug("Connecting to host " + self.conf['mqtt_host'] + ", port " + str(mqtt_port) + ", keepalive " + str(mqtt_keepalive))

Beyond that I can only suggest exec into the container and double checking that you can resolve the broker name via ping uri.domain.com and that you can manually connect to the MQTT broker port via nc -vv uri.domain.com 1883. Though, as you've already tested using the mosquitto client from within this container, this doesn't seem to be the problem. As a last resort, tcpdump -i eth0 host uri.domain.com and tcp port 1883 (assuming a relatively default docker setup and with relevant dependent packages installed) from within the container should show you at least whether any packets are leaving the container and heading towards the MQTT broker, and whether the broker (or firewall) is rejecting your connections.

[crowbarz]

Interesting, I noticed that setting the keepalive to 0, the value in enoceanmqtt.conf.sample, doesn't work (anymore?). Can you try changing it to the Paho MQTT client default of 60?

[m4dm4rtig4n]

Interesting, I noticed that setting the keepalive to 0, the value in enoceanmqtt.conf.sample, doesn't work (anymore?). Can you try changing it to the Paho MQTT client default of 60?

Umh, when i have set to keepalive at 60 => Succesfully connected to MQTT broker.

Thx for your helping ;)

[m4dm4rtig4n]

It's work but now i'm regularly disconnect :/

2020-04-28 07:31:26,092 WARNING: Unexpectedly disconnected from MQTT broker: 1                                                                                                                                       
2020-04-28 07:31:27,096 INFO: Succesfully connected to MQTT broker.                                                                                                                                                  
2020-04-28 07:31:42,103 WARNING: Unexpectedly disconnected from MQTT broker: 1                                                                                                                                       
2020-04-28 07:31:43,106 INFO: Succesfully connected to MQTT broker.                                                                                                                                                  
2020-04-28 07:31:58,113 WARNING: Unexpectedly disconnected from MQTT broker: 1                                                                                                                                       
2020-04-28 07:31:59,123 INFO: Succesfully connected to MQTT broker.                                                                                                                                                  
2020-04-28 07:32:14,130 WARNING: Unexpectedly disconnected from MQTT broker: 1                                                                                                                                       
2020-04-28 07:32:15,133 INFO: Succesfully connected to MQTT broker.  

Normal ?

[romor]

No, not normal.

It disconnects after 15 seconds. Is this the keepalive timeout you set?

[m4dm4rtig4n]

I set keepalive to 60 and mosquitto have default conf (just add passwd file)

[crowbarz]

I'd suggest re-enabling MQTT client debugging and see what MQTT packets are being sent and received, now that you have established a TCP connection to the broker. (I intend to create a PR to enable this through either a command line switch or a config file option.)

BTW the keepalive=0 issue has been reported upstream at eclipse/paho.mqtt.python#473.

[m4dm4rtig4n]

I have found, i have 2 client with same id :) And it is impossible to have 2 clients with the same id at the same time :)

Thx for your help :)