mosquitto doesn't restart properly after a device restart when mTLS is enforced for MQTT

[didier-wenzek]

Describe the bug

Restarting the device using a command sent over MQTT to the agent, let the device in a bad state when mTLS has been configured for MQTT.

• As requested, the device restarts
• But the agent and the mapper fails to reconnect:
  • mqtt_channel::connection: MQTT: failed to connect to broker at '192.168.1.20:8883': I/O: Connection refused (os error 111)
• And mosquitto reports an error too
  • Failed to associate persisted user main-agent with ACLs, likely due to changed ports while using a per_listener_settings configuration.

This is fixed by restarting mosquitto.

To Reproduce

1. Configure the mosquitto instance to use certificates

   For example if you have set everything up correctly, then your tedge config list should look something like this (exact paths might be different though, depending on how you set it up)

   # $ tedge config list | grep mqtt.client
   mqtt.client.host=localhost
   mqtt.client.port=8883
   mqtt.client.auth.ca_file=/etc/mosquitto/ca_certificates/ca.crt
   mqtt.client.auth.cert_file=/setup/client.crt
   mqtt.client.auth.key_file=/setup/client.key

2. Trigger a restart

$ tedge mqtt pub --retain te/device/main///cmd/restart/test-1 '{"status":"init"}'

3. Wait for the device to restart
4. Unexpectedly, tedge can no more reconnect over MQTT

$ tedge mqtt sub te/device/main///cmd/+/+ 
ERROR: I/O: Connection refused (os error 111)

Expected behavior

On reboot,

• all the tedge commands should be able to reconnect the MQTT broker
• one should not force the mosquitto to restart

Screenshots

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"]
• thin-edge.io version [e.g. 0.1.0]

Additional context

I found no related mosquitto issue, only a related question:

• Mosquitto logs "Failed to associate persisted..." every restart, even without changes

[reubenmiller]

@didier-wenzek Are you sure this is not actually a problem due to the mosquitto broker trying to bind too early to a network which isn't ready yet? e.g. https://github.com/eclipse/mosquitto/issues/2878

There are already suggestions to change the systemd target to network-online.target instead of network.target.

[didier-wenzek]

Thanks! Indeed, this has been fixed by changing the mosquitto systemd target to network-online.target instead of network.target.