Cluster-wide MQTT client id tracking

[dcorbacho]

Uses a ra cluster to keep the client id tracking information - in the state of the ra machine.

If nodes are decommissioned from the RMQ cluster, the command decommission_mqtt_node must be invoked first to disconnect the clients on that node and remove the node from the ra cluster.

[#135330629]

Types of Changes

• [ ] Bugfix (non-breaking change which fixes issue #NNNN)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation (correction or otherwise)
• [ ] Cosmetics (whitespace, appearance)

Checklist

• [x] I have read the CONTRIBUTING.md document
• [x] I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• [x] All tests pass locally with my changes
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added necessary documentation (if appropriate)
• [ ] Any dependent changes have been merged and published in related repositories

[michaelklishin]

@dcorbacho I cannot find the the mqtt_node module used in this code. Without it, a node cannot be started. Was everything committed and pushed?

[dcorbacho]

@michaelklishin both mqtt_node and mqtt_machine were missing. Fixed now

[michaelklishin]

Relevant log entries:

2019-06-03 15:17:05.044 [info] <0.689.0> accepting MQTT connection <0.689.0> ([::1]:57714 -> [::1]:1883)
2019-06-03 15:17:06.116 [debug] <0.718.0> MQTT accepting TCP connection <0.718.0> ([::1]:57717 -> [::1]:1883)
2019-06-03 15:17:06.116 [debug] <0.718.0> Received a CONNECT, client ID: "client-1" (expanded to "client-1"), username: undefined, clean session: true, protocol version: 4, keepalive: 60
2019-06-03 15:17:06.133 [warning] <0.689.0> MQTT disconnecting client "[::1]:57714 -> [::1]:1883" with duplicate id 'client-1'
2019-06-03 15:17:06.134 [info] <0.718.0> accepting MQTT connection <0.718.0> ([::1]:57717 -> [::1]:1883)
2019-06-03 15:17:07.144 [debug] <0.747.0> MQTT accepting TCP connection <0.747.0> ([::1]:57718 -> [::1]:1883)
2019-06-03 15:17:07.144 [debug] <0.747.0> Received a CONNECT, client ID: "client-1" (expanded to "client-1"), username: undefined, clean session: true, protocol version: 4, keepalive: 60
2019-06-03 15:17:07.152 [info] <0.747.0> accepting MQTT connection <0.747.0> ([::1]:57718 -> [::1]:1883)```

[michaelklishin]

Additional logging leads me to the following hypothesis:

1. Client A connects, client ID map is updated and now contains a new entry with A's pid
2. Client B connects, client ID map is updated and now contains an updated entry with B's pid
3. B disconnects, rabbit_mqtt_collector:unregister/2 deletes the entry in the client IDs map
4. The map is empty, even though one client still has to be connected
5. Client A reconnects and duplicate client ID is not detected by rabbit_mqtt_collector:register/2

[michaelklishin]

@kjnilsson not everything above is addressed but this is ready for another round.