Bridging to AWS IoT keeps failing with "Socket error on client local.xyz.awsiot, disconnecting."

[mludvig]

I'm trying to bridge a local broker to two remote brokers - one is Mosquitto based and the second one is AWS IoT. For some reason the connection to AWS IoT keeps closing all the time with Socket error on client local.xyz.awsiot, disconnecting.. The Mosquitto connection works well.

Here is my sample mosquitto.conf ...

### Bridge to test.mosquitto.org
connection test_mosquitto_org
address test.mosquitto.org

topic xyz/# out

bridge_protocol_version mqttv311

cleansession true
start_type automatic
notifications false
log_type all

### Bridge to AWS IOT
# AWS IoT endpoint, use AWS CLI 'aws iot describe-endpoint'
connection awsiot
address abcdefghjkl.iot.ap-southeast-2.amazonaws.com:8883

topic xyz/# both

bridge_protocol_version mqttv311
bridge_insecure false

cleansession true
start_type automatic
notifications false
log_type all

bridge_cafile   /etc/mosquitto/certs/rootCA.pem
bridge_certfile /etc/mosquitto/certs/cert.crt
bridge_keyfile  /etc/mosquitto/certs/private.key

When I start mosquitto with this config it seems to connect to both endpoints:

1572821579: mosquitto version 1.6.7 starting
1572821579: Config loaded from bridge.conf.
1572821579: Opening ipv4 listen socket on port 1883.
1572821579: Opening ipv6 listen socket on port 1883.
1572821579: Bridge local.xyz.test_mosquitto_org doing local SUBSCRIBE on topic xyz/#
1572821579: Connecting bridge (step 1) test_mosquitto_org (test.mosquitto.org:1883)
1572821579: Bridge local.xyz.awsiot doing local SUBSCRIBE on topic xyz/#
1572821579: Connecting bridge (step 1) awsiot (abcdefghjkl.iot.ap-southeast-2.amazonaws.com:8883)
1572821580: Connecting bridge (step 2) test_mosquitto_org (test.mosquitto.org:1883)
1572821580: Connecting bridge (step 2) awsiot (abcdefghjkl.iot.ap-southeast-2.amazonaws.com:8883)
1572821580: Bridge xyz.awsiot sending CONNECT
1572821580: Bridge xyz.test_mosquitto_org sending CONNECT
1572821581: Received CONNACK on connection local.xyz.test_mosquitto_org.
1572821581: Bridge local.xyz.test_mosquitto_org sending UNSUBSCRIBE (Mid: 1, Topic: xyz/#)
1572821581: Received CONNACK on connection local.xyz.awsiot.
1572821581: Bridge local.xyz.awsiot sending SUBSCRIBE (Mid: 1, Topic: xyz/#, QoS: 0, Options: 0x00)
1572821581: Received SUBACK from local.xyz.awsiot
1572821581: Received UNSUBACK from local.xyz.test_mosquitto_org

However as soon as I publish a message to the local broker the AWS IoT connection starts crashing:

1572821604: New connection from 127.0.0.1 on port 18833.
1572821604: New client connected from 127.0.0.1 as mosq-f2AcX35ovhb4OmJOhC (p2, c1, k60).
1572821604: No will message specified.
1572821604: Sending CONNACK to mosq-f2AcX35ovhb4OmJOhC (0, 0)
1572821604: Received PUBLISH from mosq-f2AcX35ovhb4OmJOhC (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))
1572821604: Sending PUBLISH to local.xyz.test_mosquitto_org (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))
1572821604: Sending PUBLISH to local.xyz.awsiot (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))
1572821604: Received DISCONNECT from mosq-f2AcX35ovhb4OmJOhC
1572821604: Client mosq-f2AcX35ovhb4OmJOhC disconnected.
1572821604: Socket error on client local.xyz.awsiot, disconnecting.

1572821610: Bridge local.xyz.awsiot doing local SUBSCRIBE on topic xyz/#
1572821610: Connecting bridge (step 1) awsiot (abcdefghjkl.iot.ap-southeast-2.amazonaws.com:8883)
1572821610: Connecting bridge (step 2) awsiot (abcdefghjkl.iot.ap-southeast-2.amazonaws.com:8883)
1572821610: Bridge xyz.awsiot sending CONNECT
1572821610: Received CONNACK on connection local.xyz.awsiot.
1572821610: Bridge local.xyz.awsiot sending SUBSCRIBE (Mid: 2, Topic: xyz/#, QoS: 0, Options: 0x00)
1572821610: Sending PUBLISH to local.xyz.awsiot (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))
1572821610: Received SUBACK from local.xyz.awsiot
1572821610: Socket error on client local.xyz.awsiot, disconnecting.
[... and so on, it keeps re-connecting and crashing ...]

I'm pretty sure it's not the AWS IoT configuration or the SSL certificates because with mosquitto_pub with the same certs I can submit messages to AWS IoT.

I'm using the latest mosquitto 1.6.7 on Linux Mint 19.1 and on Ubuntu 18.04, getting the same issue.

[karlp]

you look like you're missing "try_private false"

[mludvig]

Thanks, just tried both try_private false and try_private true and it doesn't help. Still getting Socket error ...

[ckrey]

AWS does not support Retained messages but you are publishing a retained message:

1572821604: Received PUBLISH from mosq-f2AcX35ovhb4OmJOhC (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))
1572821604: Sending PUBLISH to local.xyz.test_mosquitto_org (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))
1572821604: Sending PUBLISH to local.xyz.awsiot (d0, q0, r1, m0, 'xyz/test', ... (73 bytes))

https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html

[ethsonliu]

This is same with Google IoT, see https://cloud.google.com/iot/docs/requirements#mqtt_bridge.

[mludvig]

Thanks you’re right, publishing without retain flag works. Unfortunately we need the messages retained on the primary broker.

Is there any way to clear the flag before forwarding it to AWS or Google?

[ralight]

@mludvig Not in released versions, but this does what you need: https://github.com/eclipse/mosquitto/commit/2af260ba58afc4055bbbc3710f8662dcec92ff22

[Kajvdh]

@ralight seems like this commit was never merged in the later versions after it. Do you know if there is at this time a workaround?

[AikHong]

Hi For me, the problem is that I used the same client ID for different devices script, hence it keep disconnecting. I gave an unique client ID for each of my device and the problem disappear. Regards  Sent from Mail for Windows 10 From: Kaj Van der HallenSent: Friday, 16 October 2020 3:37 PMTo: eclipse/mosquittoCc: AikHong; CommentSubject: Re: [eclipse/mosquitto] Bridging to AWS IoT keeps failing with "Socket error on client local.xyz.awsiot, disconnecting." (#1486) @ralight seems like this commit was never merged in the later versions after it. Do you know if there is at this time a workaround?—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe. 

[jonahtomrobinson]

@ralight seems like this commit was never merged in the later versions after it. Do you know if there is at this time a workaround?

I'm also running into this issue. It would be really useful to have this forgotten feature integrated.

[meierthomas]

I just stumbled upon this as well. Given I'm not a developer and there's no released version of this means I'd have to go back to the actual devices to manage retain.

The option of turning this off completely or even by topic would be a great addition to the broker. However I'm not sure what's involved or whether there is demand for this.

@mludvig are you saying bridging your main broker to (remote) Mosquitto works even with messages containing retain=yes? If yes, my workaround may have to be to run an instance of Mosquitto on say AWS or check out the IBM Watson IoT service.

[pawel-smth]

As the topic is still open, posting for those who will be looking for similar info about "retain".

Changelog for v2.0.0 (https://mosquitto.org/ChangeLog.txt) mentions: "- Add bridge_outgoing_retain option, to allow outgoing messages from a bridge to have the retain bit completely disabled, which is useful when bridging to e.g. Amazon or Google."