Gateways stuck in loop failing MQTT TLS handshake

[firmwareguru]

I have two TTN "backer edition" gateways. They've been serviceable, useful and quite stable over the last couple of years, but both are down and out for the last 3 days now.

They seem to be stuck in a loop, trying and failing to make an MQTT connection to router.us.thethings.network (13.66.213.36).

The second LED remains in a slow blink. I have tried performing a factory reset (hold button for 5 seconds) and doing an activation again to no avail.

The following keeps repeating in the diagnostic trace output:

MAIN: Leaving state 5
MAIN: Entering state 6
INET: State change to 0
WIFI: Disabling modules
CB: Disconnect
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 7 to 1
WIFI: Entering state 3
INET: Gateway has Ethernet
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
WIFI: Enabling modules for server
WIFI: Entering state 6
SNTP: State change from 1 to 2
SNTP: State change from 2 to 6
MON: SYS Stack size: 2837
MON: TCPIP Stack size: 3761
MON: APP Stack size: 3292
MON: LoRa Stack size: 3849
MON: heap usage: 180KB (278KB), free: 159KB
CB: INET: State change to 3
INET: Ping probe
INET: Error sending probe on WiFi
INET: Ping response from PIC32INT, set as default
INET: State change to 5

MAIN: Leaving state 6
MAIN: Entering state 5
MQTT: GOT IP: 13.66.213.36
Connecting to: 13.66.213.36
MQTT: Connection Opened: Starting TLS Negotiation
MQTT: Wait for SSL Connect
MQTT: TLS Connection Negotiation Failed - Aborting

MAIN: MQTT error

Just curious, does this have anything to do with Azure's recent migration to newer TLS standards? @htdvisser

Firmware on the gateways:

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.7, commit: e9d35a30, timestamp: 1560942019
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Jun 19 2019 11:01:21

[KrishnaIyer]

@firmwareguru: You've just followed the activation procedure at https://activate.thethingsnetwork.org/ right?

[htdvisser]

Both the Account Server (from where the gateway gets its configuration) and the MQTT server use Let's Encrypt certificates, so any changes Azure made to their TLS would be unrelated.

I suspect the issue is that your gateways try to connect to the wrong MQTT address. The correct address for MQTT in US-West is bridge.us-west.thethings.network. This points to the same IP address, but that's the name on the TLS certificate our MQTT server uses.

I just confirmed that the configuration endpoint for the Kickstarter Gateway returns the correct address, so I guess you did the configuration manually? If not, what steps did you take?

[firmwareguru]

Hi guys, thanks for responding. I found some time to revisit this. I powered up the gateways yesterday and they connected and started working as if nothing happened! They even show up in the console, which they hadn't been doing for months. They had been powered off for 15 days so maybe that's something to note, but really I have done nothing different and now they're working, for the time being. They've been up for a day now.

Since my access to TTN is back I'm happy. I'll let it sit for a week and if they're still working I'll close the case.

[firmwareguru]

Still up and running! Cheers