Reboot Loop - Reboot reason: 0x10 or 0x13

htdvisser commented 6 years ago

From what I can see in the source code, that's a RESET_REASON_WDT_TIMEOUT (0x10).

Based on what I see on the forum this happens a lot.

mbarnig commented 6 years ago

In the TTN Gateway Central forum there are at least 57 identified users with a looping Gateway.

https://www.thethingsnetwork.org/forum/t/ttn-gateway-central/10976/408?u=mbarnig

johanstokking commented 6 years ago

Please provide details here, not on the forum. From here we can link to the source files, pull requests and people can watch this issue getting closed.

frbayart commented 6 years ago

Do you want this kind of details ?

gateway booting, LED1 and LED2 are ON, LED3 is blinking slowy and become ON but after few seconds LED3 switch from ON to OFF to ON and after gateway restart. ( in video it’s more easy https://youtu.be/Rb-KZeLbR-01 )

I have tried to reset from factory the gateway but I arrive on the same step every time.

Pinkfoxi commented 6 years ago

The same as all other with troubles...

From: Francois BAYART notifications@github.com Sent: Tuesday, February 6, 2018 12:48:15 PM To: TheThingsProducts/gateway Cc: Subscribed Subject: Re: [TheThingsProducts/gateway] Reboot Loop - Reboot reason: 0x10 or 0x13 (#1)

Do you want this kind of details ?

gateway booting, LED1 and LED2 are ON, LED3 is blinking slowy and become ON but after few seconds LED3 switch from ON to OFF to ON and after gateway restart. ( in video it’s more easy https://youtu.be/Rb-KZeLbR-01 )

I have tried to reset from factory the gateway but I arrive on the same step every time.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/TheThingsProducts/gateway/issues/1#issuecomment-363398968, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AdO5gYh9BocK2Z3rR9RwjqgCFliURqclks5tSDv_gaJpZM4R0NRp.

frbayart commented 6 years ago

The question is, which kind of details can help ?

grahamehorner commented 6 years ago

here’s my little video looks almost identical with LEDS and quick flash on module that a reboot loop 🤪 https://twitter.com/grahamehorner/status/957325865892958208

johanstokking commented 6 years ago

@frbayart @Pinkfoxi @grahamehorner

What could be of help for the maintainers is:

How often does this occur
Do you have logs from the gateway, and if so, can you provide them
Can you tell more about the context, i.e. which model (915, 868), ethernet or wifi connected, etc

frbayart commented 6 years ago

this occur since I have connected my gateway there are 3 days
I have tested on WiFi and ethernet ; I have also tested on a second internet connection
model TTN Gateway Backer Edition 868

At this time I have no logs from the gateway , I guess I need to connect a serial on it to get logs right ?

grahamehorner commented 6 years ago

reboot interval approx. 30 seconds
both WiFi and Ethernet inside/outside of firewall and different business ISPs
TTN Gateway Backer Edititon - European 868

OK, looks like my gateway is officially STUFFED!!! NOW!! after a few days powered off, I power on today and only the first (Power) LED is solid with the second LED flashing rapidly, and its been like this for hours!?

robdobsn commented 6 years ago

LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: RESET MODULE
LORA: ChaSNTP: State change from 0 to 0
SNTP: State change from 0 to 0

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.0, commit: 917719b9, timestamp: 1498499973
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Jun 26 2017 19:59:53
Reboot reason: 0x13
BOOT: (persisted info) 6F 72 72 65 01 03 0A 3F 15 84 13 0F B1 25 98 B0 

WIFI: Entering state 0
WIFI: Entering SCAN state 0

MAIN: Initialisation complete
LORA: Changing state from 0 to 0

MAIN: Leaving state 0
MAIN: Entering state 1
FLASH: Magic bytes found: wifi config present
FLASH: Magic bytes found: activation data present
FLASH: Magic bytes not found: no stored FOTA data present
FLASH: Loading Firmware Data
CNFG: (Firmware HASH (sha256)) FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FLASH: Loading WiFi Data
CNFG: WiFi SSID:      rdint01
CNFG: WiFi key:       golgofrinsham
CNFG: WiFi conn_type: 1
CNFG: WiFi sec_type:  4
FLASH: Loading Activation Data
CNFG: Gateway ID:         atomic14-1
CNFG: Gateway Key:        ttn-account-v2.B8leUhot-xnHx1Vbrc8s1ZNcBzYuABED6BB71BXcv7hbDV2HxTGOh19yGIOwpdxLZrw70irKTZ4T5TcvH9BjIQ
CNFG: Account Server URL: https://account.thethingsnetwork.org
CNFG: Locked:             true
CNFG: Locked first time:  false

MAIN: Leaving state 1
MAIN: Entering state 2
INET: State change to 0
LORA: Initialisation complete
LORA: Changing state from 0 to 1
WIFI: Entering state 1
ETH: IP Address: 0.0.0.0 
WIFI: Entering state 4
WIFI: Entering SCAN state 1
Scan is completed successfully
WIFI: Entering SCAN state 2
WIFI: Entering SCAN state 5
WIFI: Entering SCAN state 0
WIFI: Entering state 2
WIFI: Disabling modules
Head magic match void: trying to free an already freed block, ignore
WIFI: Entering state 3
SNTP: State change from 0 to 1
WIFI: Enabling modules for client
WIFI: Entering state 6

>WIFI: IP Address: 0.0.0.0 
CB: INET: Gateway has WiFi
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
SNTP: State change from 1 to 2
LORA: Wait init complete, waiting for application.
LORA: Changing state from 1 to 2
WIFI: IP Address: 192.168.86.68 
INET: State change to 3
INET: Ping probe
INET: Error sending probe on Eth
INET: Ping response from MRF24WN, set as default
INET: State change to 4
SNTP: State change from 2 to 3
MON: SYS Stack size: 3967
MON: heap usage: 147KB (156KB), free: 192KB
SNTP: State change from 3 to 4
SNTP: State change from 4 to 5
SNTP: State change from 5 to 6
SNTP: State change from 6 to 7
INET: State change to 5

MAIN: Leaving state 2
MAIN: Entering state 3

CNFG: Load online user config state change to 4
HTTP: Close active socket 0
CONF: ERROR REQUEST

CNFG: Downloading gateway configuration failed

CNFG: Load online user config state change to 8

CNFG: Communication ERROR

MAIN: Leaving state 3
MAIN: Entering state 2
INET: State change to 0
WIFI: Disabling modules
SNTP: State change from 7 to 8
CB: Disconnect
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 8 to 1
WIFI: Entering state 3
WIFI: Enabling modules for client
WIFI: Entering state 6
WIFI: IP Address: 0.0.0.0 
CB: INET: Gateway has WiFi
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
SNTP: State change from 1 to 2
SNTP: State change from 2 to 3
SNTP: State change from 3 to 4
SNTP: State change from 4 to 5
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
INET: State change to 3
INET: Ping probe
INET: Error sending probe on Eth
CB: INET: Ping Timeout
INET: Ping probe
INET: Error sending probe on Eth
SNTP: State change from 5 to 6
INET: Ping Timeout
INET: Ping probe
INET: Error sending probe on Eth
MON: SYS Stack size: 3961
MON: heap usage: 148KB (157KB), free: 191KB
INET: Ping Timeout
INET: State change to 1
WIFI: Disabling modules
SNTP: State change from 6 to 8
CB: Disconnect
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 8 to 1
WIFI: Entering state 3
WIFI: Entering state 4
WIFI: Entering SCAN state 1
Scan is completed successfully
WIFI: Entering SCAN state 2
WIFI: Entering SCAN state 3
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 1 to 1
WIFI: Entering SCAN state 4
WIFI: Entering SCAN state 5
WIFI: Entering SCAN state 0
WIFI: Entering state 5
WIFI: Enabling modules for server
WIFI: Entering state 6
WIFI: IP Address: 192.168.84.1 
CB: MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
INET: Trying to connect to WiFi router again (after 120 seconds)
INET: State change to 0
WIFI: Disabling modules
CB: Disconnect
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 1 to 1
WIFI: Entering state 3
WIFI: Enabling modules for client
WIFI: Entering state 6
WIFI: IP Address: 0.0.0.0 
CB: INET: Gateway has WiFi
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
SNTP: State change from 1 to 2
MON: SYS Stack size: 3961
MON: heap usage: 147KB (157KB), free: 192KB
INET: State change to 3
INET: Ping probe
INET: Error sending probe on Eth
SNTP: State change from 2 to 6
INET: Ping Timeout
INET: Ping probe
INET: Error sending probe on Eth
WIFI: IP Address: 192.168.86.68 
INET: Ping Timeout
INET: Ping probe
INET: Error sending probe on Eth
INET: Ping response from MRF24WN, set as default
INET: State change to 5

MAIN: Leaving state 2
MAIN: Entering state 3

CNFG: Load online user config state change to 4
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1289 bytes
MON: SYS Stack size: 2837
MON: heap usage: 228KB (233KB), free: 111KB
HTTP: Connection Closed
HTTP: Close active socket 1
CONF: Parsing response token: HTTP/1.1 200 OK
CONF: ROUTER URL: mqtts://bridge.eu.thethings.network:8883

CNFG: Load online user config state change to 6
FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1232 bytes
SNTP: State change from 6 to 1
HTTP: Connection Closed
HTTP: Close active socket 1

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
SNTP: State change from 1 to 2
SNTP: State change from 2 to 3
SNTP: State change from 3 to 4
SNTP: State change from 4 to 5
SNTP: State change from 5 to 6
SNTP: State change from 6 to 7
MON: SYS Stack size: 2837
MON: heap usage: 152KB (233KB), free: 187KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: RESET MODULE
LORA: ChaSNTP: State change from 0 to 0
SNTP: State change from 0 to 0

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.0, commit: 917719b9, timestamp: 1498499973
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Jun 26 2017 19:59:53
Reboot reason: 0x13
BOOT: (persisted info) 6F 72 72 65 01 03 0A 3F 15 84 13 0F B1 25 98 B0

frbayart commented 6 years ago

log from gateway console:

screenlog.0.txt

grahamehorner commented 6 years ago

@frbayart @robdobsn your gateways both reset after 3 failed attempts at configuration of the LORA module? hmm, wish my gateway got that far; not even getting anything now :'( after so may factory resets think it's dead!! :'(

grahamehorner commented 6 years ago

@frbayart @robdobsn see app_lora.c

                ErrorMessageWarning_Set(ERROR_MESSAGE_WARNING_LORA_CONFIG_FAILURE);
                SYS_DEBUG(SYS_ERROR_WARNING, "LORA: Configuration failed, retry\r\n");
                lora_config_failed_counter++;
                if(lora_config_failed_counter > 2)
                {
                    lora_config_failed_counter = 0;
                    restart_lora_configuration();
                    break;
                }

cyberjunky commented 6 years ago

Same issue here, doesn't matter if I use wifi or ethernet connection on gw, if I use my fiber Telfort Internet connection or Hotspot KPN 4G connection, with all firmware versions, even the one I compiled from github source, I got it only to register and connected once, but after a reset of settings I cannot even configure it completely it reboots before the next step, and then have all leds one, and 5 sec later off, on etc... no uart console output during this reboots, this goes on until power off. The it has console output until it start to enable/configure lora. If I don't register/configure it, its stable, after registration (during step 3) it starts rebooting, so when i downloads data from cloud.

cyberjunky commented 6 years ago

It is possible to decode this persistant info string? Reboot reason: 0x10 BOOT: (persisted info) 6F 72 72 65 01 03 9C 7E 18 6F E5 1F 80 00 4C 58

avbentem commented 6 years ago

With some new firmware that I received today and installed using an SD card, and which I think is supposed to create more logging, I get things like:

CNFG: Load online user config state change to 6
FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1232 bytes
HTTP: Connection Closed
HTTP: Close active socket 1

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
MON: SYS Stack size: 2831
MON: heap usage: 152KB (233KB), free: 187KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
MON: SYS Stack size: 2831
MON: heap usage: 152KB (233KB), free: 187KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: RESET MODULE
LORA: ChaSNTP: State change from 0 to 0
SNTP: State change from 0 to 0

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: 0407f7eb, timestamp: 1517914721
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb  6 2018 11:59:01
Reboot reason: 0x10
BOOT: (persisted info) 6F 72 72 65 01 03 A3 2C E7 16 01 49 B2 3A 50 97

As requested, I've sent several full log files to TTP today, but the above does not look much different from what was logged using the factory firmware, which is already known to TWTG/Marten from a private topic on the forum (2018-01-29) and which I'm repeating below.

To summarize my own findings from https://www.thethingsnetwork.org/forum/t/ttn-gateway-central/10976/105 and later (full log at the end):

I think I see it fetching the configuration from the internet just fine:
```
CNFG: Load online user config state change to 6
FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
MON: SYS Stack size: 2855
MON: heap usage: 227KB (233KB), free: 111KB
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1232 bytes
HTTP: Connection Closed
HTTP: Close active socket 1
```
_{As an aside: above, I am confused by HTTP: Got 1232 bytes (decimal or hexadecimal?) as fetching the logged URL https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870 in a browser gets me a much larger size in Content-Length: 3309 (decimal, excluding any HTTP headers). However, meanwhile we've seen logs of people who don't have problems, and they see the same size. So maybe the logged HTTP: Got 1232 bytes is unrelated to that URL, or the gateway gets it in some gzip encoding or some different format.}

Next, it logs LORA: Configuration failed, retry one or more times, and seems to restart with either Reboot reason: 0x13 or Reboot reason: 0x53:

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
MON: SYS Stack size: 2855
MON: heap usage: 151KB (233KB), free: 188KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: RESET MODULE
LORA: ChaSNTP: State change from 0 to 0
SNTP: State change from 0 to 0

TTN Console shows:
Before this loop, I easily got to Step 2 "Connect" and moved on to Step 3, "Configure" while still connected to gateway's AP. Next, when connected to the internet again in Step 4, "First message" the LEDs loop until 3 continous lights ("Activated") and I see some LEDs flashing on the LoRaWAN board. I've repeated a few times by holding down the reset button while powering up.
The following options make no difference; I don’t see it trying to download the firmware, maybe as it never successfully completed Step 4 of its initial activation:
A ping bridge.eu.thethings.network from the gateway itself nicely resolves to 52.169.76.203, but gets not reply, which is fine as the TTN servers don't support that:
```
ping bridge.eu.thethings.network
Ping: resolving host: bridge.eu.thethings.network
Ping: request sent to: bridge.eu.thethings.network [52.169.76.203]
Ping: done. Sent 4 requests, received 0 replies.
```
Same for account.thethingsnetwork.org. So, all this does NOT look alarming to me, and I don't think we can issue other useful commands from the gateway to investigate?
Re-seating the LoRa-board (which by itself already seemed to be seated well) did not help for me, also not when I left out the foam block altogether and applied some more downwards pressure to the board. (The length of the white plastic fasteners makes the board to not be parallel to the main PCB. It seems that there's quite some pressure on the board, but I'm not an expert.)
My LG8271 LoRa board says on its bottom side:

LG8501601782 LG-X271 REV: C
I've tried with two Dutch providers:
- For WiFi I use an XS4ALL FRITZ!Box modem. This supports IPv6 but the gateway does not seem to use that, or prefers IPv4.
- For ethernet, I use some Ziggo/UPC Technicolor modem, IPv4 only.
- When trying to use the Ziggo ethernet while the XS4ALL WiFi was still configured, the ethernet’s DHCP would just be too slow for the gateway to get a lease, so it would choose WiFi instead.
Only a few times I saw:

Request has been terminated Possible causes the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

But I've certainly not seen it for all the many failures. I don't think this is the root cause of the reboot problem; it's just a side effect I'd guess:
It would be great if https://www.thethingsnetwork.org/forum/t/ttn-gateway-faq/11173 would list the values for Reboot reason and an explanation for the status such as LORA: Changing state from 0 to 1. If such list needs to be extracted from some source code, I might be able to help doing that.

Please let us know whatever details you need! (I am not at home right now, and will be home late tonight.) And of course feel free to invite others to this private topic as well.

full log

```text ************************** * The Things Network * * G A T E W A Y * ************************** Firmware name: AmazingAckermann, type: 0, version: 1.0.0, commit: 917719b9, timestamp: 1498499973 Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298 Build time: Jun 26 2017 19:59:53 Reboot reason: 0x13 BOOT: (persisted info) 6F 72 72 65 01 03 83 2C E7 16 05 49 92 3A 54 B7 WIFI: Entering state 0 WIFI: Entering SCAN state 0 MAIN: Initialisation complete LORA: Changing state from 0 to 0 MAIN: Leaving state 0 MAIN: Entering state 1 FLASH: Magic bytes found: wifi config present FLASH: Magic bytes found: activation data present FLASH: Magic bytes not found: no stored FOTA data present FLASH: Loading Firmware Data CNFG: (Firmware HASH (sha256)) FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FLASH: Loading WiFi Data CNFG: WiFi SSID: redacted CNFG: WiFi key: redacted CNFG: WiFi conn_type: 1 CNFG: WiFi sec_type: 4 FLASH: Loading Activation Data CNFG: Gateway ID: arjanvanb-gw-1 CNFG: Gateway Key: ttn-account-v2.W4...redacted...GLyDA CNFG: Account Server URL: https://account.thethingsnetwork.org CNFG: Locked: true CNFG: Locked first time: false MAIN: Leaving state 1 MAIN: Entering state 2 INET: State change to 0 LORA: Initialisation complete LORA: Changing state from 0 to 1 WIFI: Entering state 1 ETH: IP Address: 0.0.0.0 WIFI: Entering state 4 WIFI: Entering SCAN state 1 SNTP: State change from 0 to 1 Scan is completed successfully WIFI: Entering SCAN state 2 WIFI: Entering SCAN state 5 WIFI: Entering SCAN state 0 WIFI: Entering state 2 WIFI: Disabling modules Head magic match void: trying to free an already freed block, ignore SNTP: State change from 1 to 1 WIFI: Entering state 3 WIFI: Enabling modules for client WIFI: Entering state 6 >WIFI: IP Address: 0.0.0.0 CB: INET: Gateway has WiFi INET: State change to 2 INET: Connected to a network, waiting for DHCP lease, checking validity with ping SNTP: State change from 1 to 2 WIFI: IP Address: 192.168.178.53 LORA: Wait init complete, waiting for application. LORA: Changing state from 1 to 2 MON: SYS Stack size: 3961 MON: heap usage: 147KB (156KB), free: 192KB INET: State change to 3 INET: Ping probe INET: Error sending probe on Eth INET: Ping response from MRF24WN, set as default INET: State change to 4 SNTP: State change from 2 to 3 SNTP: State change from 3 to 4 SNTP: State change from 4 to 5 SNTP: State change from 5 to 6 SNTP: State change from 6 to 7 INET: State change to 5 MAIN: Leaving state 2 MAIN: Entering state 3 CNFG: Load online user config state change to 4 HTTP: Close active socket 0 HTTP: Starting connection HTTPS: Connection Opened: Starting TLS Negotiation HTTP: Wait for TLS Connect HTTP: TLS Connection Opened: Starting Clear Text Communication HTTP: Got 1280 bytes HTTP: Connection Closed HTTP: Close active socket 1 CONF: Parsing response token: HTTP/1.1 200 OK CONF: ROUTER URL: mqtts://bridge.eu.thethings.network:8883 CNFG: Load online user config state change to 6 FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870 HTTP: Starting connection HTTPS: Connection Opened: Starting TLS Negotiation HTTP: Wait for TLS Connect MON: SYS Stack size: 2855 MON: heap usage: 227KB (233KB), free: 111KB HTTP: TLS Connection Opened: Starting Clear Text Communication HTTP: Got 1232 bytes HTTP: Connection Closed HTTP: Close active socket 1 CNFG: Load online user config state change to 7 CNFG: Configuring LoRa module LORA: Changing state from 2 to 4 LORA: Starting reconfiguration LORA: version: 01 LORA: Configuration failed, retry LORA: Starting reconfiguration LORA: version: 01 LORA: Configuration failed, retry LORA: Starting reconfiguration MON: SYS Stack size: 2855 MON: heap usage: 151KB (233KB), free: 188KB LORA: version: 01 LORA: Configuration failed, retry LORA: RESET MODULE LORA: ChaSNTP: State change from 0 to 0 SNTP: State change from 0 to 0 ```

zubairhamed commented 6 years ago

FWIW, this is for firmware 0407f7eb

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: 0407f7eb, timestamp: 1517914721
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb  6 2018 11:59:01
Reboot reason: 0x10
BOOT: (persisted info) 6F 72 72 65 01 03 3E 09 75 32 84 A3 6F 67 A9 7A 

WIFI: Entering state 0
WIFI: Entering SCAN state 0

MAIN: Initialisation complete
LORA: Changing state from 0 to 0

MAIN: Leaving state 0
MAIN: Entering state 1
FLASH: Magic bytes found: wifi config present
FLASH: Magic bytes found: activation data present
FLASH: Magic bytes not found: no stored FOTA data present
FLASH: Loading Firmware Data
CNFG: (Firmware HASH (sha256)) FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FLASH: Loading WiFi Data
CNFG: WiFi SSID:      Things-Gateway-001EC03B432F
CNFG: WiFi key:       ***
CNFG: WiFi conn_type: 4
CNFG: WiFi sec_type:  4
FLASH: Loading Activation Data
CNFG: Gateway ID:         zubair-ttn-backer-gateway
CNFG: Gateway Key:        ***
CNFG: Account Server URL: https://account.thethingsnetwork.org
CNFG: Locked:             true
CNFG: Locked first time:  false

MAIN: Leaving state 1
MAIN: Entering state 2
INET: State change to 0
LORA: Initialisation complete
LORA: Changing state from 0 to 1
WIFI: Entering state 1
ETH: IP Address: 0.0.0.0 
WIFI: Entering state 4
WIFI: Entering SCAN state 1
SNTP: State change from 0 to 1
INET: Gateway has Ethernet
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
Scan is completed successfully
WIFI: Entering SCAN state 2
WIFI: Entering SCAN state 5
WIFI: Entering SCAN state 0
WIFI: Entering state 2
WIFI: Disabling modules
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 1 to 1
WIFI: Entering state 3
ETH: IP Address: 192.168.86.155 
WIFI: Enabling modules for server
WIFI: Entering state 6

>WIFI: IP Address: 192.168.84.1 
SNTP: State change from 1 to 2
CB: SNTP: State change from 2 to 3
SNTP: State change from 3 to 4
INET: State change to 3
INET: Ping probe
INET: Error sending probe on WiFi
INET: Ping response from PIC32INT, set as default
INET: State change to 4
SNTP: State change from 4 to 5
SNTP: State change from 5 to 6
SNTP: State change from 6 to 7
INET: State change to 5

MAIN: Leaving state 2
MAIN: Entering state 3

CNFG: Load online user config state change to 4
HTTP: Close active socket 0
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
LORA: Wait init complete, waiting for application.
LORA: Changing state from 1 to 2
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1283 bytes
MON: SYS Stack size: 2837
MON: heap usage: 228KB (237KB), free: 111KB
HTTP: Connection Closed
HTTP: Close active socket 1
CONF: Parsing response token: HTTP/1.1 200 OK
CONF: ROUTER URL: mqtts://bridge.eu.thethings.network:8883

CNFG: Load online user config state change to 6
FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1232 bytes
HTTP: Connection Closed
HTTP: Close active socket 1

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
MON: SYS Stack size: 2837
MON: heap usage: 152KB (237KB), free: 187KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration

grahamehorner commented 6 years ago

interesting note in code

static void restart_lora_configuration(void)
{
    SYS_DEBUG(SYS_ERROR_FATAL, "LORA: RESET MODULE\r\n");
    DRV_USART_Close(appData.USARTHandle);
    **// TODO Verify untested fix. Here we power cycle the module as resetting (done in loraInit) the module was not
    // enough.**
    loraSet(false);
    vTaskDelay(100 / portTICK_PERIOD_MS);
    loraSet(true);
    _setState(APP_LORA_INIT);
}

grahamehorner commented 6 years ago

BOOT: (persisted info) [6F 72 72 65] = magic - errA ? should be erro according to code ? [01] = type - ERROR_MESSAGES_TYPE_POWER_CYCLE

and based on the information at hand given the magic number is incorrect according to the code I've seen; this would suggest a memory pointer issue that is corrupting something ?

cyberjunky commented 6 years ago

@johanstokking

Here my complete logs with extra debugging enabled (own compiled fw)

I compile my own fw from this github repo, I enabled extra debugging (although it's just some lora and flash debugging)

By doing this: `vi src/system_config/TTN_Gateway_v1/system_init.c

SYS_DEBUG_INIT debugInit = { .moduleInit = {0}, .errorLevel = SYS_ERROR_DEBUG }; ` I first deleted activation from gw, and flashed fw via sd card, I then started activation page, and created new gateway. Connected laptop via wifi and choose Ethernet as network, then gw restarted and I lost internet connection on my laptop, see it get activated and connected in ttn console, it send some packets and stayed running for 223 second, then it rebooted all of a sudden, you see this in log:

MON: heap usage: 263KB (269KB), free: 76KB SNTP: State change from 0 to 0 SNTP: State change from 0 to 0

and started again, but seems to fail te get response from NTP servers, it tries for a while and seem to get internet connection again, when it gets TLS traffic, shortly after that you see this in the log:

HTTP: Close active socket 1300,"radio":1,"bandw

So right after closing the TLS connection, some output of LORA module garbled:

1300,"radio":1,"bandw

Then it reboots with all 5 leds on, off, on, off on, etc etc... No logging in console. The gw doesn't get out of this loop. I can reproduce this many times since I got the gateway.

As you can see I don't have the LORA RESET messages like others have in the logs. See info screen shots and logs attached, fresh activation....txt and 1.png are before sudden reboot, it continues in sudden reboot....txt and 2.png but at that end it starts boot looping.

fresh activation and sudden reboot.txt sudden reboot cont into boot loop.txt

If you want me to compile and test with more debugging or other code, let me know.

avbentem commented 6 years ago

For me this is solved 🎉 by always flushing the LoRa UART's receive buffer before sending a new command for the LoRa configuration.

See PR https://github.com/TheThingsProducts/gateway/pull/7.

(And to test, see the firmware at https://www.thethingsnetwork.org/forum/t/ttn-gateway-central/10976/454.)

cyberjunky commented 6 years ago

My gw seems bricked, so I cannot test your code, it only displays the one led on the left, firmware flash from sd is not working anymore, no activity on any other leds. Does someone has the bootloader hex or complete memory dump to flash? (read with mplab_ide) I think I can lend an a ICD3 debugger to try to reprogram it via the ISCP

machiavello commented 6 years ago

Hi all, I have a gateway with a similar rebooting behavior; as soon as the Lora module is activated a reboot follows. Since I am living in NL I have no problem to bring the gateway to Amsterdam office for analysis if needed. Let me know if this is needed, e-mail me directly if so.

johanstokking commented 6 years ago

@machiavello there's an update on the way, incorporating #7

grahamehorner commented 6 years ago

@johanstokking any info on the root cause?

johanstokking commented 6 years ago

Yes. Something with clocks. Details on the forum tomorrow!

mrjdomingus commented 6 years ago

Ok, so my beta updates-enabled gateway has OTA updated to firmware "v1.0.1-facdef23 (2018-02-14T08:16:22Z)" but the reboot loop issue remains. If anything, the reboot will occur even faster than before (with 3-5 minutes after startup, previously I could maintain a stable connection up to 20 minutes). Is this a beta build or a stable build?

mrjdomingus commented 6 years ago

Also the following difference I observed between a working and malfunctioning gateway MAY be relevant: https://www.thethingsnetwork.org/forum/t/ttn-gateway-central/10976/512?u=mrjdomingus

avbentem commented 6 years ago

I tried the firmware linked from the forum:

This got me the old reboot loop again (LORA: Starting reconfiguration is followed by LORA: Configuration failed, retry and eventually LORA: RESET MODULE and Reboot reason: 0x10):

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: facdef23, timestamp: 1518596182
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb 14 2018 08:16:44
Reboot reason: 0x10

This seems to refer to https://github.com/TheThingsProducts/gateway/commit/facdef23.

Then I pulled develop from Git:

I enabled debug logging in debugInit, and all works...!?
Next, I also increased the debug's baud rate in DRV_USART_BAUD_RATE_IDX0 to get proper logging, and I nicely see the logging below.
Finally, I reverted my changes, so simply built the develop branch myself with logging left at SYS_ERROR_INFO and 115200 baud, and got the above reboot loop again.

Is enabling debug logging actually solving the issue!? (Like by introducing some delays, or somehow affecting the other LoRa UART, or maybe even some conditional code?)

Logging when debug logging and higher baud rate enabled based on the develop branch:

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: recv_rpl: 0x23 0x31 0x1 0x0 0x0 0x55 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x3a 0x10 0x0 0x1 0x1 0x4c 0x47 0x38 0x35 0x30 0x31 0x36 0x30 0x31 0x37 0x38 0x32 0x4 0x1 0xd
LORA: sendCommand OK
LORA: version: 01
LORA: flushing: 0d 
LORA: recv_rpl: 0x23 0x34 0x1 0x0 0x0 0x58 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x34 0x1 0x0 0x0 0x58 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x35 0x1 0x0 0x0 0x59 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x36 0x1 0x0 0x0 0x5a 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x37 0x1 0x0 0x0 0x5b 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x40 0x1 0x0 0x0 0x64 0xd
LORA: sendCommand OK
LORA: recv_rpl: 0x23 0x31 0x1 0x0 0x0 0x55 0xd
LORA: sendCommand OK
MON: SYS Stack size: 2843
MON: heap usage: 152KB (234KB), free: 187KB
LORA: recv_rpl: 0x23 0x30 0x1 0x0 0x0 0x54 0xd
LORA: sendCommand OK
LORA: configLora OK
LORA: Configuration succeeded
LORA: Starting operation

grahamehorner commented 6 years ago

@avbentem it would be great if the logging could be controlled via the web UI weather its on/off level min, info, warning, verbose, etc. and the baud rate; so firmware can be ran without rebuilding for logging.

ghost commented 6 years ago

The new firmware (as published before) somewhat helps in the sense that the gateway does not reboot as often as before. Rebooting for a few hours, then working for a few hours, and the cycle starts again. Sometimes just moving the gateway is enough to make it restart.

I have ordered a USB TTL debug cable that I've received today to capture this log, with similar issues as other as seeing: LoRa reprogramming gives error three times and then give up and reboots everything.

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: facdef23, timestamp: 1518596182
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb 14 2018 08:16:44
Reboot reason: 0x10
BOOT: (persisted info) 6F 72 72 65 01 10 3A 36 1F 11 AE 00 3A FE F9 7B 

WIFI: Entering state 0
WIFI: Entering SCAN state 0

MAIN: Initialisation complete
LORA: Changing state from 0 to 0

MAIN: Leaving state 0
MAIN: Entering state 1
FLASH: Magic bytes found: wifi config present
FLASH: Magic bytes found: activation data present
FLASH: Magic bytes found: FOTA data present
FLASH: Loading Firmware Data
CNFG: (Firmware HASH (sha256)) A0 C7 5A 2A 16 16 8E AD 95 37 EB 3B D3 FA 57 06 F8 21 28 F6 71 D0 F9 D7 82 45 48 37 0B E0 6E DD 
FLASH: Loading WiFi Data
CNFG: WiFi SSID:      Things-Gateway-001EC03AF4F8
CNFG: WiFi key:       ***
CNFG: WiFi conn_type: 4
CNFG: WiFi sec_type:  4
FLASH: Loading Activation Data
CNFG: Gateway ID:         peralta-gw-two
CNFG: Gateway Key:        ***
CNFG: Account Server URL: https://account.thethingsnetwork.org
CNFG: Locked:             true
CNFG: Locked first time:  false

MAIN: Leaving state 1
MAIN: Entering state 2
INET: State change to 0
LORA: Initialisation complete
LORA: Changing state from 0 to 1
WIFI: Entering state 1
ETH: IP Address: 0.0.0.0 
WIFI: Entering state 4
WIFI: Entering SCAN state 1
Scan is completed successfully
WIFI: Entering SCAN state 2
WIFI: Entering SCAN state 5
WIFI: Entering SCAN state 0
WIFI: Entering state 2
WIFI: Disabling modules
Head magic match void: trying to free an already freed block, ignore
WIFI: Entering state 3
SNTP: State change from 0 to 1
INET: Gateway has Ethernet
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
WIFI: Enabling modules for server
WIFI: Entering state 6

>WIFI: IP Address: 192.168.84.1 
SNTP: State change from 1 to 2
CB: INET: State change to 3
INET: Ping probe
INET: Error sending probe on WiFi
LORA: Wait init complete, waiting for application.
LORA: Changing state from 1 to 2
SNTP: State change from 2 to 6
MON: SYS Stack size: 3959
MON: heap usage: 147KB (156KB), free: 192KB
ETH: IP Address: 192.168.2.228 
INET: Ping Timeout
INET: Ping probe
INET: Error sending probe on WiFi
INET: Ping response from PIC32INT, set as default
INET: State change to 4
SNTP: State change from 6 to 1
INET: Initiated NTP request.
SNTP: State change from 1 to 2
SNTP: State change from 2 to 6
SNTP: State change from 6 to 1
INET: Initiated NTP request.
SNTP: State change from 1 to 2
SNTP: State change from 2 to 3
SNTP: State change from 3 to 4
SNTP: State change from 4 to 5
SNTP: State change from 5 to 6
SNTP: State change from 6 to 7
INET: State change to 5

MAIN: Leaving state 2
MAIN: Entering state 3

CNFG: Load online user config state change to 4
HTTP: Close active socket 0
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1479 bytes
MON: SYS Stack size: 2837
MON: heap usage: 228KB (235KB), free: 111KB
HTTP: Connection Closed
HTTP: Close active socket 1
CONF: Parsing response token: HTTP/1.1 200 OK
CONF: ROUTER URL: mqtts://bridge.eu.thethings.network:8883

CNFG: Load online user config state change to 6
FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1232 bytes
HTTP: Connection Closed
HTTP: Close active socket 1

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
MON: SYS Stack size: 2837
MON: heap usage: 152KB (236KB), free: 187KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: RESET MODULE
LORA: ChaSNTP: State change from 0 to 0
SNTP: State change from 0 to 0

cyberjunky commented 6 years ago

My gateway ~~is currently~~ was broken ~~(awaiting a possible bootloader reflash fix)~~ but if I look at some of the logs people post and look at my own I saved earlier, I also noticed these two line just before the reboots:

SNTP: State change from 0 to 0 SNTP: State change from 0 to 0

Two state changes after each other and also to same state, could be a logging flaw, but perhaps also a cause of some reboots, for example due to a time jump of several mS? The time is only synced because of TLS cert verfication? Since I don't see it appears in the logging. I noticed a hardcoded pool.ntp.org server entry in the code, should be user configurable I think.

beamzer commented 6 years ago

My TTN gateway is also stuck in a reboot loop :-( last message is also SNTP: State change from 0 to 0

reboot-snippet.txt

is somebody working on this?

cyberjunky commented 6 years ago

Is this issue still being worked on? It seems very quiet here... I was running bootloader v2 and development branch, it's stable but only until I try to activate and join an ttn uno. It reboots my gateway after sending the join accept packet. I had to downgrade to firmware to v1.0.0-917719b9 (2017-06-26T17:59:33Z) to get it to work.

These are the last log lines:

MQTT: Sending status packet MQTT: Sending status succeeded: 4 MON: SYS Stack size: 2843 MON: heap usage: 262KB (267KB), free: 76KB MON: SYS Stack size: 2843 MON: heap usage: 262KB (267KB), free: 76KB LORA: Packet dropped! Bad CRC MON: SYS Stack size: 2843 MON: heap usage: 262KB (267KB), free: 76KB MQTT: Sending status packet MQTT: Sending status succeeded: 5 LORA: Accepted packet MQTT: Sending UPLINK OK MQTT: Received DOWNLINK MON: SYS Stack size: 2843 MON: heap usage: 265KB (267KB), free: 73KB SNTP: State change from 0 to 0 SNTP: State change from 0 to 0

Do you want me to create a new bugreport?

ReiniervdL commented 6 years ago

I tried to update using a FAT32 formatted SD card, exactly followed instructions, but first LED is not blinking after power on, and I do not see firmware version 1.0.1. The unit keeps rebooting.

Firmware name: AmazingAckermann, type: 0, version: 1.0.0, commit: 917719b9, timestamp: 1498499973 Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298 Build time: Jun 26 2017 19:59:53 Reboot reason: 0x13

Will be happy to provide more details if it helps the team.

jurrienjurrien commented 6 years ago

When I connect my gateway to power after a couple of days disconnected, it connects and sends some packets so it comes online in the gateway overview of TTN Console. Then after couple of minutes, the gateway reboots and keeps rebooting every time after about one minute.

This morning I've compiled the latest develop branch but still the same issue: after about one minute reboot with reason: 0x10. See the attached log of the console.

Please let me know how to provide more details in order to determine if this is firmware related or that the gateway is just broken... putty-20180303-095549.log ...

smbunn commented 6 years ago

Has this reboot issue been fixed. Mine gets to 3 LEDs on then all 5 come on for 7 seconds and it all starts again. Is there a guide and a file for the firmware upgrade via an SD card mentioned above?

skaldo commented 6 years ago

Hello, we just received the new TTN Gateway. Unfortunately we're also facing the reboot loop issue (see the logs below). The issue is present in the factory 1.0.0 firmware. We've also tried the following firmwares without success.

Is this a HW issue? Do we need to send the Gateway back, or is there some SW fix going to be released soon?

Firmwares:

Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: a730dbcc, timestamp: 1519644505
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb 26 2018 11:28:45

Firmware name: AmazingAckermann, type: 0, version: 1.0.0, commit: 917719b9, timestamp: 1498499973
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Jun 26 2017 19:59:53

Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: 4129a80e, timestamp: 1518792985
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb 16 2018 14:56:43

Boot log:

**************************
*   The Things Network   *
*      G A T E W A Y     *
**************************
Firmware name: AmazingAckermann, type: 0, version: 1.0.1, commit: a730dbcc, timestamp: 1519644505
Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298
Build time: Feb 26 2018 11:28:45
Reboot reason: 0x10
BOOT: (persisted info) 6F 72 72 65 01 03 CB 2E 3A 5C DE 27 A1 B3 9B 13 

WIFI: Entering state 0
WIFI: Entering SCAN state 0

MAIN: Initialisation complete
LORA: Changing state from 0 to 0

MAIN: Leaving state 0
MAIN: Entering state 1
SDCARD: Mounted disk
FLASH: Magic bytes found: wifi config present
FLASH: Magic bytes found: activation data present
FLASH: Magic bytes not found: no stored FOTA data present
FLASH: Loading Firmware Data
CNFG: (Firmware HASH (sha256)) 60 3B 2C 8C 0F 28 27 11 EA 16 8E 2D C7 BA D9 04 C0 57 F3 DA 32 FD 77 6E FF 5F 2D D4 E5 8F 05 7C 
FLASH: Loading WiFi Data
CNFG: WiFi SSID:      Things-Gateway-001EC03F899A
CNFG: WiFi key:       ***
CNFG: WiFi conn_type: 4
CNFG: WiFi sec_type:  4
FLASH: Loading Activation Data
CNFG: Gateway ID:         hochschule_mannheim
CNFG: Gateway Key:        ***
CNFG: Account Server URL: https://account.thethingsnetwork.org
CNFG: Locked:             true
CNFG: Locked first time:  false

MAIN: Leaving state 1
MAIN: Entering state 2
INET: State change to 0
LORA: Initialisation complete
LORA: Changing state from 0 to 1
WIFI: Entering state 1
ETH: IP Address: 0.0.0.0 
WIFI: Entering state 4
WIFI: Entering SCAN state 1
SNTP: State change from 0 to 1
INET: Gateway has Ethernet
INET: State change to 2
INET: Connected to a network, waiting for DHCP lease, checking validity with ping
Scan is completed successfully
WIFI: Entering SCAN state 2
WIFI: Entering SCAN state 5
WIFI: Entering SCAN state 0
WIFI: Entering state 2
WIFI: Disabling modules
Head magic match void: trying to free an already freed block, ignore
SNTP: State change from 1 to 1
WIFI: Entering state 3
ETH: IP Address: 141.19.78.83 
WIFI: Enabling modules for server
WIFI: Entering state 6

>WIFI: IP Address: 192.168.84.1 
SNTP: State change from 1 to 2
CB: SNTP: State change from 2 to 3
SNTP: State change from 3 to 4
INET: State change to 3
INET: Ping probe
INET: Error sending probe on WiFi
INET: Ping response from PIC32INT, set as default
INET: State change to 4
SNTP: State change from 4 to 5
SNTP: State change from 5 to 6
SNTP: State change from 6 to 7
INET: State change to 5

MAIN: Leaving state 2
MAIN: Entering state 3

CNFG: Load online user config state change to 4
HTTP: Close active socket 0
HTTP: Starting connection
LORA: Wait init complete, waiting for application.
LORA: Changing state from 1 to 2
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1291 bytes
MON: SYS Stack size: 2837
MON: heap usage: 228KB (233KB), free: 111KB
HTTP: Connection Closed
HTTP: Close active socket 1
CONF: Parsing response token: HTTP/1.1 200 OK
CONF: ROUTER URL: mqtts://bridge.eu.thethings.network:8883

CNFG: Load online user config state change to 6
FREQ: APP_URL_Buffer: https://account.thethingsnetwork.org/api/v2/frequency-plans/EU_863_870
HTTP: Starting connection
HTTPS: Connection Opened: Starting TLS Negotiation
HTTP: Wait for TLS Connect
HTTP: TLS Connection Opened: Starting Clear Text Communication
HTTP: Got 1232 bytes
HTTP: Connection Closed
HTTP: Close active socket 1

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module
LORA: Changing state from 2 to 4
LORA: Starting reconfiguration
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
MON: SYS Stack size: 2837
MON: heap usage: 152KB (233KB), free: 187KB
LORA: version: 01
LORA: Configuration failed, retry
LORA: Starting reconfiguration
SNTP: State change from 0 to 0
SNTP: State change from 0 to 0

Pixelworlds commented 6 years ago

Any word on the progress with this issue?!

skaldo commented 6 years ago

@smbunn the firmware upgrade procedure is described here: https://github.com/TheThingsProducts/gateway/tree/master/firmware#installation Unfortunately, our Gateway was stuck with all three firmware versions mentioned in my last comment.

Please let us know if you get yours up and running.

gniersma commented 6 years ago

Same reboot issue here: `

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module LORA: Changing state from 2 to 4 LORA: Starting reconfiguration MON: SYS Stack size: 2873 MON: heap usage: 151KB (233KB), free: 188KB SNTP: State change from 0 to 0 SNTP: State change from 0 to 0

The Things Network *
G A T E W A Y *

Firmware name: AmazingAckermann, type: 0, version: 1.0.0, commit: 917719b9, timestamp: 1498499973 Bootloader revision: 1, commit: 7167873a, timestamp: 1496411298 Build time: Jun 26 2017 19:59:53 Reboot reason: 0x13 BOOT: (persisted info) 6F 72 72 65 01 03 E9 0A AF 78 F0 54 6B E1 27 26`

Any progress on this ??

cyberjunky commented 6 years ago

The new bootloader code makes my gateway more stable, but it still reboots when it's transfering/receiving packets. As reported here https://github.com/TheThingsProducts/gateway/issues/1#issuecomment-369336514

ReiniervdL commented 6 years ago

I had the chance to compare two TTN 915 MHz gateways, one working and one DOA with the reboot issue. Both have the same 1635D4 LoRa module connector. Replacing the module in the connector did not help getting the defective unit to function.

I exchanged the LoRa module in both gateways, and found that the issue swaps with the Microchip LG9271 LoRa module. The defective gateway works after placing the LG9271 from the working gateway. Previously I was not able to update the firmware using an SD card on the DOA gateway. After inserting the LoRa module from the functioning gateway, the firmware updated to 1.0.1. On the working LoRa module the SX1301 chip is running hot (69C), the microchip PIC is 52C. This is normal according to @nestorayuso. On the defective LoRa module the chip temperatures are much lower.

martinichka commented 6 years ago

@cyberjunky can you verify that https://github.com/TheThingsProducts/gateway/commit/2e56f3b54f8d4d7d655201eb32c2d2efa78994e6 solves the downlink problem?

You can download the full image from: https://thethingsproducts.blob.core.windows.net/the-things-gateway/v1/develop/firmware-with-bootloader.hex

ReiniervdL commented 6 years ago

Adding to previous message: Replacement of module = resettling. I found that the DOA unit with the working LG9271 was not operating stable, dropping connection, etc. As the difference between the two unit main boards is firmware, I suspect that firmware 1.0.1 is causing the instability. When the working LG9271 LoRa module is placed back on it's original board with firmware 1.0.0 it has been running stable.

cyberjunky commented 6 years ago

@martinichka the first tests done just now with the new firmware are looking very good, no issues. The ttn uno can activate itself without crashing the gateway, which didn't work a single time before. I will add some sensors to the uno to push data through the gw during the next hours, nice job!

Version Info
Hardware:	v1
Bootloader:	r2-c463e87e (2018-02-23T14:42:40Z)
Firmware:	v1.0.1-2e56f3b5 (2018-03-12T12:51:37Z)

martinichka commented 6 years ago

@cyberjunky thanks.

For those without a programmer: a test firmware is available to be updated via SD card as well: #19

You can test it by following the instructions at https://github.com/TheThingsProducts/gateway/blob/develop/firmware/README.md#installation using these files:

ReiniervdL commented 6 years ago

Just tried the new firmware on a DOA gateway. I find that the firmware only takes place after successful configuration of the LoRa module. With a broken Lora Module (LG9271), the software does not get to the firmware update:

Here is what I see when using a non-functional LoRa module:

CNFG: Configuring LoRa module LORA: Changing state from 2 to 4 LORA: Starting reconfiguration LORA: version: 02 LORA: Configuration failed, retry LORA: Starting reconfiguration MON: SYS Stack size: 2837 MON: heap usage: 151KB (233KB), free: 188KB LORA: version: 02 LORA: Configuration failed, retry LORA: Starting reconfiguration LORA: version: 02 LORA: Configuration failed, retry LORA: RESET MODULE LORA: ChaSNTP: State change from 0 to 0 SNTP: State change from 0 to 0

Here’s the Same gateway, but with a functional LG9271 module:

CNFG: Load online user config state change to 7

CNFG: Configuring LoRa module LORA: Changing state from 2 to 4 LORA: Starting reconfiguration LORA: version: 02 LORA: Configuration succeeded LORA: Starting operation LORA: Changing state from 4 to 6 MON: SYS Stack size: 2837 MON: heap usage: 151KB (233KB), free: 188KB LORA: Changing state from 6 to 3 LORA: GOING ASYNC LORA: Changing state from 3 to 9

CNFG: Load online user config state change to 9

MAIN: Leaving state 3 MAIN: Entering state 4 FLASH: Erasing FOTA Data FLASH: Loading FOTA Data FIRM: Requesting key ... HTTP: Starting connection HTTPS: Connection Opened: Starting TLS Negotiation HTTP: Wait for TLS Connect HTTP: TLS Connection Opened: Starting Clear Text Communication HTTP: Got 515 bytes HTTP: Connection Closed HTTP: Close active socket 1 FIRM: Starting download FIRM: available bytes: 79 FIRM: (Downloaded FOTA key) A0 C7 5A 2A 16 16 8E AD 95 37 EB 3B D3 FA 57 06 F8 21 28 F6 71 D0 F9 D7 82 45 48 37 0B E0 6E DD FIRM: (Stored FOTA key) FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FLASH: Erasing FOTA Data and Image MON: SYS Stack size: 2831 MON: heap usage: 168KB (252KB), free: 171KB

On Mar 13, 2018, at 9:59 AM, Marten Lootsma notifications@github.com wrote:

@cyberjunky https://github.com/cyberjunky thanks.

For those without a programmer: a test firmware is available to be updated via SD card as well: #19 https://github.com/TheThingsProducts/gateway/pull/19 You can test it by following the instructions at https://github.com/TheThingsProducts/gateway/blob/develop/firmware/README.md#installation https://github.com/TheThingsProducts/gateway/blob/develop/firmware/README.md#installation using these files:

https://thethingsproducts.azureedge.net/the-things-gateway/v1/feature/fixes_lora_module_communication/firmware.hex https://thethingsproducts.azureedge.net/the-things-gateway/v1/feature/fixes_lora_module_communication/firmware.hex https://thethingsproducts.azureedge.net/the-things-gateway/v1/feature/fixes_lora_module_communication/checksums https://thethingsproducts.azureedge.net/the-things-gateway/v1/feature/fixes_lora_module_communication/checksums — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/TheThingsProducts/gateway/issues/1#issuecomment-372740116, or mute the thread https://github.com/notifications/unsubscribe-auth/AMMNpkV1mLCnlC6dZrrY-_j3aL-9t1_pks5td_sDgaJpZM4R0NRp.

cyberjunky commented 6 years ago

@martinichka you're welcome. It still runs without any issues uploading dust sensor data every 30 seconds for hours in a row. capture _2018-03-13-20-22-32

jurrienjurrien commented 6 years ago

@martinichka thanks for instructions!

I just recompiled the firmware and started the gateway after being disconnected for about a week. The gateway was working great for about 20 minutes. A lopy device on my desk was able to send messages but after 20 minutes no messages were shown anymore in the TTN console.

After rebooting it looks like the communication with the LORA modules has an issue because it is now in a loop trying to reopen the UART

LORA: UART TIMEOUT
LORA: UART TIMEOUT
LORA: Close and reopen UART on 115200 baud.
LORA: Changing state from 2 to 1
MON: SYS Stack size: 2853
MON: heap usage: 151KB (233KB), free: 188KB
LORA: Wait init complete, check for communication on 115200 baud.
LORA: Changing state from 1 to 2
LORA: UART TIMEOUT

TheThingsProducts / gateway

Reboot Loop - Reboot reason: 0x10 or 0x13 #1