OpenEEW Sensor occasionally not connecting

[neilh10]

For my sensor, I am seeing that it won't connect to the local network Arthur2004Sid that it is configured for. This happens whether it is WiFi or WiFi/Ethernet. (I haven't tested for just Ethernet no WiFi case yet)

This happens also after reboots. That is it was connected to the network, and for whatever reason it reboots, and then it won't reconnect - at least in the following 15minutes. On power cycling - removing USB C, waiting 5 seconds, and plugging back in, it comes up successfully.

I would expect it to reconnect to a working network - within 5minutes. (Just a time I picked for testing, maybe this could be refined to 1minute) At this stage, for a production ready hardware node, I would expect it to connect to a network 99.9% of the time. That is if there was 2000 power cycles for the same device, it may fail 1 time, and require manual intervention.
Is 99.9% at this stage a reasonable target?. Perhaps to phrase this another way, depending on what the failure mechanism is, it could be that if there are 2000devices, on power failure, one is likely to require a manual intervention.

This is partially a note to myself that I could start looking at it next week when I get some time. (I am working on a somewhat similar problem, but probably different reasons for a Digi S6 WiFi device). If this has already been figured out, I'm happy to retest next week (I'm away till the 18th)

OpenEEW Sensor Application (v1.5.0) ESP32 WiFi interface ready Stored networks : 1 ESP32 WiFi started Completed scan for access points WiFi Network scan done 9 network(s) found 1: Arthur2004Sid (-42) 2: LazySky (-74) 3: SonicAdslSsid (-82) 4: AzondeNetSsid (-85) 5: mp@home (-88) 6: Triplets 2.4 (-89) 7: DIRECT-57-HP OfficeJet 3830 (-89) 8: HomeBaseWifi104 (-91) 9: DIRECT-58-HP DeskJet 3700 series (-92)* ESP32 WiFi interface ready Reading stored networks from NVM ESP32 WiFi started ESP32 WiFi started Found network Arthur2004Sid , xxxxxx Disconnected from WiFi access point LED_LISTEN_WIFI - Blue LED_LISTEN_WIFI - Blue LED_LISTEN_WIFI - Blue LED_LISTEN_WIFI - Blue LED_LISTEN_WIFI - Blue There was a problem connecting to WiFi Got no match for network Got no match for network Got no match for network Got no match for network Got no match for network Got no match for network Got no match for network Got no match for network Found no matches for saved networks ESP32 WiFi interface ready ESP32 WiFi started Unhandled Network Interface event : 14 Waiting for SmartConfig. ESP32 soft-AP stop ESP32 soft-AP stop .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue .LED_LISTEN_WIFI - Blue

[johnwalicki]

Would be great to get your device onto the very latest version. Some important things in the log above. Strong wifi signal: Arthur2004Sid (-42)* The firmware finds a SSID it knows about and tries for 60 seconds to connect. Fails for some reason.

Found network Arthur2004Sid , xxxxxx
Disconnected from WiFi access point
LED_LISTEN_WIFI - Blue
LED_LISTEN_WIFI - Blue
LED_LISTEN_WIFI - Blue
LED_LISTEN_WIFI - Blue
LED_LISTEN_WIFI - Blue
There was a problem connecting to WiFi

Skips the 8 networks it doesn't have secrets for. Then drops into SmartConfig polling.

[johnwalicki]

If will stay indefinitely in SmartConfig polling mode. Maybe after 10 minutes the device should just force restart itself and try to cycle through the process again.

Here's the use case. The device normally connects to a network. There's a power outage in the building. After power is restored, the device comes up first/fast. The wifi router is not yet available. The device goes into SmartConfig polling mode and never recovers. Eventually the network router comes up but the device is stuck because of its timing. A restart 10 minutes later would bring it back to normal.

Not sure it would fix your issue, but I imagine it could be a common problem in the field. We would not want to dispatch someone just to power cycle the sensor.

[johnwalicki]

So let's turn your odd case into a more robust solution. Is 10 minutes the right amount of time?

If the user really does want to start the mobile app and complete the SmartConfig pairing, can they be expected to do so within 10 minutes?

[johnwalicki]

Is 10 minutes insufficient to download/install/run the mobile provisioning app? Maybe 15 minutes? We're trying to solve for the remote unattended device where it's normal network has temporarily disappeared

[johnwalicki]

@andygrillo ^^^ Opinions?

[johnwalicki]

I created a patch: Recover from down WiFi router ~ 10 minutes https://github.com/openeew/openeew-firmware/commit/3b64071241b449aba090ffe9cc5e823ed2639252

[neilh10]

Great - I've been away camping, now catching up. I agree with your logic of retry. The device has nothing it can do, purpose in the world, till it connects to a provisioned networks, or gets informed through the smart config of another network. I hadn't thought about the falling through to smart config Will figure out how to update to latest.

I would think it would be valuable to describe a state machine for connection attempts, Smart Config pairing, and reboot timing. Reboot after 10minutes sounds a long time, but seems to me in the right ballpark when considering smart config operation. I wonder if there is a cold boot and hot boot. Its just a bit strange that it reboots and doesn't then connect. Possibly needs a portion of memory to be reserved for reboot reason (not initialized by cc startup) ,

I need to also understand the Smart Config process - it worked super well for me on the first attempt.

I agree with the value of the MQTT factory initialization in case something comes unstuck, or just starting again.

Some of the networking issues, the WiFi device hardware/software is complex (possibly needs characterizing ugh!), host networks can be many flavours and is evolving (WiFi6), multiple host networks configured, the install process could be two or more networks including initial device checking and then another for deployment, host network can become isolated (connects but no internet), partial Smart Config with user being called away in the process (happened to me) - so may not be possible to cover all possibilities, but some tradeoffs.

[neilh10]

All the cases so far tested for on version150 are now working. Except for one which is documented here https://github.com/openeew/openeew-firmware/issues/14