search

sparkfun / SparkFun_RTK_Everywhere_Firmware

Centimeter precision GPS/GNSS using L1/L2/L5 signals broadcast over Bluetooth, WiFi, and Ethernet.
http://docs.sparkfun.com/SparkFun_RTK_Everywhere_Firmware/
Other
8 stars 6 forks source link

(EVK) Initial provisioning fails over Ethernet #363

Open nseidle opened 1 week ago

nseidle commented 1 week ago

The device should use ethernet, if available, to provision/update keys during POR.

To replicate:

Device will fail to provision because in States.ino the line:

else if (wifiNetworkCount() == 0 && strlen(settings.pointPerfectCurrentKey) == 0)

is WiFi centric.

Instead, we should check for ethernet connectivity as well and enter STATE_KEYS_PROVISION_STARTED or add states to start ethernet.

PaulZC commented 3 days ago

Investigating...

I see:

[ 14262][D][NVM.ino:174] recordSystemSettingsToFileSD(): Settings recorded to SD: /SFE_EVK_Settings_0.txt
[ 14290][D][NVM.ino:204] recordSystemSettingsToFileLFS(): Removing LittleFS: /SFE_EVK_Settings_0.txt
[ 14524][D][NVM.ino:216] recordSystemSettingsToFileLFS(): Settings recorded to LittleFS: /SFE_EVK_Settings_0.txt
STATE_ROVER_NOT_STARTED --> STATE_ROVER_NO_FIX, 2024-06-27 12:49:52.260
[ 15058][D][Begin.ino:1386] deleteSDSizeCheckTask(): sdSizeCheck Task deleted
Log file name: /SFE_EVK_240627_124952.ubx
MQTT Client start
Rover Accuracy (m): 0.221, SIV: 32 GNSS State: 3D Fix
STATE_ROVER_NO_FIX --> STATE_ROVER_FIX, 2024-06-27 12:49:53.403
Log file size: 3380 - Generation rate: 0.7kB/s
Ethernet IP address: 192.168.0.91
[ 16227][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16281][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16335][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16389][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16449][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16503][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16557][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16611][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[ 16670][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
Ethernet IP address: 192.168.0.91

The rolling MQTT Client start continues until eventually the heap goes bang:

[119724][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
[119778][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
MQTT Client start
Ethernet IP address: 192.168.0.91
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
MQTT Client start
Ethernet IP address: 192.168.0.91

abort() was called at PC 0x4026f3c3 on core 1

Backtrace: 0x4008326d:0x3ffd23a0 0x4009853d:0x3ffd23c0 0x4009dcfd:0x3ffd23e0 0x4026f3c3:0x3ffd2460 0x4026f3f8:0x3ffd2480 0x4026f4d3:0x3ffd24a0 0x4026f56a:0x3ffd24c0 0x40109589:0x3ffd24e0 0x400f38cd:0x3ffd2510 0x400fbfac:0x3ffd2530 0x401044d5:0x3ffd2550 0x40148a0c:0x3ffd2570

ELF file SHA256: 345561d16134ed87
ecoding stack results
0x4008326d: panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c line 466
0x4009853d: esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c line 84
0x4009dcfd: abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/abort.c line 38
0x4026f3c3: __cxxabiv1::__terminate(void (*)()) at /builds/idf/crosstool-NG/.build/HOST-i686-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc line 48
0x4026f3f8: std::terminate() at /builds/idf/crosstool-NG/.build/HOST-i686-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc line 58
0x4026f4d3: __cxxabiv1::__cxa_throw(void*, std::type_info*, void (*)(void*)) at /builds/idf/crosstool-NG/.build/HOST-i686-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc line 98
0x4026f56a: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/HOST-i686-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/new_op.cc line 54
0x40109589: NetworkClientSecure::NetworkClientSecure() at C:\Users\pc235\AppData\Local\Arduino15\packages\esp32\hardware\esp32\3.0.1\libraries\NetworkClientSecure\src\NetworkClientSecure.cpp line 35
0x400f38cd: mqttClientUpdate() at C:\Users\pc235\Documents\GitHub\SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere/RTKNetworkClient.h line 268
0x400fbfac: networkUpdate() at C:\Users\pc235\Documents\GitHub\SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere/Network.ino line 1237
0x401044d5: loop() at C:\Users\pc235\Documents\GitHub\SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere/RTK_Everywhere.ino line 1203
0x40148a0c: loopTask(void*) at C:\Users\pc235\AppData\Local\Arduino15\packages\esp32\hardware\esp32\3.0.1\cores\esp32\main.cpp line 73

So, we need to fix the rolling MQTT Client start. We have a memory leak too, but that may be inside NetworkClientSecure...

PaulZC commented 3 days ago

mqttClientStart is only called by mqttClientUpdate when mqttClientState is MQTT_CLIENT_OFF.

mqttClientStart calls mqttClientStop(false). mqttClientStop should be advancing the state to mqttClientSetState(MQTT_CLIENT_ON). Is it?

Turning on debugMqttClientState and debugNetworkLayer.

[ 13983][D][NVM.ino:174] recordSystemSettingsToFileSD(): Settings recorded to SD: /SFE_EVK_Settings_0.txt
[ 14004][D][NVM.ino:204] recordSystemSettingsToFileLFS(): Removing LittleFS: /SFE_EVK_Settings_0.txt
[ 14233][D][NVM.ino:216] recordSystemSettingsToFileLFS(): Settings recorded to LittleFS: /SFE_EVK_Settings_0.txt
STATE_ROVER_NOT_STARTED --> STATE_ROVER_NO_FIX, 2024-06-27 13:17:48.233
[ 14768][D][Begin.ino:1386] deleteSDSizeCheckTask(): sdSizeCheck Task deleted
Log file name: /SFE_EVK_240627_131748.ubx
Ethernet connected
FreeHeap: 84256 / HeapLowestPoint: 79092 / LargestBlock: 4128756 / Used PSRAM: 63240
MQTT Client start
MQTT Client Off Last Uptime: 0 00:00:00.000 (Reconnects: 0)
FreeHeap: 84256 / HeapLowestPoint: 79092 / LargestBlock: 4128756 / Used PSRAM: 63240
MQTT_CLIENT_OFF --> MQTT_CLIENT_ON
Rover Accuracy (m): 0.153, SIV: 31 GNSS State: 3D Fix
STATE_ROVER_NO_FIX --> STATE_ROVER_FIX, 2024-06-27 13:17:49.397
Log file size: 4125 - Generation rate: 0.8kB/s
*WIFI_STATE_OFF
Ethernet started with DHCP
Network starting user MQTT Client on Active
Network request to start Active
Network layer starting Active
Network State: NETWORK_STATE_OFF --> NETWORK_STATE_DELAY
MQTT_CLIENT_ON --> MQTT_CLIENT_NETWORK_STARTED
networkTypeUpdate, network->type: Active --> Ethernet
networkTypeUpdate, network->requestedNetwork: Active --> Ethernet
Network starting Ethernet
Network State: NETWORK_STATE_DELAY --> NETWORK_STATE_CONNECTING
Network connected to Ethernet
Network State: NETWORK_STATE_CONNECTING --> NETWORK_STATE_IN_USE
Ethernet IP address: 192.168.0.91
MQTT_CLIENT_NETWORK_STARTED --> MQTT_CLIENT_CONNECTING_2_SERVER
[ 16009][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
Failed to read from LittleFS: /SFE_EVK_certificate_0.txt
MQTT Client Connecting Last Uptime: 0 00:00:00.000 (Reconnects: 0)
Freeing mqttSecureClient
FreeHeap: 84224 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 69584
FreeHeap: 84224 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 65552
Network stopping user MQTT Client on Ethernet
MQTT_CLIENT_CONNECTING_2_SERVER --> MQTT_CLIENT_OFF
FreeHeap: 84224 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 65552
MQTT Client start
MQTT Client Off Last Uptime: 0 00:00:00.000 (Reconnects: 0)
FreeHeap: 84224 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 65552
MQTT_CLIENT_OFF --> MQTT_CLIENT_ON
Network starting user MQTT Client on Ethernet
MQTT_CLIENT_ON --> MQTT_CLIENT_NETWORK_STARTED
MQTT_CLIENT_NETWORK_STARTED --> MQTT_CLIENT_CONNECTING_2_SERVER
[ 16116][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
Failed to read from LittleFS: /SFE_EVK_certificate_0.txt
MQTT Client Connecting Last Uptime: 0 00:00:00.000 (Reconnects: 0)
Freeing mqttSecureClient
FreeHeap: 84192 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 71888
FreeHeap: 84192 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 67856
Network stopping user MQTT Client on Ethernet
MQTT_CLIENT_CONNECTING_2_SERVER --> MQTT_CLIENT_OFF
FreeHeap: 84192 / HeapLowestPoint: 79092 / LargestBlock: 4063220 / Used PSRAM: 67856
MQTT Client start
PaulZC commented 3 days ago

OK. This is critical:

Failed to read from LittleFS: /SFE_EVK_certificate_0.txt
MQTT Client Connecting Last Uptime: 0 00:00:00.000 (Reconnects: 0)
Freeing mqttSecureClient

SFE_EVK_certificate_0.txt doesn't exist because it is provided during ZTP. And we haven't yet performed ZTP.

If I open the PointPerfect menu and then Provision Device, all goes well:

MQTT Client start
MQTT Client Off Last Uptime: 0 00:00:00.000 (Reconnects: 0)
FreeHeap: 83940 / HeapLowestPoint: 79064 / LargestBlock: 4063220 / Used PSRAM: 83976
MQTT_CLIENT_OFF --> MQTT_CLIENT_ON

SparkFun RTK EVK d99.99
** Bluetooth SPP and BLE broadcasting as: EVK Rover-8EF2 **
Menu: Main
1) Configure GNSS Receiver
2) Configure GNSS Messages
3) Configure Base
4) Configure Ports
5) Configure Logging
6) Configure WiFi
7) Configure TCP/UDP
e) Configure Ethernet
f) Firmware Update
i) Configure Corrections Priorities
n) Configure NTP
p) Configure PointPerfect
r) Configure Radios
s) Configure System
t) Configure Instrument Setup
u) Configure User Profiles
+) Enter Command line mode
x) Exit
p

Menu: PointPerfect Corrections
Days until keys expire: No keys
1) PointPerfect Corrections: Enabled
2) Toggle Auto Key Renewal: Enabled
3) Provision Device
4) Show device ID
c) Clear the Keys
k) Manual Key Entry
g) Geographic Region: EU
x) Exit
3
[ 23711][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
[ 23722][D][NetworkManager.cpp:83] hostByName(): Clearing DNS cache
[ 24033][D][NetworkManager.cpp:123] hostByName(): DNS found IPv4 52.209.93.91
[ 24040][V][ssl_client.cpp:68] start_ssl_client(): Free internal heap before TLS 83696
[ 24048][V][ssl_client.cpp:75] start_ssl_client(): Starting socket (domain 2)
[ 24130][V][ssl_client.cpp:166] start_ssl_client(): Seeding the random number generator
[ 24139][V][ssl_client.cpp:174] start_ssl_client(): Setting up the SSL/TLS structure...
[ 24148][V][ssl_client.cpp:194] start_ssl_client(): Loading CA cert
[ 24157][V][ssl_client.cpp:288] start_ssl_client(): Setting hostname for TLS session...
[ 24167][V][ssl_client.cpp:301] start_ssl_client(): Free internal heap after mbedtls_ssl_setup 83416
[ 24176][V][ssl_client.cpp:311] ssl_starttls_handshake(): Performing the SSL/TLS handshake...
[ 24612][V][ssl_client.cpp:332] ssl_starttls_handshake(): Verifying peer X.509 certificate...
[ 24621][V][ssl_client.cpp:340] ssl_starttls_handshake(): Certificate verified.
[ 24628][V][ssl_client.cpp:356] ssl_starttls_handshake(): Free internal heap after TLS 83216
client connected to api.thingstream.io on port 443
[ 24637][V][HTTPClient.cpp:230] beginInternal(): url: https://api.thingstream.io/ztp/pointperfect/credentials
[ 24651][D][HTTPClient.cpp:275] beginInternal(): switching host from '' to 'api.thingstream.io'. disconnecting first
[ 24662][D][HTTPClient.cpp:366] disconnect(): tcp stop
[ 24667][V][ssl_client.cpp:362] stop_ssl_socket(): Cleaning SSL connection.
[ 24677][D][HTTPClient.cpp:281] beginInternal(): protocol: https, host: api.thingstream.io port: 443 url: /ztp/pointperfect/credentials
[ 24690][D][HTTPClient.cpp:552] sendRequest(): request type: 'POST' redirCount: 0

[ 24697][E][NetworkClient.cpp:315] setSocketOption(): fail on 0, errno: 9, "Bad file number"
[ 24706][D][NetworkManager.cpp:123] hostByName(): DNS found IPv4 52.209.93.91
[ 24713][V][ssl_client.cpp:68] start_ssl_client(): Free internal heap before TLS 83204
[ 24721][V][ssl_client.cpp:75] start_ssl_client(): Starting socket (domain 2)
[ 24761][V][ssl_client.cpp:166] start_ssl_client(): Seeding the random number generator
[ 24769][V][ssl_client.cpp:174] start_ssl_client(): Setting up the SSL/TLS structure...
[ 24777][V][ssl_client.cpp:194] start_ssl_client(): Loading CA cert
[ 24787][V][ssl_client.cpp:288] start_ssl_client(): Setting hostname for TLS session...
[ 24797][V][ssl_client.cpp:301] start_ssl_client(): Free internal heap after mbedtls_ssl_setup 83088
[ 24806][V][ssl_client.cpp:311] ssl_starttls_handshake(): Performing the SSL/TLS handshake...
[ 25212][V][ssl_client.cpp:332] ssl_starttls_handshake(): Verifying peer X.509 certificate...
[ 25221][V][ssl_client.cpp:340] ssl_starttls_handshake(): Certificate verified.
[ 25228][V][ssl_client.cpp:356] ssl_starttls_handshake(): Free internal heap after TLS 83088
[ 25236][D][HTTPClient.cpp:1090] connect():  connected to api.thingstream.io:443
[ 25373][V][HTTPClient.cpp:1179] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
[ 25381][V][HTTPClient.cpp:1179] handleHeaderResponse(): RX: 'Date: Thu, 27 Jun 2024 13:26:32 GMT'
[ 25390][V][HTTPClient.cpp:1179] handleHeaderResponse(): RX: 'Content-Type: application/json'
[ 25398][V][HTTPClient.cpp:1179] handleHeaderResponse(): RX: 'Content-Length: 5663'
[ 25406][V][HTTPClient.cpp:1179] handleHeaderResponse(): RX: 'Connection: keep-alive'
[ 25414][V][HTTPClient.cpp:1179] handleHeaderResponse(): RX: ''
[ 25420][D][HTTPClient.cpp:1235] handleHeaderResponse(): code: 200
[ 25426][D][HTTPClient.cpp:1238] handleHeaderResponse(): size: 5663
[ 25432][D][HTTPClient.cpp:596] sendRequest(): sendRequest code=200

[ 25459][V][HTTPClient.cpp:1377] writeToStreamDataBlock(): connection closed or file end (written: 5663).
[ 25468][D][HTTPClient.cpp:364] disconnect(): tcp keep open for reuse
[ 25483][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_certificate_0.txt does not exist, no permits for creation
[ 25555][D][NVM.ino:1462] recordFile(): File recorded to LittleFS: /SFE_EVK_certificate_0.txt
[ 25567][E][vfs_api.cpp:99] open(): /littlefs/SFE_EVK_privateKey_0.txt does not exist, no permits for creation
[ 25645][D][NVM.ino:1462] recordFile(): File recorded to LittleFS: /SFE_EVK_privateKey_0.txt
[ 25693][D][HTTPClient.cpp:364] disconnect(): tcp keep open for reuse
[ 25699][V][ssl_client.cpp:362] stop_ssl_socket(): Cleaning SSL connection.
[ 25710][V][ssl_client.cpp:362] stop_ssl_socket(): Cleaning SSL connection.
[ 25717][V][ssl_client.cpp:362] stop_ssl_socket(): Cleaning SSL connection.
Device successfully provisioned. Keys obtained.
[ 25728][D][NVM.ino:155] recordSystemSettingsToFileSD(): Removing from SD: /SFE_EVK_Settings_0.txt
[ 25815][D][NVM.ino:174] recordSystemSettingsToFileSD(): Settings recorded to SD: /SFE_EVK_Settings_0.txt
[ 25856][D][NVM.ino:204] recordSystemSettingsToFileLFS(): Removing LittleFS: /SFE_EVK_Settings_0.txt
[ 26082][D][NVM.ino:216] recordSystemSettingsToFileLFS(): Settings recorded to LittleFS: /SFE_EVK_Settings_0.txt
wifiStop called by C:\Users\pc235\Documents\GitHub\SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere\menuPP.ino 1592
networkStop called by C:\Users\pc235\Documents\GitHub\SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere\WiFi.ino 446

Menu: PointPerfect Corrections
Days until keys expire: 31
1) PointPerfect Corrections: Enabled
2) Toggle Auto Key Renewal: Enabled
3) Update Keys
4) Show device ID
c) Clear the Keys
k) Manual Key Entry
g) Geographic Region: EU
x) Exit
x

SparkFun RTK EVK d99.99
** Bluetooth SPP and BLE broadcasting as: EVK Rover-8EF2 **
Menu: Main
1) Configure GNSS Receiver
2) Configure GNSS Messages
3) Configure Base
4) Configure Ports
5) Configure Logging
6) Configure WiFi
7) Configure TCP/UDP
e) Configure Ethernet
f) Firmware Update
i) Configure Corrections Priorities
n) Configure NTP
p) Configure PointPerfect
r) Configure Radios
s) Configure System
t) Configure Instrument Setup
u) Configure User Profiles
+) Enter Command line mode
x) Exit
x
[159074][D][NVM.ino:155] recordSystemSettingsToFileSD(): Removing from SD: /SFE_EVK_Settings_0.txt
[159158][D][NVM.ino:174] recordSystemSettingsToFileSD(): Settings recorded to SD: /SFE_EVK_Settings_0.txt
[159189][D][NVM.ino:204] recordSystemSettingsToFileLFS(): Removing LittleFS: /SFE_EVK_Settings_0.txt
[159574][D][NVM.ino:216] recordSystemSettingsToFileLFS(): Settings recorded to LittleFS: /SFE_EVK_Settings_0.txt
*WIFI_STATE_OFF
Network shutting down Ethernet, no users
networkStop called by C:\Users\pc235\Documents\GitHub\SparkFun_RTK_Everywhere_Firmware\Firmware\RTK_Everywhere\Network.ino 1204
Network layer stopping
Network State: NETWORK_STATE_IN_USE --> NETWORK_STATE_OFF
Network starting user MQTT Client on Active
Network request to start Active
Network layer starting Active
Network State: NETWORK_STATE_OFF --> NETWORK_STATE_DELAY
MQTT_CLIENT_ON --> MQTT_CLIENT_NETWORK_STARTED
Rover Accuracy (m): 0.164, SIV: 30 GNSS State: 3D Fix
Log file size: 2067121 - Generation rate: 100.1kB/s
networkTypeUpdate, network->type: Active --> Ethernet
networkTypeUpdate, network->requestedNetwork: Active --> Ethernet
Network starting Ethernet
Network State: NETWORK_STATE_DELAY --> NETWORK_STATE_CONNECTING
Network connected to Ethernet
Network State: NETWORK_STATE_CONNECTING --> NETWORK_STATE_IN_USE
Ethernet IP address: 192.168.0.91
MQTT_CLIENT_NETWORK_STARTED --> MQTT_CLIENT_CONNECTING_2_SERVER
File loaded from LittleFS: /SFE_EVK_certificate_0.txt
File loaded from LittleFS: /SFE_EVK_privateKey_0.txt
MQTT client connecting to pp.services.u-blox.com
[159849][D][NetworkManager.cpp:123] hostByName(): DNS found IPv4 63.33.125.32
[159856][V][ssl_client.cpp:68] start_ssl_client(): Free internal heap before TLS 81376
[159864][V][ssl_client.cpp:75] start_ssl_client(): Starting socket (domain 2)
[159969][V][ssl_client.cpp:166] start_ssl_client(): Seeding the random number generator
[159978][V][ssl_client.cpp:174] start_ssl_client(): Setting up the SSL/TLS structure...
[159986][V][ssl_client.cpp:194] start_ssl_client(): Loading CA cert
[159996][V][ssl_client.cpp:265] start_ssl_client(): Loading CRT cert
[160006][V][ssl_client.cpp:274] start_ssl_client(): Loading private key
[160017][V][ssl_client.cpp:288] start_ssl_client(): Setting hostname for TLS session...
[160027][V][ssl_client.cpp:301] start_ssl_client(): Free internal heap after mbedtls_ssl_setup 81196
[160036][V][ssl_client.cpp:311] ssl_starttls_handshake(): Performing the SSL/TLS handshake...
[160931][D][ssl_client.cpp:324] ssl_starttls_handshake(): Protocol is TLSv1.2 Ciphersuite is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
[160943][D][ssl_client.cpp:326] ssl_starttls_handshake(): Record expansion is 29
[160950][V][ssl_client.cpp:332] ssl_starttls_handshake(): Verifying peer X.509 certificate...
[160959][V][ssl_client.cpp:340] ssl_starttls_handshake(): Certificate verified.
[160966][V][ssl_client.cpp:356] ssl_starttls_handshake(): Free internal heap after TLS 81196
FreeHeap: 81196 / HeapLowestPoint: 79064 / LargestBlock: 3997684 / Used PSRAM: 142268
MQTT_CLIENT_CONNECTING_2_SERVER --> MQTT_CLIENT_SUBSCRIBE_KEY
MQTT client subscribed to key distribution topic
MQTT_CLIENT_SUBSCRIBE_KEY --> MQTT_CLIENT_SUBSCRIBE_SPARTN
MQTT client subscribed to corrections topic
MQTT_CLIENT_SUBSCRIBE_SPARTN --> MQTT_CLIENT_SERVICES_CONNECTED
Rover Accuracy (m): 0.165, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.164, SIV: 31 GNSS State: 3D Fix
Log file size: 2084657 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.163, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.161, SIV: 31 GNSS State: 3D Fix
Log file size: 2102322 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.161, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.160, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.159, SIV: 31 GNSS State: 3D Fix
*WIFI_STATE_OFF
Log file size: 2119987 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.158, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.158, SIV: 31 GNSS State: 3D Fix
Log file size: 2137652 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.158, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.157, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.156, SIV: 31 GNSS State: 3D Fix
Log file size: 2155197 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.156, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.156, SIV: 31 GNSS State: 3D Fix
*WIFI_STATE_OFF
Log file size: 2172727 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Log file size: 2190257 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Log file size: 2207795 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.155, SIV: 31 GNSS State: 3D Fix
Rover Accuracy (m): 0.153, SIV: 31 GNSS State: 3D Fix
*WIFI_STATE_OFF
Log file size: 2225325 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.671, SIV: 31 GNSS State: RTK Float
STATE_ROVER_FIX --> STATE_ROVER_RTK_FLOAT, 2024-06-27 13:29:33.515
Rover Accuracy (m): 0.224, SIV: 31 GNSS State: RTK Float
Log file size: 2243228 - Generation rate: 3.6kB/s
Rover Accuracy (m): 0.157, SIV: 31 GNSS State: RTK Float
Rover Accuracy (m): 0.120, SIV: 31 GNSS State: RTK Float
Rover Accuracy (m): 0.099, SIV: 31 GNSS State: RTK Float
Log file size: 2260479 - Generation rate: 3.5kB/s
Rover Accuracy (m): 0.087, SIV: 31 GNSS State: RTK Float
Rover Accuracy (m): 0.080, SIV: 31 GNSS State: RTK Float

I think this is just a cart-before-horse issue. If SFE_EVK_certificate_0.txt does not exist, we need to attempt ZTP to get it.

PaulZC commented 3 days ago

Has the keys states WiFi dependency been fixed? No. That's still there.

menuPointPerfect option 3 (Provision Device / Update Keys) has had a facelift. It now works for non-WiFi networks. I.e. Ethernet. pointperfectProvisionDevice is called if the certificate file does not exist or the current key or key distribution topic are null length or the certificates are invalid. That's excellent. We need a similar strategy for the states.

pointperfectUpdateKeys is different. It is only called when the certificates are present and valid. It gets the keys by subscribing to the MQTT key distribution topic (also provided during ZTP). It falls back to pointperfectProvisionDevice if it can't connect to the broker or the topic.

pointperfectProvisionDevice calls pointperfectTryZtpToken. pointperfectTryZtpToken does an HTTP POST to retrieve the certificates, key distribution topic, regional correction topics, and keys.

So, we have two things to do. The states also need to check if valid certificates etc. are present. And we need to fix the WiFi dependency.

PaulZC commented 3 days ago

pointperfectUpdateKeys uses a separate instance of NetworkClientSecure (secureClient) to connect to the PointPerfect broker via MQTT. It uses a separate instance of MqttClient (menuppMqttClient) on top of that secureClient. It isn't 'integrated' with the main MQTT functionality - subscribing to the correction topic. It isn't a network user. It does a blind connect and hopes that there is a network there to provide connectivity. If no network is connected, it will fail. It has no way to request and wait on a network before attempting to connect.

PaulZC commented 3 days ago

pointperfectProvisionDevice calls pointperfectTryZtpToken. pointperfectTryZtpToken also uses a separate instance of NetworkClientSecure (client). It uses a separate instance of HTTPClient (httpClient) on top of that client. It isn't a network user. It does a blind begin and hopes there is a network to provide connectivity. If no network is connected, it will fail. It has no way to request and wait on a network before attempting to connect.

I think we need to update both pointperfectUpdateKeys and pointperfectProvisionDevice so they use the network layer in a more integrated way...

PaulZC commented 3 days ago

MQTT_Client already subscribes to the pointPerfectKeyDistributionTopic. The keys arrive in UBX format and get pushed straight to the ZED. There's duplication / redundancy here we can use.

PaulZC commented 2 days ago

I'm in the process of adding a HTTP_Client to handle ZTP. Work in progress:

https://github.com/sparkfun/SparkFun_RTK_Everywhere_Firmware/commits/Fix_%23363/

nseidle commented 2 days ago

I believe the rolling MQTT Client start was fixed by Lee's PR merged a few minutes ago.

PaulZC commented 2 days ago

Agreed! I'll revert the new CHECK_CERTS state. And add the new timeout code to HTTP_Client.