Lot of RTK Fix lost since firmware 3.6

Stefal commented 1 year ago

Subject of the issue

Since firmware 3.9, I noticed i have much more RTK fix lost. Much more than v3.6. After some investigations it looks like there is a problem with the rtcm transfer to the F9P when the base station send too much messages for the Surveyor/Express/Facet, as I don't see this problem with base station that send less messages.

Your workbench

RTK Surveyor/Express/Facet
firmware 3.6, 3.8, 3.9, 3.10
Default settings
Steps to reproduce

Connect your rover to any standard base station with many messages (you can try with the mount point EMONIE on caster.centipede.fr:2101) If you're too far away from this base station, you won't get a fix, but you can monitor the correction age, which goes very high.

You won't see this problem with a GPS/Glonass only base station.

(Plots with RTKPlot and the nmea file logged by the rover)

Correction age with 3.6

Correction age with 3.8

Correction age with 3.9

Correction age with 3.10

I'm sure you can fix this bug, but I hope the hardware still have some available bandwidth, or we will have some problems with the coming L1/L2/L5 base station which will send even more data.

Stefal commented 1 year ago

Hmm, strange, with a brand new RTK Express with firmware 3.8 I have the same problem : A retest with the surveyor that was successful during my previous test and 3.6/3.8, failed today. Perhaps I was near the bandwith limit, and now at this exact time, the base station is receiving more sats than sunday.

Stefal commented 11 months ago

If you want to test a base station with a lot of data, you can use the mount point CT02 on caster.centipede.fr:2101 It's a L1/L2/L5 Mosaic X5 receiver. The bandwidth is about 70% higher than the other centipede base station.

nseidle commented 11 months ago

After some investigations it looks like there is a problem with the rtcm transfer to the F9P when the base station send too much messages for the Surveyor/Express/Facet, as I don't see this problem with base station that send less messages.

Good detective work! Thank you. Are you running an RC or a full release firmware? I ask because the RC firmware should output additional debug messages that may help us.

The Bluetooth link is getting overwhelmed. This bluetooth library we use has changed a bit over the months and years. I suspect something is broken. I'll try to get this tested this week.

Stefal commented 11 months ago

I've installed a RC, and here is what I see with CT02 (Mosaic X5 L1/L2/L5) :

Rover Accuracy (m): 2.4817, SIV: 12
[1539446][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
[1540603][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
[1540651][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 590 bytes
[1540652][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 6 bytes
Batt (76%): Voltage: 4.00V Charging: 9.57%/hr Green
Rover Accuracy (m): 2.4848, SIV: 12
[1541540][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 316 bytes
[1541623][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 190 bytes
[1541624][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 6 bytes
[1542764][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
[1542856][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 90 bytes
[1542856][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 6 bytes
Rover Accuracy (m): 2.4866, SIV: 12
[1543466][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
[1543571][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 190 bytes
[1543571][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 6 bytes
[1544699][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
[1544742][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 339 bytes
Rover Accuracy (m): 2.4849, SIV: 12
[1545515][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes

and with EMONIE (F9P L1/L2)

Rover Accuracy (m): 1.3403, SIV: 11
[1732918][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 46 bytes
[1733742][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 55 bytes
Rover Accuracy (m): 1.3424, SIV: 11
[1734548][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 65 bytes
Rover Accuracy (m): 1.3525, SIV: 11
[1736496][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
Batt (76%): Voltage: 4.03V Charging: 4.99%/hr Green
[1737826][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 65 bytes
Rover Accuracy (m): 1.3603, SIV: 11
[1738644][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 55 bytes
[1739696][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 55 bytes
Rover Accuracy (m): 1.3626, SIV: 11
[1740587][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 110 bytes
[1741819][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 167 bytes
Batt (76%): Voltage: 4.03V Charging: 4.99%/hr Green
Rover Accuracy (m): 1.3599, SIV: 11
[1742652][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 55 bytes
Rover Accuracy (m): 1.3561, SIV: 11
[1744825][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 171 bytes
[1745610][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 416 bytes
[1746428][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 203 bytes
Rover Accuracy (m): 1.3578, SIV: 11
Batt (76%): Voltage: 4.03V Charging: 4.99%/hr Green
[1747631][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 55 bytes
Rover Accuracy (m): 1.3552, SIV: 11
[1749704][E][BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 55 bytes
Rover Accuracy (m): 1.3470, SIV: 11

nseidle commented 11 months ago

Indeed, there's the problem. The BT Serial buffer should be large (2k I think is the default) but something is wrong. I'll have a look.

nseidle commented 11 months ago

Ok, I think I found the problem. 6 months ago we changed the default size of the Bluetooth SPP buffer from 2k, down to 1k to save RAM. It should likely be increased again.

Please try this:

Press a key to open the config menu
's' for the system menu
'o' to configure RTK operation
'7' to set the SPP RX buffer size.
Increase this to 2048 bytes
exit all menus (so the settings are saved)
power cycle or reset the device
try connecting to your mount point

After doing the above, all the [BluetoothSerial.cpp:330] esp_spp_cb(): RX Full! Discarding 172 bytes warnings went away and RTK Fix was achieved faster, and maintained.

This error is affecting, slightly, all units, but most units can maintain RTK Fix. It's really only a problem when a mount point is as vociferous as the mount points you're using.

I'll increase the default do a RC release shortly.

tonycanike commented 11 months ago

Thanks for the fast turnaround Nate!

For a moment I thought this might have been causing problems I had been seeing, but I'm using radios between my base and rover Facets so not related. I simply continue to challenge the range of the 900MHz 1W radios in mountainous terrain. I hope to finally test the L-Band in this challenging terrain next week.

nseidle commented 11 months ago

Happy to help! Sorry for the re-emergent issue.

Stefal commented 11 months ago

I did some quick tests yesterday. That was really better. I still had some RX Full message with the L1/L2/L5 base station. I'll try to log for several hours in the next days.

nseidle commented 11 months ago

Excellent. I've increased the default to 2k in the RC. Feel free to increase the buffer for your specific rover. We have to be careful with our RAM usage.

Stefal commented 11 months ago

Tests results

Setup

RTK Express with firmware 3.11RC
base station CT02 on caster.centipede.fr. Mosaic X5 (L1/L2/L5), all bands and messages enabled, 1hz, msm7 messages, 20kb/s
- rtcm messages: 1004, 1005, 1006, 1008, 1012, 1019, 1020, 1033, 1042, 1045, 1046, 1077, 1087, 1097, 1107, 1117, 1127, 1137, 1230
base station EMONIE on caster.centipede.fr. F9P (L1/L2), all bands and message enabled except SBAS, 1hz, msm7 messages, 12kb/s
- rtcm messages: 1004,1005(10),1006,1008(10),1012,1019,1020,1033(10),1042,1045,1046,1077,1087,1097,1107,1127,1230

3 hours log, connected on CT02 : 3746 "RX Full" messages => 1250 "RX Full" per hour. correction age plot :

4,5 hours log, connected on EMONIE : 38 "RX Full" messages => less than 9 "RX Full" per hour. correction age plot :

My conclusion

I think we can say that a 2k buffer is ok for a F9P base station, but too low for a Mosaic X5 base station.

Stefal commented 11 months ago

A RTK Facet user told me he still has some high correction age with the 2048 buffer. I will try a new log but during 24H.

Stefal commented 11 months ago

Near 24H log, but I forgot to log the serial output to catch the RX full messages

nseidle commented 11 months ago

Should be fixed in v4.0. Let me know if there's still issues.

sparkfun / SparkFun_RTK_Firmware