Radar accuracy/calibration

[maciejwie]

Hi @zacharyedwardbull, thanks for putting together the library and @tensorturtle for adding radar support. I wrote a little hobby application (https://github.com/maciejwie/radar-logger) using your work this weekend and it was (mostly) pretty painless. Good news is that I can add a datapoint that the RTL515 works.

One issue I ran into was around the accuracy of the data returned by the sensor, which consistently read high. For example, detected pedestrians are reported as moving at 10-11 km/h where a more likely value is ~5 km/h. I also asked someone to drive at consistent and known speeds and the data there as well was pretty consistently 5 km/h high.

Below are some data I collected using my application. The data section is a list of [speed, distance] tuples. All of the speeds also fall off as the vehicle gets close to the sensor but I believe that's just behaviour of the radar.

Pass 1, vehicle traveling at 30 km/h. Data:

{"timestamp": "2023-06-04 16:02:02", "max": 37, "95% max": 37, "average": 34.1, "data": [[37, 93], [37, 92], [37, 91], [37, 90], [37, 88], [36, 87], [36, 86], [36, 85], [36, 84], [36, 83], [36, 82], [36, 81], [36, 80], [35, 78], [35, 77], [35, 76], [35, 75], [35, 74], [35, 73], [35, 72], [35, 71], [35, 70], [35, 69], [35, 68], [35, 67], [35, 66], [36, 65], [36, 64], [36, 63], [36, 62], [36, 61], [36, 59], [36, 58], [36, 57], [36, 56], [36, 55], [36, 53], [36, 52], [37, 51], [37, 50], [37, 49], [37, 47], [37, 47], [37, 46], [37, 45], [37, 44], [37, 43], [37, 41], [37, 39], [36, 37], [36, 35], [36, 34], [35, 33], [35, 32], [35, 30], [35, 29], [35, 28], [35, 27], [34, 25], [34, 24], [34, 22], [34, 21], [34, 20], [34, 19], [34, 18], [34, 17], [34, 16], [34, 15], [34, 14], [33, 14], [33, 13], [33, 13], [33, 12], [33, 12], [33, 11], [33, 10], [33, 7], [34, 6], [34, 5], [33, 4], [33, 4], [32, 4], [30, 3], [29, 3], [28, 3], [26, 3], [26, 3], [24, 3], [23, 3], [23, 3], [23, 2], [23, 1], [23, 0]]}

Pass 2, vehicle traveling at 20 km/h. Data:

{"timestamp": "2023-06-04 16:05:04", "max": 26, "95% max": 26, "average": 23.4, "data": [[25, 95], [25, 94], [25, 93], [25, 92], [25, 91], [25, 91], [25, 90], [25, 89], [24, 88], [24, 87], [24, 86], [24, 86], [24, 85], [24, 84], [24, 83], [24, 83], [24, 82], [24, 81], [24, 81], [24, 80], [24, 79], [24, 78], [24, 78], [24, 77], [24, 76], [24, 76], [24, 75], [24, 74], [24, 74], [24, 73], [24, 72], [25, 72], [25, 71], [25, 70], [25, 69], [25, 69], [25, 68], [25, 67], [25, 67], [25, 66], [25, 65], [25, 64], [25, 63], [24, 63], [24, 62], [24, 61], [24, 61], [24, 60], [24, 60], [24, 59], [23, 58], [23, 57], [23, 56], [23, 55], [23, 54], [23, 53], [23, 53], [24, 52], [24, 52], [25, 51], [25, 50], [26, 49], [26, 49], [26, 48], [26, 47], [26, 46], [26, 45], [26, 45], [26, 44], [26, 43], [26, 42], [26, 41], [26, 39], [26, 38], [26, 37], [25, 36], [25, 35], [25, 34], [25, 33], [25, 32], [25, 31], [25, 30], [25, 29], [25, 29], [24, 28], [24, 27], [24, 26], [24, 25], [24, 25], [24, 24], [24, 23], [24, 22], [24, 21], [24, 20], [24, 19], [24, 19], [24, 18], [24, 17], [23, 16], [23, 15], [23, 15], [23, 14], [23, 13], [23, 13], [23, 12], [23, 12], [23, 11], [22, 11], [22, 10], [22, 10], [22, 9], [22, 9], [22, 8], [22, 7], [22, 6], [21, 5], [21, 5], [21, 5], [21, 4], [21, 4], [21, 4], [20, 4], [20, 4], [20, 4], [20, 4], [20, 4], [19, 4], [19, 4], [19, 4], [18, 4], [17, 4], [17, 4], [17, 3], [17, 3], [17, 2], [17, 1], [17, 1], [17, 0]]}

Pass 3, vehicle traveling at 35 km/h. Data:

{"timestamp": "2023-06-04 16:07:36", "max": 43, "95% max": 43, "average": 39.5, "data": [[43, 99], [43, 98], [43, 97], [43, 96], [43, 95], [43, 93], [43, 92], [43, 91], [43, 89], [43, 88], [43, 87], [43, 85], [43, 84], [43, 83], [43, 82], [43, 80], [42, 79], [42, 77], [42, 76], [42, 75], [41, 74], [41, 72], [41, 70], [41, 69], [41, 68], [41, 67], [41, 65], [41, 64], [41, 63], [41, 62], [41, 61], [41, 60], [41, 58], [41, 58], [41, 56], [41, 55], [41, 54], [41, 53], [41, 52], [41, 50], [40, 49], [40, 47], [40, 46], [40, 45], [40, 44], [40, 42], [40, 41], [39, 39], [39, 38], [39, 36], [39, 35], [39, 33], [39, 31], [39, 29], [39, 28], [39, 27], [39, 25], [39, 24], [39, 22], [39, 21], [39, 19], [39, 18], [40, 17], [39, 15], [39, 14], [39, 13], [39, 12], [39, 11], [39, 10], [38, 10], [38, 9], [38, 8], [38, 6], [37, 5], [37, 4], [37, 4], [36, 3], [35, 3], [34, 3], [33, 3], [31, 3], [28, 2], [27, 2], [27, 1], [27, 0]]}

Pass 4, vehicle traveling at 40 km/h. Data:

{"timestamp": "2023-06-04 16:11:05", "max": 49, "95% max": 49, "average": 45.3, "data": [[48, 111], [48, 109], [48, 108], [49, 107], [49, 105], [49, 104], [49, 102], [49, 101], [49, 99], [49, 97], [49, 96], [49, 94], [49, 92], [49, 91], [49, 89], [49, 87], [49, 86], [48, 84], [48, 83], [48, 82], [48, 80], [47, 78], [47, 77], [47, 76], [47, 74], [46, 73], [46, 72], [46, 71], [46, 69], [46, 68], [45, 66], [45, 65], [45, 63], [45, 62], [45, 61], [45, 60], [45, 58], [45, 56], [45, 55], [45, 53], [45, 52], [45, 50], [45, 49], [46, 47], [46, 46], [46, 45], [46, 43], [47, 42], [47, 40], [47, 39], [47, 37], [47, 35], [47, 33], [47, 31], [46, 30], [46, 28], [46, 26], [45, 25], [45, 23], [45, 21], [45, 19], [45, 17], [45, 16], [44, 14], [44, 13], [44, 12], [44, 11], [43, 10], [43, 9], [43, 7], [42, 6], [42, 4], [42, 4], [41, 3], [41, 3], [40, 3], [39, 3], [38, 2], [34, 2], [31, 1], [31, 0]]}

In your testing, did you see something similar? I added calibration parameters on my end to handle this, but am wondering if it's something that affects more people upstream.

I also looked at how other projects handle this. The only open ones which uses BLE (most use ANT+) I found is OpenWorkoutTracker and they don't use the speed bytes at all and instead calculate them using distance and time. See: https://github.com/msimms/OpenWorkoutTracker/blob/4c42b61c5169208026d0d9aaa794079a05d2296e/IOS/Sensors/BtleDiscovery.m#L283 and Tacho (https://github.com/Wunderfitz/harbour-tacho/blob/cc7e23ab7c97a11d32acbf7688ed5233060d7dba/src/variaconnectivity.cpp#L161) which uses the values as-is like we do.

MyBikeTraffic is a CIQ app which uses the Garmin API (which returns m/s for speed) and also uses it as-is (after converting to km/h or mph): https://github.com/kartoone/mybiketraffic/blob/4af398d180085f58dd0aa3bd2c695ee6def5ddb4/source/MyBikeTrafficFitContributions.mc#L126

There is also closed-source ARadar on watchOS which should also use BLE (https://robertoviola.cloud/2022/05/27/garmin-varia-on-apple-watch-finally/ ) and who may also have some insight or experience with it.

I think the threat_id is also not quite right and I only saw values between 128 and 191, so it seems like only the bottom 6 bits are used for this. Not sure what the upper 2 bits might be. The ANT+ documentation (https://developer.garmin.com/connect-iq/api-docs/Toybox/AntPlus/RadarTarget.html) references 2-bit THREAT_SIDE and THREAT_LEVEL (https://developer.garmin.com/connect-iq/api-docs/Toybox/AntPlus.html#ThreatLevel-module) but those may go in byte 0.

Don't have any patches to submit, just looking to compare notes right now.

[tensorturtle]

Hi @maciejwie thanks for your kind words and your project looks exciting! Unfortunately I did not do any meaningful speed accuracy testing. I think you're probably right about threat_id.

[zacharyedwardbull]

Hey! Thanks for doing this detailed investigation and writeup. Sadly I don't have a radar so I haven't been able to do any work on this myself. If you work out what needs updating please create a CL and I'll merge it in!

Cheers, Zach