Competition Day 3 - post mortem

[smnschfr]

Hi @SijmenHuizenga , we've had a look at the recorded data from today's Trackdrive event and tried to find out why our system failed. Our first conclusion is that the GSS velocity data was less accurate than it used to be. Because our SLAM gave a high weight to this data, the localisation became incorrect.

Do you have any ideas what might have caused this? Perhaps some change in the system, a higher load due to more spectators?

Best regards Simon from KA-RaceIng

P.S.: Thanks for making this simulation event possible at all!

[SijmenHuizenga]

Hi Simon, thanks for looking into this! I was surprised as well why KA-RaceIng didn't finish trackdrive. Let's try to find the root cause!

I can't think of any differences between day 2 and 3, apart from the map. There were an equal number of spectators: previous days only my computer and on the final day only the livestreaming computer at the studio. The same instances were used in the same google cloud region and zone. There were no changes to the bridge or simulation and the settings.json used was the same as the one during previous days.

Can you share your observations leading up to the "GSS less accurate" statement? I would love to know how you determine accuracy of GSS information! I would love to compare the rosbags with other teams and multiple days to see if there is a pattern.

Maybe there is something inside the last competition map that causes this, but now I'm just guessing. To prove or disprove this, we can test map 2 and map 3 on a standalone computer to see if there is any difference?

What boggles my mind most is that KA-RaceIng and MITeamDelft both were able to finish autocross but kissed a corner in the first lap of autocross. This would indicate that something changed between autocross and trackdrive. Are you able to see any differences in sensordata between autocross and trackdrive?

[smnschfr]

Hi Sijmen,

thanks for looking into this as well. I think a build containing the map would be useful. Then we could test if we're able to reproduce the issue locally.

We haven't analyzed the data in detail. The "GSS less accurate" theory is from an experiment where we replayed the recorded sensor data from the trackdrive and gave a higher uncertainty to the velocity data (and rely more cone observations).

In the following visualization, the light blue path is the odometry-based path and the green path is the path corrected with more weight on cone observations. It seems the recording of the live stream isn't available (yet?), but the green path matches my memory of the actual performance much better.

Also already in the AutoX, it seems the pose jumped much more than usually in this simulation, indicating more corrections were necessary.

[SijmenHuizenga]

@smnschfr Testing accuracy like that is super cool! It seems like the corrected (green) path is shifted to the right in relation to the odometry-based (blue) path. Throughout the run this offset seems pretty stable. I'm at a loss to what could cause this, do you have any ideas? Do you see the same offset during trackdrive & autocross?

A built version with map 3 included is now available (v1.4.1). Maybe you can try to reproduce it?

I also did some testing:

• Drove around in competition map 2 and 3 while watching the fps (stat fps) and game loop statistics (stat UnitGraph). Performance seems to be identical on both maps.
• Tried running a spectator and using these parameters to simulate network delay between the simulator and spectator. This is something that was different between competition day 2 and 3: on day 1 and 2 the spectator ran in Delft while at competition day 3 the spectator ran in Budapest both times connecting to the same virtual computer in a Google datacenter in the Netherlands. Although unlikely, I thought this delay might influence performance. Looking at plots of GSS and odometry I can't find any prove of this theory. (yes this is very non-scientifically but I don't know how to better test this in a reasonable time).
• Looked at the google cloud statistics about the instances (cpu, memory and network usage) and didn't see anything out of the ordinary for day 3.

Regarding the unavailability of the livestream, I asked FSOnline and apparently it is only limited to 2 hours for some users, but not for everyone. Someone at FSOnline is looking into it.

[smnschfr]

Thanks for uploading the new maps. I couldn't reproduce the same catastrophic failure locally yet (with a few attempts), but our system is having some issues with this map already in autocross. I think at least partly this is due to the fact that the system assumes the vehicle is staged 6m behind the starting line at the center of the track. However, due to the curve just before the finish line, the vehicle is staged at the righthand boundary of the track instead.

[SijmenHuizenga]

The distance between playerstart and start-finish line is around the same as the other tracks. In hindsight we probably should have moved the playerstart more to the center of the track. I'm sorry if this caused your DNF.

Is there anything else you would like to investigate regarding this issue?

[smnschfr]

I'll run another test or two on this map, but I don't really expect to get a final answer. Probably a combination of assumptions in our system that didn't quite hold combined with bad luck.

[SijmenHuizenga]

Thanks for looking into this whole thing!