As many people already mentioned, training on the previous image, or the next angle/throttle, i.e. shifting the y in (x,y) forward by one index, gives a better and forward looking behavior which looks like having more "planning" and less "observe and react if you can". Given that the inference time makes up the largest part of a single vehicle loop, it does make sense to use the shift. It is easy to implement, because we support the sequence models which grab a continuous list of (x, y), so we can adjust this to grab a set of two consecutive data and use image from the first and steering, throttle from the second entry.
Shifting my a factor of 4 helps for online races anecdotally (compensating for frame lag)
Shifting in general improves fit quite substantially so it's definitely worth adding and you probably want it configurable
As many people already mentioned, training on the previous image, or the next angle/throttle, i.e. shifting the y in (x,y) forward by one index, gives a better and forward looking behavior which looks like having more "planning" and less "observe and react if you can". Given that the inference time makes up the largest part of a single vehicle loop, it does make sense to use the shift. It is easy to implement, because we support the sequence models which grab a continuous list of (x, y), so we can adjust this to grab a set of two consecutive data and use image from the first and steering, throttle from the second entry.
Shifting my a factor of 4 helps for online races anecdotally (compensating for frame lag) Shifting in general improves fit quite substantially so it's definitely worth adding and you probably want it configurable