Client-side prediction

[JeroMiya]

This issue is to discuss the API design for client-side prediction. Currently we have two proposals:

Proposal 1: Introduce new getPredicted*State functions to get the predicted state, which pass the prediction time (relative or absolute? configurable with an enum?), and perhaps the velocity (from where?), and return the state predicted to the target time using client side information.

Opt-in. Potentially results in applications with mismatched prediction times, and some state being predicted and other state being raw. Existing applications need to change to reduce judder everywhere. Troublesome for client applications to keep track of a prediction time and pass it around. Easiest to implement.

Proposal 2: Introduce new functions to set a Client Context scoped value for prediction time. Interfaces internally track velocity, and the existing get*State functions would now return the predicted state as opposed to the last report. We might then still introduce getPredicted*State functions for more explicit control over prediction time, or to get the raw state value.

Opt-out. Ensures most applications will have consistent prediction times and less judder by default. Does change the behavior of existing API (e.g. state API no longer returns the same value as the last callback), but most apps should still be fine. New state functions give applications explicit control only if they need it.

[russell-taylor]

Context: We already have something like Proposal 2 set up with the HDK, except that we don't have a run-time way to adjust the prediction. There is a prediction time set up in the code that constructs it that adds a dead-reckoning orientation-prediction tracker and makes it the default /me/head. This can be overridden with a dead-reckoning tracker of arbitrary (but still fixed) delay and adding another alias explicitly in the configuration file. This is basically using the configuration tree to insert prediction in a way that the application does not have to know about to benefit from.

The default HDK configurations insert a fixed-duration server-side predictor for /me/head when an OSVR HDK is configured. This is incompatible with client-side prediction, which expects a pose + velocity pair time stamped at the original message time. Enabling client-side prediction currently requires overriding the /me/head alias in the config file to point it back at the base tracker.

We should consider what we want to have be the default behavior going forward.

(I've added the above as another issue on Core.)

[russell-taylor]

Context: Client applications that are using RenderManager will have access to client-side predictive tracking without changes to their code; they just need to use a server configuration file that has client-side prediction configured. RendeManager handles doing the prediction along with the rest of its functions and hands back predicted times to the application for rendering.

[russell-taylor]

Core applications will not be able to reduce judder over the existing fixed-prediction interval unless they adjust the prediction interval every frame based on how long it will be until the frame presents itself. This means that their existing code will have to change to make use of the improved prediction. This means that we may as well expose the opt-in interface so that they can decide whether they prefer OSVR Core to do their prediction or whether they prefer to do their own. In fact, consider:

Proposal 3: Introduce a new PredictFuturePose(Pose_in, Pose_velocity_in, time_seconds_in, Pose_Out) function that the application can use if it wants to. The function should be robust to having invalid velocity information, just returning the original pose in that case. This function can be lifted from RenderManager into Core https://github.com/sensics/OSVR-RenderManager/blob/master/osvr/RenderKit/RenderManagerBase.cpp#L99 and renamed appropriately (perhaps including the fact that it is doing dead-reckoning).

Trying to change the state interface seems like it would require a separate function call each frame to set the prediction interval, and the client may as well just make a call to do the prediction. We can include this in the example programs and recommend it as standard practice.

[JeroMiya]

Applications will need to change for both proposals. In Proposal 1, they only need to change in ONE place (the part of the application that actually knows how long it will be until the frame presents itself) vs proposals 2 and 3 where EVERY part of the application that wants to get the "right" interface state to use at that point in time has to know what to pass in for velocity and time in seconds. Most applications won't bother or know that they need to bother, or won't have access to the render timing info from that part of the program or the velocity value to use (by the way, I don't support requiring velocity values for any proposal - those should be handled by the interface internally). In all such cases, they will continue to use the old state functions and get stale, judder-ed state.

Another problem with proposals 2 and 3 is that the old state and callback APIs are now "poisonous" in the sense that you really shouldn't use them in most cases. It's like the current DisplayConfig API in OSVR-Core: if you're using the RenderManager, you can't use the DisplayConfig values for surface sizes or projection matrices because that API will just give you invalid values every time (unless the overfill factor is 1.0 and yadda yadda...etc...). Now, with prediction turned on in RenderManager, you can't even use the head/eye poses from the DisplayConfig API because those will be wrong too, which begs the question of why we even have that API if nobody should be using it.

In my view it is MUCH better to have everything work and be consistent by default. The state and callback APIs people have been using all along should return the same values as the DisplayConfig API and the RenderManager given poses. If the odd client app needs to get the raw values, introduce the new API to get the raw values, or the predicted values to a different time in the future, or with a different velocity, but by default everything should just work without all this ceremony.

[JeroMiya]

Applications will need to change for both proposals. In Proposal 1, they only need to change in ONE place (the part of the application that actually knows how long it will be until the frame presents itself) vs proposals 2 and 3 where EVERY part of the application that wants to get the "right" interface state to use at that point in time has to know what to pass in for velocity and time in seconds. Most applications won't bother or know that they need to bother, or won't have access to the render timing info from that part of the program or the velocity value to use (by the way, I don't support requiring velocity values for any proposal - those should be handled by the interface internally). In all such cases, they will continue to use the old state functions and get stale, juddery state.

Another problem with proposals 2 and 3 is that the old state and callback APIs are now "poisonous" in the sense that you really shouldn't use them in most cases. It's like the current DisplayConfig API in OSVR-Core: if you're using the RenderManager, you can't use the DisplayConfig values for surface sizes or projection matrices because that API will just give you invalid values every time (unless the overfill factor is 1.0 and yadda yadda...etc...). Now, with prediction turned on in RenderManager, you can't even use the head/eye poses from the DisplayConfig API because those will be wrong too. If an API shouldn't be used because it almost always gives you the wrong values, either it should be changed to give you the right answer or it should be eliminated.

In my view it is MUCH better to have everything work and be consistent by default. The state and callback APIs people have been using all along should return the same values as the DisplayConfig API and the RenderManager given poses. If the odd client app needs to get the raw values, introduce the new API to get the raw values, or the predicted values to a different time in the future, or with a different velocity, but by default everything should just work without all this ceremony.