immersive-web / webvr-polyfill

Use WebVR today, without requiring a special browser build.
http://immersive-web.github.io/webvr-polyfill/examples/
Apache License 2.0
1.4k stars 325 forks source link

Periodic stutter when turning head #168

Open epascal opened 8 years ago

epascal commented 8 years ago

I'm developping a 3D viewer in WebGL that works fine at 60fps very stable (added a stats display to ensure stability afterward). I enabled Gear VR developper mode for low persistence and it removed completely the blur phenomenon I had. Now I see clearly a phenomenon that is strange, there is every second a slight jump in the display as if the head pose skip a position. As I use pose prediction, it seems to me strange that the transitions are not perfectly smooth and the periodicity of the jump make me think that it's not due to external applications. Do you have any idea on the reason of this phenomenon. Except for that, it really a very good algorithm you have implemented. Here is my configuration:

window.WebVRConfig = WebVRConfig = { // Flag to disabled the UI in VR Mode. CARDBOARD_UI_DISABLED: true, // Default: false

// Forces availability of VR mode, even for non-mobile devices. FORCE_ENABLE_VR: false, // Default: false.

// Complementary filter coefficient. 0 for accelerometer, 1 for gyro. K_FILTER: 0.98, // Default: 0.98.

// Flag to disable the instructions to rotate your device. ROTATE_INSTRUCTIONS_DISABLED: false, // Default: false.

// How far into the future to predict during fast motion (in seconds). PREDICTION_TIME_S: 0.0, // Default: 0.040.

// Flag to disable touch panner. In case you have your own touch controls. TOUCH_PANNER_DISABLED: false, // Default: true.

// Enable yaw panning only, disabling roll and pitch. This can be useful // for panoramas with nothing interesting above or below. YAW_ONLY: false, // Default: false.

// To disable keyboard and mouse controls, if you want to use your own // implementation. MOUSE_KEYBOARD_CONTROLS_DISABLED: true, // Default: false.

// Prevent the polyfill from initializing immediately. Requires the app // to call InitializeWebVRPolyfill() before it can be used. DEFER_INITIALIZATION: false, // Default: false.

// Enable the deprecated version of the API (navigator.getVRDevices). ENABLE_DEPRECATED_API: true, // Default: false.

// Scales the recommended buffer size reported by WebVR, which can improve // performance. BUFFER_SCALE: 0.5, // Default: 0.5.

// Allow VRDisplay.submitFrame to change gl bindings, which is more // efficient if the application code will re-bind its resources on the // next frame anyway. This has been seen to cause rendering glitches with // THREE.js. // Dirty bindings include: gl.FRAMEBUFFER_BINDING, gl.CURRENT_PROGRAM, // gl.ARRAY_BUFFER_BINDING, gl.ELEMENT_ARRAY_BUFFER_BINDING, // and gl.TEXTURE_BINDING_2D for texture unit 0. DIRTY_SUBMIT_FRAME_BINDINGS: false // Default: false. };

borismus commented 8 years ago

Sounds like you might be dropping frames. You can check if this is happening in the devtools.

On Tue, Oct 25, 2016 at 9:51 PM epascal notifications@github.com wrote:

I'm developping a 3D viewer in WebGL that works fine at 60fps very stable (added a stats display to ensure stability afterward). I enabled Gear VR developper mode for low persistence and it removed completely the blur phenomenon I had. Now I see clearly a phenomenon that is strange, there is every second a slight jump in the display as if the head pose skip a position. As I use pose prediction, it seems to me strange that the transitions are not perfectly smooth and the periodicity of the jump make me think that it's not due to external applications. Do you have any idea on the reason of this phenomenon. Except for that, it really a very good algorithm you have implemented. Here is my configuration:

window.WebVRConfig = WebVRConfig = { // Flag to disabled the UI in VR Mode. CARDBOARD_UI_DISABLED: true, // Default: false

// Forces availability of VR mode, even for non-mobile devices. FORCE_ENABLE_VR: false, // Default: false.

// Complementary filter coefficient. 0 for accelerometer, 1 for gyro. K_FILTER: 0.98, // Default: 0.98.

// Flag to disable the instructions to rotate your device. ROTATE_INSTRUCTIONS_DISABLED: false, // Default: false.

// How far into the future to predict during fast motion (in seconds). PREDICTION_TIME_S: 0.0, // Default: 0.040.

// Flag to disable touch panner. In case you have your own touch controls. TOUCH_PANNER_DISABLED: false, // Default: true.

// Enable yaw panning only, disabling roll and pitch. This can be useful // for panoramas with nothing interesting above or below. YAW_ONLY: false, // Default: false.

// To disable keyboard and mouse controls, if you want to use your own // implementation. MOUSE_KEYBOARD_CONTROLS_DISABLED: true, // Default: false.

// Prevent the polyfill from initializing immediately. Requires the app // to call InitializeWebVRPolyfill() before it can be used. DEFER_INITIALIZATION: false, // Default: false.

// Enable the deprecated version of the API (navigator.getVRDevices). ENABLE_DEPRECATED_API: true, // Default: false.

// Scales the recommended buffer size reported by WebVR, which can improve // performance. BUFFER_SCALE: 0.5, // Default: 0.5.

// Allow VRDisplay.submitFrame to change gl bindings, which is more // efficient if the application code will re-bind its resources on the // next frame anyway. This has been seen to cause rendering glitches with // THREE.js. // Dirty bindings include: gl.FRAMEBUFFER_BINDING, gl.CURRENT_PROGRAM, // gl.ARRAY_BUFFER_BINDING, gl.ELEMENT_ARRAY_BUFFER_BINDING, // and gl.TEXTURE_BINDING_2D for texture unit 0. DIRTY_SUBMIT_FRAME_BINDINGS: false // Default: false. };

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/googlevr/webvr-polyfill/issues/168, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQ8gDeCJ5hgdT58kTnfwakLlUHQEIiTks5q3txRgaJpZM4Kgwcp .

epascal commented 8 years ago

I have optimized my code and the 60fps are totally stable. My analysis revealed that frame skipping where happening very seldom a lot less than the visual result, so the issue is not coming from there. After further analysis I understand what is happening, it seems that from time to time the head pose go back one step then continue in the direction my head is turning. As if you briefly press fast-backward then play. It's happing subtly but breaks the magic (the oculus low persistence mode reveal it very clearly). Oculus Gear VR experience though I know there sensor have a higher refresh rate is very very stable as if the scene was in front of you. I understand that it's not possible to reach the same perfection but it seems that solving the issue I described would bring the polyfill on step closer to this experience.

borismus commented 7 years ago

I don't know what the action is here.

dustinkerstein commented 6 years ago

@epascal did you ever resolve this issue? This is the same behavior I was describing in https://github.com/mrdoob/three.js/issues/9748

devmobileaim commented 6 years ago

I solved the issue using a Cordova plugin using the Google SDK. Kept WebVR for the 2d mode without headset.

dustinkerstein commented 6 years ago

@borismus Sorry about bringing this thread back from the grave, but do you have any ideas on what could be causing this? I'm just comparing these two sites on my Android phone:

  1. https://www.360cities.net/image/milky-way-galaxy-over-the-cloud-nine-surfing-area-general-luna-siargao-philippines
  2. https://webvr.info/samples/03-vr-presentation.html?polyfill=1

I am not sure what code 360 Cities uses, but their viewer doesn't seem to exhibit these "micro-stutters" when moving my phone, whereas the latter does. I've monitored the FPS on the polyfill examples and it's rock solid at 60fps so it's not that.

FYI, I do not see this issue when using the non-polyfill WebVR API on my Pixel XL during VR presentation (though this would be maybe tough to detect with the asynchronous reprojection).

dustinkerstein commented 6 years ago

@devmobileaim which Cordova plug-in did you end up using?

epascal commented 6 years ago

The one I made, but I never opensourced it, if you are interested I can share

On Mon, Feb 5, 2018 at 12:50 AM dustinkerstein notifications@github.com wrote:

@devmobileaim https://github.com/devmobileaim which Cordova plug-in did you end up using?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/immersive-web/webvr-polyfill/issues/168#issuecomment-362950972, or mute the thread https://github.com/notifications/unsubscribe-auth/ACJQC5vGDfV57ZOqsh46EX3qByGXsFvJks5tRkJcgaJpZM4Kgwcp .

dustinkerstein commented 6 years ago

@epascal that'd be really appreciated. Thanks! Feel free to post here or email to my username at Gmail. Thanks!

jsantell commented 6 years ago

What kind of device is this occurring on? Is this only occurring in WebViews when using Cordova? What browser/version are you seeing this drops in? And to confirm you're seeing the issue in the webvr.info samples when the polyfill is on?

dustinkerstein commented 6 years ago

This happens on all Android devices I have tested (Pixel, 6P, Moto Z) and has been like this for well over a year (since I started testing it). You should be able to directly reproduce on the current Chrome 63 Android (and beta and dev) using this link https://webvr.info/samples/03-vr-presentation.html?polyfill=1 - You'll notice small jitters that should be smoothed out with the sensor fusion of the polyfill, but they're not - and further, you'll occasionally see larger jumps that seem like frame drops, but they aren't.

Update - The above is for non-VR presenting. Ie. magic window mode which the above webvr sample supports by default. When in VR mode, either it's not an issue, or the asynchronous reprojection masks it.

jsantell commented 6 years ago

Note that the webvr.info samples link use origin trials, so even disabling WebVR on a Daydream device like Pixel will still enable the native VR display.

Trying a pixel on this example with no Origin Trials to use the polyfill directly, I'm not seeing the large frame drops, only some slight jittering when moving fast. To be honest, I'm not terribly familiar with the sensor fusion code in CardboardVRDisplay.

dustinkerstein commented 6 years ago

I can replicate the same thing on https://immersive-web.github.io/webvr-polyfill/examples

It just seems like there's something off, especially when compared to @borismus original sensor fusion code here - http://borismus.github.io/sensor-fusion and comparing alternative sensor fusion based devicemotion implementations like https://www.360cities.net/image/milky-way-galaxy-over-the-cloud-nine-surfing-area-general-luna-siargao-philippines and https://krpano.com/plugins/gyro2/gyro1-gyro2-comparison/

Any ideas on who to contact to dig in a little further? I'd be happy to help get more debug if you point me in the right direction.

jsantell commented 6 years ago

Boris worked on the sensor fusion components for the polyfill, so I imagine it being similar to that sensor fusion example. If that example does not have any stuttering, nor the 360cities/krpano examples, that should be something to go off of at least. I'll check around. If you're interested, the fusion-pose-sensor is what handles this in the polyfill

devmobileaim commented 6 years ago

I tweaked the cordova-plugin-device-sensor-fusion to use google cardboard and overcome this issue. Here is the link to the modified version : http://kepascal.free.fr/cordova-plugin-device-sensor-fusion.zip

dustinkerstein commented 6 years ago

Sounds good @jsantell - Let me know how I can help. Thanks!

@devmobileaim Thanks for posting the Cordova (Android Java) plugin. Though I still am hoping to figure out a WebVR-based solution.

dustinkerstein commented 6 years ago

FYI, this can also be replicated on iOS in Mobile Safari (tested on an iPhone 6 Plus and iPhone 5s) and the jitter effect is in fact much more severe on these devices even though the FPS meter is rock solid at 60fps.

Here's a crappy portrait phone video that attempts to show this effect - https://drive.google.com/open?id=1aDm4htS5ZSBs38-nudfdNhDc2LbaNywi - Make sure you actually download the video as the Google Drive preview is only 30fps whereas the video is actually 60fps. You should be able to easily compare the background environment smoothness (ie. behind the phone) with the jittery "60fps" rendering on the phone.

In my personal opinion, the polyfill experience causes a bit too much visual discomfort for devices that rely on the polyfill for VR headset rendering (which are probably the majority at this moment).

Update - I also just tested the latest polyfill build on those iOS devices (compared to the webvr.info build version - 0.9.41) and the behavior is the same.

dustinkerstein commented 6 years ago

Some more notes - I can replicate this even with 100% gyro and no prediction via K_FILTER: 1 and PREDICTION_TIME_S: 0 in the polyfill config.

Here's a very simple demo that shows the jitter / stutter with the default K_FILTER and PREDICTION_TIME_S - https://files.panomoments.com/js/polyfill.html - Feel free to use / modify it for testing.

Update - On iOS I sometimes notice that the stuttering gets worse over time (give it 30 seconds or so of panning around) and that I can sometimes get to fix itself somewhat by tapping the screen... but then it will happen again in a few seconds. Android doesn't seem to behave like this.

dustinkerstein commented 6 years ago

Hey! Just checking in. @jsantell, were you able to dig up anything? Let me know if there's anything I can help with.

jsantell commented 6 years ago

@dustinkerstein thanks for being on the ball with this! Just getting back from some time off, so picking this up again today :smile:

dustinkerstein commented 6 years ago

@jsantell Nice! I think @cvan might also be taking a look. I pinged him on Twitter a couple days ago as I'm able to replicate this same issue on the A-Frame examples. Let me know if I can help with anything.

jsantell commented 6 years ago

Was exploring this earlier, some notes:

A bit stumped with this. Looking through the sensor-fusion demo repo, it's mostly the same code used here. Unsure if I could detect the "blip" via the visualization on this demo, to be honest.

dustinkerstein commented 6 years ago

A bit stumped with this. Looking through the sensor-fusion demo repo, it's mostly the same code used here. Unsure if I could detect the "blip" via the visualization on this demo, to be honest.

I was in fact able to just replicate this using the sensor-fusion demo repo on my Moto Z. It's harder to see but the easiest way I found was to spin in a swivel chair at a constant speed while watching the blue line. You can easily see small little "stair steps" that appear at the frequency I'm used to seeing the jitters. Interestingly the blips don't seem to correlate with the other lines on the graph and it doesn't seem to respond at all to the kFilter or predictionTime variables. Not sure that will help much, but maybe it's something.