VR-Mode for Videosphere does not work on iPad iOS 14.3 for A-Frame version 1.1.0 and 1.2.0

[bryancsc-umich]

Description: VR-Mode for Videosphere does not work on iPad iOS 14.3 for A-Frame version 1.1.0 and 1.2.0

• A-Frame Version: 1.1.0 and 1.2.0
• Platform / Device: iOS 14.3 / iPad Air 2
• Reproducible Code Snippet or URL: (See Video Below)

Not sure if this was intentional but VR-mode doesn't work on iPad Air 2 running iOS 14.3 with A-Frame version 1.1.0 and 1.2.0. (Example shown below.)

The videosphere is also unable to use a canvas element as its src beginning A-Frame version 0.8.0.

https://user-images.githubusercontent.com/55333712/107645407-a237ab80-6c46-11eb-9eeb-f21496cebe29.mp4

[dmarcos]

Canvas as video src is no longer supported because of performance reasons.

On iPad we should not enter VR / Cardboard mode since it's not useful.

[bryancsc-umich]

Ah. That makes sense for no VR/Cardboard on the iPad.

I am an amateur at this but I feel that not having canvas as a video src severely limits the videosphere as you can't apply transformations on the video src to the videosphere.

We can use captureStream on the canvas element and stream the canvas to a video element and then to the videosphere but that just adds another layer of inefficiency than streaming directly from the canvas. Is there a method I am unaware of? Else, might be nice to allow some way for people to apply transformations before streaming to the videosphere.

[dmarcos]

Unfortunately, a canvas is not going to yield good performance. What transformations you need to apply to the video?

You can look into shaders to apply effects to the video. This is an example to look at: https://github.com/nikolaiwarner/aframe-chromakey-material.

Keep in mind that a videosphere is just a shorthand for:

<a-entity 
   geometry="primitive: sphere; radius: 500; segmentsWidth: 64; segmentsHeight: 32" 
   material="color: white; shader: flat; side: back; npot: true" 
   scale="-1 1 1">
</a-entity>

[bryancsc-umich]

Thanks for the quick response! I will certainly look into shaders!

Answer to your question: I am actually streaming a distorted video and am transforming it into equirectangular image via a canvas before passing it into the videosphere.

Like CV stuff.

[bryancsc-umich]

Hi! Don't know if I should open a new issue or not as it seems related to https://github.com/aframevr/aframe/issues/4752 .

I finally got my hands on an iPhone running on iOS 13.4 and tried out VR mode for the examples at:

https://aframe.io/aframe/examples/boilerplate/panorama/

and

https://aframe.io/aframe/examples/test/videosphere/

Unfortunately, the videosphere's VR mode behaves the same way as on the iPad (VR mode just shows a white screen)

while the panorama mode initially shows the image but quickly blanks out after rotating past a certain angle.

Videos linked here:

Videosphere: https://user-images.githubusercontent.com/55333712/107861917-0315de00-6e17-11eb-976c-bb540c3d53a8.mp4

Panorama: https://user-images.githubusercontent.com/55333712/107861591-ca750500-6e14-11eb-9604-e9ca885400a8.mp4

I decided to test with more basic examples such as: Hello World: https://aframe.io/aframe/examples/boilerplate/hello-world/

and glTF 3D-model: https://aframe.io/aframe/examples/boilerplate/3d-model/

The Hello World example looks fine but the green ground plane disappears when you enter VR mode. The camera also teleports to a weird angle after you exit VR mode.

The 3D-model example has a distorted model if you look around a bit in VR mode.

Videos linked: Hello World: https://user-images.githubusercontent.com/55333712/107861857-a1557400-6e16-11eb-88a2-8f0beaad7110.mp4

glTF 3D-model: https://user-images.githubusercontent.com/55333712/107861862-a74b5500-6e16-11eb-820d-b3cc2e975754.MP4

[mdl-andriantam]

hi, im on ios 14.4. regular portrait and landscape mode works well with gyro but once i hit VR mode, i get the same white display for each lens.

the loader works, but the rendered photospheres are gone.

not sure if this is related but i see that iOS is not supported by webvr anymore? https://webvr.rocks/

could this be the reason and is there any workaround to this? thanks!

[dmarcos]

@andriantam-mdl WebVR / WebXR has never been supported on iOS. We use a polyfill to provide some compatibility. Make sure you are using A-Frame 1.1.0. The photoshpere example works fine for me on iPhone / iPad 14.4

https://aframe.io/aframe/examples/boilerplate/panorama/

[mdl-andriantam]

thanks for the info @dmarcos , im on aframe 1.1.0. but im still figuring out why its not working.its a handover project so there might be something in the implementation that wasnt right. so far, no way getting ios device browsers into full screen yea?

[dmarcos]

@andriantam-mdl

Does the photosphere example work for you? https://aframe.io/aframe/examples/boilerplate/panorama/

The only way a page can enter fullscreen in iOS is by pinning the site to the homescreen and launch it from there. Safari policy.

[mdl-andriantam]

@dmarcos yes the photosphere example works for me. so i got some debugging to do.

and yes, pinning the site to homescreen does the charm! thanks

[mdl-andriantam]

@dmarcos so i just realized, even with your sky example demo, it turns to white in vr after looking around. looks like a memory issue.

https://user-images.githubusercontent.com/72718229/108483608-e0cc0680-72d5-11eb-8106-1caca997afa4.MP4

[dmarcos]

Any console errors?

[mdl-andriantam]

@dmarcos

these errors appeared even before the googles went white. after a minute, my safari inspector crashed

let me try to get the errors in text for u

Error: Error: No input element in the page matches .docs-search-sidebar .search-field

WebGL: ERROR: unsupported shader version

THREE.WebGLShader: gl.getShaderInfoLog() vertex
ERROR: unsupported shader version1: #version 300 es
2: in vec2 uv;
3: in vec3 position;
4: uniform mat4 projectionMatrix;
5: uniform mat4 modelViewMatrix;
6: out vec2 vUV;
7: void main(void) {
8:   gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
9:   vUV = uv;
10: }"
"THREE.WebGLShader: gl.getShaderInfoLog() fragment
ERROR: unsupported shader version1: #version 300 es
2: precision highp float;
3: uniform float alphaTest;
4: uniform float opacity;
5: uniform sampler2D map;
6: uniform vec3 color;
7: in vec2 vUV;
8: out vec4 fragColor;
9: #ifdef GL_OES_standard_derivatives
10:   float contour(float width, float value) {
11:     return smoothstep(0.5 - value, 0.5 + value, width);
12:   }
13: #else
14:   float aastep(float value, float afwidth) {
15:     return smoothstep(0.5 - afwidth, 0.5 + afwidth, value);
16:   }
17: #endif
18: #define BIG_ENOUGH 0.001
19: #define MODIFIED_ALPHATEST (0.02 * isBigEnough / BIG_ENOUGH)
20: #define ALL_SMOOTH 0.4
21: #define ALL_ROUGH 0.02
22: #define DISCARD_ALPHA (alphaTest / (2.2 - 1.2 * ratio))
23: void main() {
24:   #ifdef GL_OES_standard_derivatives
25:     vec2 uv = vUV;
26:     vec4 texColor = texture(map, uv);
27:     float dist = texColor.a;
28:     float width = fwidth(dist);
29:     float alpha = contour(dist, width);
30:     float dscale = 0.353505;
31:     vec2 duv = dscale * (dFdx(uv) + dFdy(uv));
32:     float isBigEnough = max(abs(duv.x), abs(duv.y));
33:     if (isBigEnough > BIG_ENOUGH) {
34:       float ratio = BIG_ENOUGH / isBigEnough;
35:       alpha = ratio * alpha + (1.0 - ratio) * dist;
36:     }
37:     if (isBigEnough <= BIG_ENOUGH) {
38:       vec4 box = vec4 (uv - duv, uv + duv);
39:       alpha = (alpha + 0.5 * (
40:         contour(texture(map, box.xy).a, width)
41:         + contour(texture(map, box.zw).a, width)
42:         + contour(texture(map, box.xw).a, width)
43:         + contour(texture(map, box.zy).a, width)
44:       )) / 3.0;
45:     }
46:     if (alpha < alphaTest * MODIFIED_ALPHATEST) { discard; return; }
47:   #else
48:     vec4 texColor = texture(map, vUV);
49:     float value = texColor.a;
50:     float afwidth = (1.0 / 32.0) * (1.4142135623730951 / (2.0 * gl_FragCoord.w));
51:     float alpha = aastep(value, afwidth);
52:     float ratio = (gl_FragCoord.w >= ALL_SMOOTH) ? 1.0 : (gl_FragCoord.w < ALL_ROUGH) ? 0.0 : (gl_FragCoord.w - ALL_ROUGH) / (ALL_SMOOTH - ALL_ROUGH);
53:     if (alpha < alphaTest) { if (ratio >= 1.0) { discard; return; } alpha = 0.0; }
54:     alpha = alpha * ratio + (1.0 - ratio) * value;
55:     if (ratio < 1.0 && alpha <= DISCARD_ALPHA) { discard; return; }
56:   #endif
57:   fragColor = vec4(color, opacity * alpha);
58: }"

WebGL: INVALID_OPERATION: useProgram: program not valid

[k-tests]

Hello, anything new on this? All of the A-Frame website examples not working in VR mode (cardboard) on ios 14.4. 360 images example here: https://aframe.io/examples/showcase/sky/ I cannot find a way around this... :(

[Luze26]

Hi, is there any workaround for this ? I'd like to be able to use the cardboard view with a videosphere on iOs. For the moment it seems not possible since version 1.1.0, I am missing something ?

[dmarcos]

@Luze26 1.2.0 works for me on iPhone X / iOS 14.5

https://aframe.io/examples/showcase/sky/

[Luze26]

@dmarcos it works for me too with a picture. I have the issue with a video, the lens become blank in VR mode.

[dchan38098]

@dmarcos I'm experiencing the same thing as Luze26. VR mode just goes blank with a video within a videosphereon iOS. Is there any workaround to this? Thank you.