resizeCanvas() doesn't always work correctly

[qWici]

Description

We are facing a problem when we change the orientation of the mobile device on iOS in Safari - from portrait mode to landscape mode. All the contents of the applications squeezes randomly. We have studied this issue and we realized that the function resizeCanvas() doesn't always work correctly after turning the screen. It seems that the function/resize triggers before Safari swaps height and width. We thought that maybe we could solve this issue by triggering the function ourselves after some timeout. Is it possible to run resizeCanvas() in the namespace of user scripts? As I understand this is also a known issue. Are there any solutions available?

URL: https://toyota.3dconfiguration.com/ Video: Video with bug Device: iPhone X Browser: Safari 12.4

[willeastcott]

Can you post screenshots of what you're seeing (just to make sure we're observing the same problem)?

[qWici]

@willeastcott

Screenshots


[Maksims]

There is an inconsistencies with the way browsers trigger few events, like: resize and orientationchange on window object. Some browsers trigger them when window.innerWidth and window.innerHeight are changed, some trigger them before those values change. This leads to inconsistencies. Projects made from the Editor, have __start__.js file, it subscribes to those events. Quick fix would be to setInterval with reasonable frequency (200 for example) and check if window.innerWidth is different from what is was with last reflow call. If it is changed, then do the reflow.

We probably should update __start__.js and add such functionality.

[slimbuck]

I think this issue is slightly different.

The aspect on iPhone is incorrect only on landscape when the toolbar is visible. Hiding the toolbar results in correct aspect. So I suspect some element math is perhaps incorrect...

[yaustar]

Created project to showcase this issue on iOS. Both white rectangles are anchored to the top left and bottom right of the screen: https://playcanvas.com/editor/scene/938141

On iPhone 5 wile landscaped, iOS 12.4.5, Safari, the app is stretch vertically.

Edit: added button to trigger reflow trying @Maksims suggestion which unfortunately doesn't work on iPhone 5 device.

[yaustar]

Poked a bit more and I had to make a few changes to the CSS for Max's quick fix to work as seen in the JS file: [Redacted]

(Completely untested on any other device)

// initialize code called once per entity
Sandbox.prototype.initialize = function() {
    this.app.once('postrender', function() {
        document.getElementById('application-canvas').style.position = 'relative';
        document.body.style.minHeight = '100%';
        document.body.style.height = 'auto';
        document.body.style.maxHeight = 'auto';

        window.addEventListener('resize', function() {
            setTimeout(function() {
                window.scrollTo(0,1);
            }, 200);
        }, false);
    });
};

Edit: I'd another crack at this to try and get full screen on iOS and the following style modification works on my iPhone 5 test device. Again not fully tested across multiple projects: https://playcanvas.com/editor/scene/938162

// initialize code called once per entity
Sandbox.prototype.initialize = function() {
    this.app.once('postrender', function() {
        document.getElementById('application-canvas').style.position = 'fixed';
        document.body.style.minHeight = '-webkit-fill-available';
        document.documentElement.style.height = '-webkit-fill-available';
    });

    this.entity.element.on('touchstart', function(e) {
        this.app.graphicsDevice.fullscreen = true;
    }, this);
};

(taken from: https://allthingssmitty.com/2020/05/11/css-fix-for-100vh-in-mobile-webkit/)

This will need more testing across different projects and devices 😅

[qWici]

@yaustar I tested this solution on iPhone X/11 via browserstack - works fine. However, I'm not sure if this bug was there before. So, can you make one build with turn off your script?

[yaustar]

@qWici you can grab any other public project like this one https://playcanvas.com/editor/scene/474021 Thanks for testing :)

Edit:

With the iOS CSS rules above: Editor: https://playcanvas.com/editor/scene/938162 Build (no iframe): https://playcanv.as/e/p/h95tjRTT/

With the script disabled: Editor: https://playcanvas.com/editor/scene/938265 Build (no iframe): https://playcanv.as/e/p/gL9c9BGq/

[yaustar]

Please note that this code in my R&D only works without the iFrame that the publish URL uses. You can get to the iframeless version by adding a /e before /p in the URL.

From the experimenting I did last night without this script (ie the current behaviour on the PlayCanvas platform), it looks like we are rendering at the correct resolution of the viewport but the DOM is being stretched to the display dimensions and the render looks vertically stretched when in landscape.

[malDuffin]

I saw some similar "squishing" issues, and reported them here ( see screenshots in second post )

https://forum.playcanvas.com/t/different-speeds-and-crash-on-ios-with-new-model-viewer-starter-kit-scene/8978

[yaustar]

If I download a self hosted build and change the CSS to the following this is the best result I got so far on my iOS device. No scaling and fills the browser viewport no matter how many rotations.

Changes marked with /**/

My concern is the use of 'fixed' for positioning of the canvas.

html {
    height: 100vh; /*used to be 100%*/
    background-color: #1d292c;
}
body {
    margin: 0;
    max-height: 100%;
    height: 100%;
    overflow: hidden;
    background-color: #1d292c;
    font-family: Helvetica, arial, sans-serif;
    position: relative;
    width: 100%;
}

#application-canvas {
    display: block;
    position: fixed; /*used to be absolute*/
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
}
#application-canvas.fill-mode-NONE {
    margin: auto;
}
#application-canvas.fill-mode-KEEP_ASPECT {
    width: 100%;
    height: auto;
    margin: 0;
}
#application-canvas.fill-mode-FILL_WINDOW {
    width: 100%;
    height: 100%;
    margin: 0;
}

canvas:focus {
    outline: none;
}

[Maksims]

CSS Viewport Units is widely supported for quiet some time, so should be ok: https://caniuse.com/#feat=viewport-units Same applied for position fixed: https://caniuse.com/#feat=css-fixed

It is matter of testing on all different platforms, and ensuring it looks ok with and without bottom panel on playcanv.as as well.

[yaustar]

I've created a test site to make this easier: https://xenodochial-wozniak-590156.netlify.app/no-iframe or https://tinyurl.com/y9nczbtq

Which deploys from this repo https://github.com/yaustar/ios-landscape-rotation-fun

It works on my Android Google Pixel 2 XL, iPhone 5S, Windows Chrome.

Trying to use Browserstack but it's giving me different results to real devices which is weird. Anyone with physical iOS device, I be very grateful for extra testing :)

Create a new tab (important!)

Start in portrait, load site, turn landscape (expect to go full screen), turn portrait (expect minimal URL bar) and back to landscape (expect to go full screen)

Do the same but start in landscape.

Please also check that https://playcanv.as/e/p/cp3OGFrJ/ does stretch, misalign, etc as well for a baseline.

Thanks!

[yaustar]

Ok, I think I have a version that fixes all the issues that I had previously on my iPhone 5S.

I'm not sure I like the solution but it works (TM) and the CSS does make sense. It feels like a terrible hack. 😭

Same site: https://xenodochial-wozniak-590156.netlify.app/no-iframe or https://tinyurl.com/y9nczbtq

Please try to break it on any device that you have 😅

[yaustar]

And it breaks when the user taps the bottom or top of the screen in landscape and shows the browser navigation :(

[Maksims]

And it breaks when the user taps the bottom or top of the screen in landscape and shows the browser navigation :(

Yeah, this is another thing - when user touches screen, and moves around, it can bring that nasty nav bar. The go to solution would be to figure out how to always have that nav bar visible. And only hide it if user actually goes fullscreen. This is basically a behaviour on Android, and iOS should behave same.

That "magical" nav in Safari on iOS - is a pain.

[yaustar]

As iOS doesn't support fullscreen API yet, they will always have the nav bar 😅

This branch does a 'hack' which scrolls to the top of the page 500ms after reflow to get the canvas in the right place. https://github.com/yaustar/ios-landscape-rotation-fun/tree/scroll-hack/no-iframe

That "magical" nav in Safari on iOS - is a pain.

I concur 😭

[Maksims]

I've been investigating it more. Only to conclude: iOS sux :D There are promising solutions. The way we are doing resize of canvas element in order to fill window, might change. As it relies on unstable behaviour of setting style width to window.innerWidth/Height, then getting clientWidth/Height and then setting canvas resolutions to such, removing style from element. The problem lies under window.innerWidth/Height - is not reported correctly in Safari (other browsers on iOS work great).

In Safari, behaviour of changing from portrait to landscape is pretty straight forward, it hides navbar (fullscreens page), and reports resolutions correctly. But then changing back from landscape to portrait, hides bottom bar and slides top bar slightly, leading to higher resolution, but then it fails to report properly resolution, and it does not fires "resize" event on window, which logically it should. It is unique issue of Safari on iOS.

There are few workarounds, one of them is actually calling reflow method on requestAnimationFrame, and doing nothing when window.innerWidth/Height has not changed. This is lightweight, and bulletproof. @willeastcott let me know if this approach you think would be viable?

[yaustar]

There are few workarounds, one of them is actually calling reflow method on requestAnimationFrame, and doing nothing when window.innerWidth/Height has not changed. This is lightweight, and bulletproof.

Will this work in the case of in landscape mode, the user taps the bottom or the top of the screen to show the URL bar (which overlaps content)?

[Maksims]

There are few workarounds, one of them is actually calling reflow method on requestAnimationFrame, and doing nothing when window.innerWidth/Height has not changed. This is lightweight, and bulletproof.

Will this work in the case of in landscape mode, the user taps the bottom or the top of the screen to show the URL bar (which overlaps content)?

So far in tests, it works - no bar is showing. I will test further. And detail a solution. It actually not an engine related, but Launcher and playcanv.as template, the way it does reflow.

[yaustar]

So far in tests, it works - no bar is showing.

Really? That's awesome!

[yaustar]

Alright, turns out on iOS 14 mobile Safari will go full screen if ONLY one tab is open when rotating to landscape. More than one tab will show the nav bar.

Also, on iPadOS, the full screen API is available (https://developer.apple.com/forums/thread/133248).

[spencer-e-pnt]

I am facing a similar issue with iOS Safari, wherein stretching doesn't seem to occur, but a white bar is visible when I switch from portrait to landscape. Safari automatically enters fullscreen, and the innerHeight no longer reflects the actual available height.

There are few workarounds, one of them is actually calling reflow method on requestAnimationFrame, and doing nothing when window.innerWidth/Height has not changed. This is lightweight, and bulletproof.

I tried this ↑ workaround and it didn't seem to work.

After rotating to landscape:

I am running iOS 15.1, and have the new default settings:

• bottom toolbar
• tab bar, not single tab

[yaustar]

Sorry, where's the white bar? Is this from an exported build with PlayCanvas.com?

[spencer-e-pnt]

At the bottom.

[yaustar]

Oh, the page background is white so I didn't see the image is larger than I thought it is.

Is this with a build that has been exported from PlayCanvas.com editor?

[spencer-e-pnt]

It is from an exported build which uses the start.js @yahstar

[spencer-e-pnt]

Was this the correct way to implement your aforementioned workaround?

// do the first reflow after a timeout because of
// iOS showing a squished iframe sometimes
setTimeout(function () {
    pcBootstrap.reflow(app, canvas);
    pcBootstrap.reflowHandler = function() {
        pcBootstrap.reflow(app, canvas);
        requestAnimationFrame(pcBootstrap.reflowHandler);
    };

    // window.addEventListener('resize', pcBootstrap.reflowHandler, false);
    // window.addEventListener('orientationchange', pcBootstrap.reflowHandler, false);
    pcBootstrap.reflowHandler();
    ...

reflow: function (app, canvas) {
    if(lastWindowHeight === window.innerHeight && lastWindowWidth === window.innerWidth) {
        return;
    }

    this.resizeCanvas(app, canvas);

    /*
    // Poll for size changes as the window inner height can change after the resize event for iOS
    // Have one tab only, and rotrait to portrait -> landscape -> portrait
    if (windowSizeChangeIntervalHandler === null) {
        windowSizeChangeIntervalHandler = setInterval(function () {
            if (lastWindowHeight !== window.innerHeight || lastWindowWidth !== window.innerWidth) {
                this.resizeCanvas(app, canvas);
            }
        }.bind(this), 100);

        // Don't want to do this all the time so stop polling after some short time
        setTimeout(function() {
            if (!!windowSizeChangeIntervalHandler) {
                clearInterval(windowSizeChangeIntervalHandler);
                windowSizeChangeIntervalHandler = null;
            }
        }, 2000);
    }
    */
}

[yaustar]

I can't reproduce this on a build I just exported today on iOS 15.0

https://user-images.githubusercontent.com/16639049/145436478-b8391e11-f8d7-49f5-96cc-c647b1ddef2e.mp4

[yaustar]

Are you using an external library like 8th Wall? Or anything else that could affect the canvas?

[spencer-e-pnt]

Thank you for testing, @yaustar. I am using 8th wall with PlayCanvas. I have forked their AR World Tracking Starter Kit project, and am able to reproduce it. My project launch link.

I tried adding the XRExtras.FullWindowCanvas module to the pipeline, and it appears to full-size the canvas in landscape. However, this module makes the camera view black or hides it altogether. Additionally, even with this (potential) fix, the Start button in my pc scene doesn't position correctly. It is still using the incorrect innerWidth, so it's positioned higher than it should be. Please check the portrait screenshot above for reference.

Have you tried adding some ui to your project and seeing if it positions correctly on landscape rotate?

Landscape with XRExtras.FullWindowCanvas module:

[spencer-e-pnt]

OK, it's black simply because that 8th wall module adds a black background-color style rule to the body. Removing that brings it back to white. Here's the module for reference.

When I hover over the canvas with the browser dev tools, I see that it is actually not resizing the canvas in fullscreen. It simply ruins the camera view.

[yaustar]

Without 8th Wall, iOS 15.0 with pink squares anchored to the corners and sides, looks fine.

https://user-images.githubusercontent.com/16639049/145444552-d3ea67bc-acac-46c3-875f-6cc0d56f856e.mp4

It's very possible that 8th Wall is modifying the canvas in some way and should also approach the 8th Wall team support too

[spencer-e-pnt]

Thank you for testing again, @yaustar. Without the XRExtras.FullWindowCanvas module, nothing in the other included modules in their starter project modify the canvas size. It could be that the 8th wall engine source code (not open source) is modifying the canvas size. I'll reach out to 8th wall support regarding this.