iOS 17 (physical device) mathfield unresponsive to touch until page relayout (after device rotation)

[screambeard]

Description

When opening the mathlive editor (on https://github.com/cortex-js/cortexjs.io) on a iOS 17 device (confirmed in iPad and iPhone with iOS 17.1), the virtual keyboard cannot be opened when clicking the mathbox. Clicking the virtual keyboard icon also does not show the virtual keyboard. When rotating the device (forcing a repaint), the virtual keyboard can be opened by clicking the virtual keyboard icon.

Additional issues (probably related to the above)

• can also not move cursor inside the mathbox by clicking
• can also not open contextMenu

Both the other issues are also "resolved" when rotating the device.

Steps to Reproduce

1. Open https://github.com/cortex-js/cortexjs.io on an iPad or iPhone with iOS 17
2. Click the mathbox and see that the keyboard does not open and the cursor is placed at the start (instead of where you clicked)
3. Click the virtual keyboard toggle button and see that it does not open the keyboard
4. Rotate device (and optionally rotate back)
5. Click the virtual keyboard toggle button and see that it now does open the keyboard
6. Click somewhere inside the mathbox and see that it now also places the cursor on that location.

Actual Behavior

It does not open the virtual keyboard when clicking the toggle button in step 3.

Expected Behavior

It opens the virtual keyboard when clicking the toggle button in step 3.

Environment

This issues is not present in iOS 16. Confirmed on iPhone and iPad with both Safari and Chrome for iOS 17. Mac OSx has no issues, seems to be limited to touch devices. Mathlive Version 0.98.0.

Additional information

We used iPhones at our office and real iPads via BrowserStack.

[arnog]

That is very odd. I've tried to reproduce this on a device with iOS 17.1.2 but have not been able to.

Did you try with the page at https://cortexjs.io/mathlive/demo/ ?

[screambeard]

That is interesting: on https://cortexjs.io/mathlive/demo/ it does work, but on https://cortexjs.io/mathlive/ it still has the issue as I described. Tested this on iPad, since I don't have an iPhone with iOS 17 available right now.

[arnog]

Ah, ok. Got it now. Thanks.

[screambeard]

I confirmed the same thing this morning on an iPhone with both the Safari and Chrome browser. Do you have any idea what causes this? It seems based on the results of the tests of the two pages that it depends on how the math-field is setup/initialized. We face the same issue on our own platform and maybe there already exist an (temporary) fix. If we already could setup our math-field differently (the way it is configured on the demo page), we might be able to fix this already.

[arnog]

I have no idea so far. It doesn't reproduce using the iOS Simulator. I'm going to guess this is a bug in WebKit, but I have no idea what might be triggering it and why it would only happen on physical devices... Probably something to do with the event handling? A peculiarity of the mathfield in the page that doesn't work is that it's embedded in another web component (the "playground" component that hosts the editable source code). Don't know if that's relevant, though.

[screambeard]

We investigated this a bit further from our side and found some additional information which might help finding the root-cause of the bug.

In short: it seems that the click events are not registered on the virtual keyboard toggle button for iOS 17 if the mathfield does not automatically open the keyboard itself on focus.

How did we come to this conclusion.

Testing in out own codebase

• in our codebase we add MathField elements programmatically via this method in the documentation and disabled the automatic opening of the virtual keyboard.
• we added click listener on the virtual keyboard toggle element after mounting using the following code

  const vkToggle = mathFieldElement.shadowRoot.querySelector('[part=virtual-keyboard-toggle]');
  vkToggle.addEventListener('click', () => console.log('click'));

• click events are registered in normal browsers on our laptops and output "click" in the console
• we opened our own local test page via BrowserStack with iOS 16 and iOS 17 to see if we could see the click event in the console
  • for iOS 16: we see "click" in console (and the keyboard does open on clicking the button)
  • for iOS 17: we don't see the "click" in console when clicking the button (and the keyboard does not open)

It confirms there the click listener is not triggered for iOS 17. We did manage to get the click listener to work via the following way in iOS 17:

• make sure the mathfield is focussed (e.g. by clicking in the mathfield)
  • NOTE: clicking the virtual keyboard toggle button does not work at this moment
• rotate your device
• click the toggle button and see the keyboard appear
  • NOTE: we also see the click listener emit the "click" in the console now

Note that rotating the device without having the mathfield focussed does not result in a successful toggling of the keyboard.

Testing on https://cortexjs.io/mathlive/

Here we validated the above by executed the same steps on https://cortexjs.io/mathlive/ (where the bug is present)

• make sure the mathfield is focussed (e.g. by clicking in the mathfield)
  • NOTE: clicking the virtual keyboard toggle button does not work at this moment
• rotate your device
• click the toggle button and see the keyboard appear
  • NOTE: we also see the click listener emit the "click" in the console now
• refresh the page and repeat this without focussing the mathfield in the first step and see that the toggle button does not work.

When testing this out, we noticed another difference between the mathfield on https://cortexjs.io/mathlive/ and on https://cortexjs.io/mathlive/demo/

• the https://cortexjs.io/mathlive/ mathfield does not automatically open the virtual keyboard on focus (for mobile/touch devices)
• the https://cortexjs.io/mathlive/demo/ mathfield does automatically open the virtual keyboard on focus (for mobile/touch devices)

Next steps

We used BrowserStack to validate/investigate our local codebase (by connecting BrowserStack with our local development environment via their "local testing" feature) and replicate the problem on iOS 17 devices, since they have real devices and not simulators.

Based on the above our current assumptions on where the problems might arise from:

• click listeners are not registered properly on iOS 17
• it seems that "focus" it part of the solution/problem since it influences whether we can manage to get the button to work with our rotate workaround/fix
• automatically opening the keyboard seems to prevent the issue from appearing in the first place

Are you able to replicate the above and maybe have a better idea on how to investigate the root-cause?

I would advice using BrowserStack or an equivalent to validate ideas. It helped us to replicate the problem and play around with our local dev environment where we can quickly alter code and add listeners.

If we can be of help, let us know. FYI: I will be away for the weekend, so might not respond until Monday.

[screambeard]

A small update on validating that when opening the Virtual Keyboard automatically on touch devices the problem does not occur on iOS 17.

• in our codebase we have mathfieldElement.mathVirtualKeyboardPolicy = 'manual'; since the keyboard is large and the detection on which to open the keyboard automatically is not perfect and our users reported annoyance over this.
• to validate if setting it to auto via mathfieldElement.mathVirtualKeyboardPolicy = 'auto'; is indeed a way to "fix" the problem we changed this in our local environment and tested this with iOS 17.

Conclusion: when we have automatic opening of the keyboard, the click handler works and thus the toggle button works. Unfortunately, this does not solve the problem for using Mathlive with the policy set to manual but might help figuring out where the problem is located.

[arnog]

Thanks for the additional info.

I can replicate your findings on physical devices (iPhone and iPad):

• https://cortexjs.io/mathlive/: the click event handlers do not get triggered until after a physical rotation of the device
• https://cortexjs.io/mathlive/demo/: the click event handler does seem to get triggered

However, both fields have a mathVirtualKeyboardPolicy set to the default, which is "auto".

I haven't been able to reproduce a minimal test case and I don't understand what is the significant difference between pages that work and those that don't.

[arnog]

Also, just for the sake of clarity, it might be obvious but the issue is not limited to the virtual keyboard toggle button: when the mathfield is in that state, it is not responsive to touch interaction at all: if there is content in the mathfield, the content cannot be selected via touch. So, this issue really isn't about the virtual keyboard, but about the fact that touch-interaction with the mathfield is not possible until a relayout (and not just a repaint).

Also, I've tested with the iOS 17.2 beta and the problem still occurs.

[screambeard]

@arnog thanks for the clarification! One thing I noticed before on cortexjs.io is the following:

• when I click the the math-field on https://cortexjs.io/mathlive/ while I'm in responsive mode in my Chrome Browser on my macbook (so nothing to do with iOS), the virtual keyboard does not open automatically
• when I click the the math-field on https://cortexjs.io/mathlive/demo while I'm in responsive mode in my Chrome Browser on my macbook (so nothing to do with iOS), the virtual keyboard does open automatically

This resulted in my assumption about mathVirtualKeyboardPolicy I stated in my previous comments.

Therefore, I tested this assumption in our own codebase by changing from mathVirtualKeyboardPolicy "manual" to "auto" and that "fixed" the problem in the way we use it. I'm not saying the mathVirtualKeyboardPolicy is the problem, but might be hint to the root-cause.

Hope that can limit the scope of the investigation.

[arnog]

Oh... Thank you for that, I didn't notice it. OK, so looking into it that's because on this page the mathfield is inside another web component, the code playground, and when the mathfield is focused, the code playground actually gets the focus event (or rather its the target of the event), so the mathfield doesn't react by showing the virtual keyboard). I'm guessing that this setup probably also plays in role in the issue in general.

In your own setup, is your mathfield inside an iframe or another element that listens to events?

[arnog]

I have fixed the issue with the keyboard not showing on the MathLive landing page. Not sure it's related to the root cause, though.

I think I know why setting the mathVirtualKeyboardPolicty to "auto" prevent the problem from occurring. In that case the keyboards get displayed after the mathfield is focused. As a result, the document is relaid out, which has an effect similar to rotating the device.

A potential workaround to this issue would be to set a one-shot timer in the main page and after a few milliseconds make a change that would cause the DOM to reflow, for example inserting an invisible element. It's a horrible hack, but...

[screambeard]

PoC Ugly Fix

@arnog thanks for idea on how to fix it in an ugly way :). I've tried a proof of concept of this fix as follows:

• on focus add or remove a single non-visible element in the root of the shadowRoot of the mathfield. In short: toggle an element on focus.

This does solve the issue for iOS 17 and allows the user to toggle the keyboard and select parts of the expression.

Real fix

I tried to see if a very simple shadowRoot component caused an issue on iOS 17, but that is not the case. Using the code below will give click events on iOS 17. So the problem lays deeper inside mathLive I think.

const div = document.createElement('div');
div.style.height = '100px';
div.attachShadow({ mode: "open" });
const div2 = document.createElement('div');
div2.style.height = '100%';
div2.addEventListener('click', () => console.log('click on inside div'))
div.shadowRoot.appendChild(div2);
document.appendChild(div);

[arnog]

Given that the mathfield works:

• in Firefox
• in Chrome and Edge on desktop and mobile devices
• in Safari on desktop
• in the Safari mobile emulator
• in Safari on mobile devices prior to iOS 17

but not:

• in Safari on iOS 17 when running on a physical device

I'm going to lean towards the problem being in Webkit/iOS.