search

ianstormtaylor / slate

A completely customizable framework for building rich text editors. (Currently in beta.)
http://slatejs.org
MIT License
29.65k stars 3.23k forks source link

fix editing with "soft keyboards" (eg. Android, IMEs) #2062

Closed ianstormtaylor closed 4 years ago

ianstormtaylor commented 6 years ago

Do you want to request a feature or report a bug?

Bug.

What's the current behavior?

The issue with using Slate on Android is complex, and due to a difference in how its OS keyboard is designed. On mobile devices, keyboards are starting to move away from the "keys" concepts in many ways...

Because of this, it sounds like the Android team (reasonably from their point of view) decided to not reliably fire key-related events.

As soft input methods can use multiple and inventive ways of inputting text, there is no guarantee that any key press on a soft keyboard will generate a key event: this is left to the IME's discretion, and in fact sending such events is discouraged. You should never rely on receiving KeyEvents for any key on a soft input method. —KeyEvent, Android Reference

It sounds like the behavior is:

A few different resources:

What's the expected behavior?

The fix for this is also complicated. There are a handful of different, overlapping pieces of logic that need to change, to accommodate a handful of different input types...

The first stage is to handle basic insertions, and auto-suggestions...

This is actually the same starting steps as is required for https://github.com/ianstormtaylor/slate/issues/2060, so I'd recommend we solve that issue in its entirety first, to work from a solid base.

This fixes the actual text insertion pieces, and probably deletions as well. Splitting blocks can still be handled by enter because it still provide proper key codes.

And then there's some selection issues, which apparently can mess up Android's IME (and potentially others) if the selection is manually changed during a composition.

I think this would solve the 90% case for soft keyboard input.

Separately, there's still another question of how to properly handle these kinds of behaviors other plugins. For example if a plugin uses backspace at the start of a block to reset that block, that won't work on Android. So after solving the input issues, we need to step back to an architectural level and solve this plugin handling problem. But that can wait.

ianstormtaylor commented 6 years ago

@thesunny you're right, that sounds like a bug to me. It should use the passed in range, and never the value.selection.

thesunny commented 6 years ago

There is an Android issue that I'm not sure we can fix.

I edited "rich" to change to "Rich" using Android's auto-suggest and it returns this in the DOM:

image

It obliterates the <span data-offset-key="3:1"> and only keeps the <b> tags.

Any ideas? I'm wondering if this is even fixable without some really hard work to rebuild the DOM manually.

I have to say, this is really really hard, most of all because the time to check a piece of code after writing it is about 30 seconds. Android studio is slow, requires a hard reload, and crashes often requiring restarts and reconnecting the debugger.

Nantris commented 6 years ago

Wow it even changes it from a <strong> tag inside to a <b> tag huh?

To add context for anyone reading, this is the before state of the DOM: image

This is definitely something that's fixable, but yeah, the only solutions I see on the face of it are really ugly. Hopefully @ianstormtaylor has some better thoughts on how to manage this?

@thesunny can you describe your current testing setup? What's the oldest API you're trying to cover, and are you testing with each API version from that one up to the present? I guess, in short, how many emulators do you need to run and how many physical devices do you have?

Is the trouble with Android Studio on Mac or Windows?

I don't necessarily have anything to offer, but it seems that it would help if you had more physical devices. Does that seem likely? I don't know how Android Studio behaves with multiple devices connected though.

If you think that would help, which APIs do you need physical versions of? We can try to get some unused Android devices together to cover that need.

thesunny commented 6 years ago

@Slapbox

I have a few Android devices but I couldn't get the Chrome debugger on the desktop to connect so I've been using Android Studio and restarting often.

My proof of concept React Editor (not using Slate) was set up with 6 different versions and all the basics worked. With Slate, I'm stuck on two issues with API28 so haven't moved on to earlier APIs. I'm fighting with Slate which is great but doesn't have the necessary abstractions so I'm creating new ones along with unexpected Android behavior which really makes no sense whatsoever. Android Chrome is worse than fighting early versions of IE.

My two biggest problems right now are:

I had other problems which mostly boiled down to the incosistent state of the DOM and given events which I resolved by modifying the input tester which you can see here https://thesunny.github.io/input-methods/index.html which now shows both the DOM state at a given event as well as the selection position.

That compositionEnd not being called is tricky because I cannot re-render the DOM while the selection is on it because it moves the selection to the beginning of the node. So, I tried to update Slate's state on the input after compositionUpdate (which is when the DOM is correct) but then the cursor gets forced into the wrong place by Android.

My solution which I will try is to wait for the cursor to move to its new location, remembering where the last cursor position is, updating the state in the place where the cursor previously was. Hopefully this won't reset the cursor position because it's in a different node... however, I haven't tested it yet so don't know if it will work. To get this to work I have to create another abstraction which will allow Slate to update itself from an arbitrary Slate node.

I feel like this wouldn't be that hard if it weren't for the fact that I'm working in molasses. It's hard to remain focused when everything takes so long.

Nantris commented 6 years ago

@thesunny let me know if there's any way I can help. Tinkering in slow motion is exasperating. Are all the Chrome debugger/Android Studio issues on Mac or Windows? On Windows 7, I've found them to be somewhat buggy, but not terribly so.

thesunny commented 6 years ago

It’s on the Mac. Maybe I should try windows. How is the page reload speed?

thesunny commented 6 years ago

Update:

I solved the Android issue where the compositionEnd won't fire if you move the selection after starting a composition. Essentially you can get a composition that spans several nodes. I solved it by keeping a Set of all the nodes that were touched during the composition. When the input after compositionEnd fires, I then go and resolve all the nodes against the Slate DOM. Seems to work.

Now I'm at the issue (final issue that I've seen) of the React nodes being removed when you choose a suggestion for a word that is wholly within a node like <b>bold</b>. In this case Android fires all the events outside of the composition so if your cursor was between o and l and you wanted to switch it to <b>Bold</b> it would do something like compositionEnd, delete, delete, insert "B" insert "o". But it would do it all simultaneously with the events firing in a weird order (keydown, input, etc. are out of order for some reason). I have to keep trying things until it works but I have a few angles I'll try.

So, I said I'd put in 1-2 weeks and this is 2 so I'll keep plugging but won't be full-time. Sorry it's been slow but Android is slow to develop with and frankly the wait events fire doesn't make much sense. This only solves API28 but my feeling is that a lot of the abstractions necessary to fix the problem are there now so the other APIs should be easier to tackle.

quentez commented 6 years ago

We're currently having issues with the long-press keyboard behavior for diacritics on macOS and I was wondering if this could be a fix for that.

Our problem is with this feature:

screen shot 2018-09-17 at 3 35 43 pm

When picking one of the options, the original letter that was typed doesn't get replaced, resulting in two letters which is not the desired behavior.

(Also, is there a simplified version of the code somewhere that works on major desktop browsers and that we could try to experiment with to see if we could help with the non-Android bugs that have been reported throughout the thread?).

thesunny commented 6 years ago

Hi @quentez

This is a candidate to fix that problem.

These are the main concepts in understanding what problems it can solve:

The problem you cite appears to fall into the category of solvable issues so long as you sync the document after the modified character is inserted and not before.

I do have a proof of concept here:

https://github.com/thesunny/composition-editor

With a demo here:

https://composition-editor.now.sh/

Can't remember how far along it was though so not sure if it would be useful for you or not.

Nantris commented 6 years ago

@thesunny, sorry I haven't had a chance to test your work still. Still having trouble updating my Android version to the proper API.

thesunny commented 5 years ago

Mini Update:

I posted a short update in Slack which I'll summarize and expand on here.

Our project is prioritizing mobile which includes Android support. I will start again on Android within 2-8 weeks and I estimate 2-3 weeks of work which includes:

javan commented 5 years ago

A few different resources

Here's one more that I built recently: https://input-inspector.now.sh/

It's inspired by https://github.com/danburzo/input-methods, and has a couple nice additions:

  1. A record of all DOM mutations
  2. Shareable URLs

Source code is here, if you're interested. Hope you find it useful! ✌️

jayohms commented 5 years ago

I want to shed some light on Android OS versions and how those relate to Chrome and embedded WebView's in an app. I noticed some discussion here about testing and supporting particular OS versions, but I don't believe that's a great path for producing reliable input or tests.

The more relevant piece of information is the WebView or Chrome version installed on the device. I've outlined below how the Android OS versions relate to their WebView counterpart. (I've started at Android 4.4, because I don't believe it's worth anyone's time to support Android 4.3 and lower.)

Android 4.4

Starting in Android 4.4, the system WebView component is based on Chromium. https://developer.android.com/guide/webapps/migrating

WebView: shipped with and locked at version 30.0.0.0 Chrome: latest available (currently 70.x)

Android 4.4.3

WebView: shipped with and locked at version 33.0.0.0 Chrome: latest available (currently 70.x)

Android 5.x - 6.x

Starting in 5.0, the WebView component became unbundled from the OS and regular updates are provided via the Android System WebView app in the Play Store. https://play.google.com/store/apps/details?id=com.google.android.webview

WebView: latest System WebView available (currently 70.x) Chrome: latest available (currently 70.x)

Android 7.x - 9.x

Starting in 7.0, the WebView is no longer backed by the Android System WebView app, but instead uses the installed version of Chrome on the device to render embedded WebView's. https://developer.android.com/about/versions/nougat/android-7.0#chrome--webview-together

WebView: latest Chrome available (currently 70.x) Chrome: latest available (currently 70.x)

Support across OS versions

Given the self-updating nature of Chrome and embedded WebView's on Android 5.x+, it'd make sense to focus on those devices. A (relatively new) minimum Chrome/WebView instance could be required as a baseline based on how far back you want to support.

With all the well known composition event issues on Android, Level 2 beforeinput / input events seem like a good path forward, like @javan mentioned. These events are supported in Chrome 60+ and Android System WebView 60+ across all Android OS versions 5.0 and up.

Javan has started implementing a separate Level 2 events controller for mobile in Trix, and so far it's looking really promising. Here's the progress so far: https://github.com/basecamp/trix/compare/level-2-input. The basic gist is handle beforeinput, and implement a method for each event.inputType https://www.w3.org/TR/input-events-2/#interface-InputEvent-Attributes

You can test if Level 2 events are supported in a particular browser/device by running this in the console:

(typeof InputEvent != "undefined" && 
  "inputType" in InputEvent.prototype && 
  "data" in InputEvent.prototype)
ianstormtaylor commented 5 years ago

@jayohms thank you so much for that detailed write up, I really appreciate you taking the time to share your Trix learnings! Starting with only the versions that have auto-updating behavior sounds like an extremely reasonable approach.

Question for you, does Chrome really support Input Events Level 2? Everything I've read so far stated that Chrome supports Level 1, and Safari supports Level 2. (Although my testing revealed that some of the Chrome events were cancelable, so maybe they really are at Level 2 and just haven't said so?)

javan commented 5 years ago

Yup, they're supported in Chrome: https://www.chromestatus.com/features/5656380006465536

Nantris commented 5 years ago

@jayohms - Thanks a ton for dropping in with that knowledge! That's a huge help.

@javan - Perhaps you can shed some light - The issue you linked to links out to the W3C specification for Input Events Level 1, but if you go to that specification it shows this:

image

That's encouraging, but it doesn't seem to explicitly state that Level 2 is supported and the Chromium issue it links to doesn't provide any further insight. Is there any additional information you know of that we can reference to confirm Level 2 support? That would be a dream.

javan commented 5 years ago

I think they just have an old link on https://www.chromestatus.com/features/5656380006465536

https://bugs.chromium.org/p/chromium/issues/detail?id=585875 links to

I'm pretty certain Level 2 input events are here to stay in Chrome. Also, Safari supports them and Firefox is workin' on it.

birtles commented 5 years ago

Chrome doesn't support level 2 yet. As Masayuki points out in the Firefox bug you linked to there is no support for "insertFromComposition", "deleteByComposition" or "deleteCompositionText".

Support for inputType should land in Firefox very soon (there was a lot of work to get to this point) with work on beforeinput coming after that. (There's still a number of things that need to be fixed in the spec first, however.)

javan commented 5 years ago

Apologies, I think I have my terminology wrong. I was using level 2 to mean beforeinput/input events with inputType, data, etc. properties vs. the old generic input event without those properties.

What's the difference between level 1 and 2? At a glance, the specs look similar.

javan commented 5 years ago

Oh, I see:

According to the source code, Chrome implements Level 1. https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/events/InputEvent.h?l=26-72&rcl=d4c3823079fe41cd1cbef1ffe2e8b24a173f5559

Since there are not definition of "insertFromComposition", "deleteByComposition" and "deleteCompositionText". So, perhaps, we should follow Level 1. I don't know how much stable of the specs, though.

Chrome doesn't implement those newer inputTypes for compositions, but it does implement "insertCompositionText". Here's a composition in Chrome (Level 1) and Safari (Level 2): https://input-inspector.now.sh/profiles/KXHldfRK9AtSj19nRR3O,q1nsTzNNdb8927yUR1vW. So, it's entirely possible to handle compositions in both browsers by responding all four composition inputTypes. I have it working in Trix: https://trix-level-2.now.sh/

ianstormtaylor commented 5 years ago

Last I checked I thought the big difference between Level 1 and Level 2 was that in Level 1 many of the events were not preventable using event.preventDefault()? So you were required to let the event happen and then try to figure out what happened. But maybe this is no longer true?

Nantris commented 5 years ago

@ianstormtaylor I think Level 2 encompassed quite a bit more than that. I wouldn't be surprised if some browsers implemented an ability to cancel the events without implementing the rest of the draft specification. That said, last I checked the Chrome team said they'd not be making any accommodations with regards to onBeforeInput and preventDefault. Last news I saw was about a year ago now though.

Regarding preventing events, see the first two notes in this section of the draft spec

javan commented 5 years ago

Trix, for the most part, does not cancel events so the inability to event.preventDefault() Level 1 input events isn't an issue. Generally, our approach is:

  1. Listen for events and translate them into editing operations on our internal document model
  2. Render the document with every change to a detached element (we call it a shadow element)
  3. Observe the editor element with a MutationObserver and compare it to the shadow element after each mutation
  4. Replace the editor element's contents with the shadow element's unless they are equal

This way, we avoid touching/disrupting the editor element when contenteditable does the "right" thing, which is fairly often when typing normally. When it doesn't do what we want, we step in. We never have to "try to figure out what happened" since the internal document model is always the source of truth.

With Level {1,2} input events, step 1 becomes a lot easier. We just listen for beforeinput and translate the event's inputType, data, and getTargetRanges(). In browsers that don't support them we handle a dozen or so key and composition events.

thesunny commented 5 years ago

Firstly, sorry for missing the deadline for starting work on Slate Android support. Unfortunately, I've had more work to do on our app before I could start dedicated work on Slate Android. I will try and start around the new year. I've compiled everything I know and what my plan is and pasted them below in a kind of state of Slate Android support so you know where I am:

Sunny's State of Slate Android Support

I've tried to write this many times and failed because there are so many overlapping ideas. After many failed attempts, I think there's value in doing a brain dump even if it's imperfect. So, my apologies, if this comes out as a stream of consciousness post.

The following is a run-down on the current state of my attempts at supporting Android on Slate, what has failed or proved difficult and how I plan on implementing Android support.

I also felt the need to publish all of this because it informs the approach I'm taking at the end. Without this preamble, it would be hard to understand the approach.

1. The problem is hard to solve because of many incompatible Android versions.

Android support is hard. It is not hard like getting Chrome, or Safari or Firefox to work. It is more like it's hard like getting Chrome, Safari and Firefox to work. That is because there are at least 6 versions of Android we need to support and they all work differently. They fire different events in different order, in varying amounts and at different times. In essence, you aren't supporting one new browser, you are supporting 6. In fact, the differences between versions of Android are greater than the differences between actually different browsers in my opinion.

Also, unlike desktop browsers, Android browsers are tied to Android versions. It's easy to support the latest few versions of Chrome on desktop because you know they auto update. Mobile OSes, however, do not, so you end up having to support the last few versions in order to get any meaningful browser support.

Note that the most highest rate of usage of an Android version as of today is API 23 with 21.3% which is 4 versions away so even if you only wanted to support the most popular API version and newer, that's 5 versions. The next earlier version is API 22 with 14.4% and then after that is API 21 with 3.5% which I think is reasonable enough to cut off. The latest 6 versions encompasses 85.4% of Android API versions.

2. High number of input methods and situational edge cases

There are many different input methods that need to work to support Android propertly. International input methods, autocorrect, spellcheck correct, gesture writing and what I call word break insertion in which browser behaves differently when a word breaking character is entered.

Furthermore these scenarios may behave differently at the beginning of a word, in the middle of a word and at the end of the word as well as at the beginning of a line, end of a line and the beginning and end of a document.

Also, in some API versions a single composition can take place across multiple separate DOM nodes. That is, multiple words can be edited as part of a single composition which must be handled as a single composition or it fails. In my opinion this is a browser bug but never-the-less needs to be handled.

3. Inability to do partial support

Certain bugs appearing in Slate are inconvenient but acceptable even if annoying. For example, if a user's input fails in 10% of the input cases and that left the text a little garbled and then you go back and fix that garbled text, a case could be made that it would be acceptable. However, if 10% of the input cases left the editor in an inconsistent state in which errors compound and can never be recovered from without resetting the editor, that to me is unacceptable.

Our users get upset losing any amount of data. Imagine somebody composing a blog post and then losing their post. Android bugs tend to fall into the space of putting the editor in an inconsistent state which results in data loss and therefore, it's unfortunate but, partial support feels like a bad option unless the edge cases are truly edge (maybe 1%).

4. Don't construct Editor State from individual events

I've been advocating for this a long time, but for anybody else attempting otherwise, do not reconstruct Slate's internal state by individually reading events and trying to figure out what the user did in Android. It works in other browsers but it doesn't work in Android. Worse, it looks like it will work and in certain, very simple scenarios, it does work, but it always fails when some combination of different input methods or Android API versions are used and then you have to throw away all your Android compatibility work.

Just looking at the enormous differences from the events I logged between different input methods and Android API versions shows basically how hard it is. Honestly, I think it's actually impossible to make this work reliably because I believe there is not enough information provided in the events themselves to reconstruct in all scenarios.

5. Do Use DOM Reconciliation

Something I got working quite reliably (in one day actually) with a proof of concept editor I built was to use DOM reconciliation. Basically, you wait for a compositionStart event to happen, then you ignore everything else that happens until you get to a compositionEnd event. Then when you get one of those, you read the DOM, then you update the Editor's internal state using the DOM. If all goes well, React's DOM Reconciliation ends up being like a no op because the DOM matches the internal state. I did this successfully in my prototype also using transactions like Slate does. Then I also got this to work in almost all cases except one that I found in (I think) two versions of Android.

6. Nieve Reconciliation Fails

The version of DOM Reconciliation I built works under the idea that it is safe to ignore all the events after a compositionStart and that it is safe to read the DOM during a compositionEnd and to update the DOM immediately after a compositionEnd. Unfortunately, this is sometimes true but not always true. It is true and untrue depending on which Android version you are using and what you are doing. In some cases, it is true on an event that happens immediately after the compositionEnd but not on compositionEnd itself. Ultimately, it is unpredictable to know without testing every Android API version multiplied by every input scenario.

So one aspect of making Android work is to discover when what would be our ideal compositionStart and compositionEnd happens. We may wish to call these different events to distinguish them from the native compositionStart and compositionEnd. At any rate, as long as it follows our rules which is safe to ignore after start, safe to read at end and safe to do a React DOM refresh after then everything should work.

7. Must Support Multiple DOM Nodes in a Single Composition

Initially, I assumed that a composition must occur within a single DOM node. For example, if you started editing a word in a paragraph then clicked into another paragraph, I assumed that the first composition would end, then a second composition would begin in the second paragraph. In fact, this held true for the initial API version I tested against; however, in switching to a different API version, I found that a single composition could span multiple nodes.

So the solution must be able to know in which nodes the composition has taken place in, then reconcile all the changes across multiple nodes. I handled this by creating a JavaScript "Set" of nodes and adding the DOM node whenever a compositionUpdate event took place. Since a Set doesn't allow for duplicates, you end up with a nice clean list of DOM nodes in which you edited. I have gotten this to work in Slate.

8. Android Studio is hard to work with

There is no other way to put it than working on Android support in Android Studio is the most frustrating, slowest, and hardest development process I have ever gone through in my entire life. It executes an order of magnitude slower than an actual device, it gets slower as you work on it, then it crashes fairly soon. Sometimes you can't figure something out, and the virtual device just needed to be restarted.

In my opinion, it's more than just a matter of having more patience. When you make a code change and you have to make sure that code change didn't break the 20 other cases before it that you have already fixed, you need to be able to do those tests quickly.

Furthermore, the longer it takes to run each test, the more shortcuts you end up taking like ignoring some cases like maybe just doing a basic gesture edit and not testing at the end of a line and only in the middle. When you finally get to do the full test again, you're not sure which of the last changes broke Android support.

The slowness is compounded by the fact that these tests can't be automated short of using a robot finger (which I'm not about to build) and that these tests compound for each version of Android you need to support.

9. Use Native Devices

So use native devices where possible for the device testing. My initial two week dedicated work on Slate was done in Android Studio because I couldn't get the native device debugging to work with Chrome and I just needed to move forward. That problem has been solved now and I recommend not trying to implement Android support using Android Studio with one caveat which I will get to.

10. Regression testing

Given that (a) Android studio is hard to get running and when it is running we need 6 versions and those 6 versions can crash unreliably and (b) basically nobody has 6 Android native devices each having a different Android version that needs to be supported, then my biggest worry is regression testing. How can we possibly do regression testing when the only affordable way to test is prohibitive to set up and run and the only reliable way is expensive and would still take prohibitively long to test manually.

When somebody makes an update to Slate, there is a non-zero chance that it breaks Android support in one or more versions so there needs to be a way to test.

The only way I can see this working is to record Android events and DOM state during each event, save them in a test, replay them to Slate and see if we get what we expect out of it.

11. Recording Android Events

We need to create a web app that closely mirrors Slate's event system. Then we need to record the events into a JSON compatible format extracting the minimal amount of data for us to properly test. In addition to the events, we need to record the DOM state at the time of the event as that information is necessary to tell us if, for example, a change has been commited to the DOM at the time an event has been called. This can vary between versions of Android.

In the scenario of recording the events, this may be a good area where we can use Android Studio since we only need the recordings to execute once.

12. setTimeout and requestAnimationFrame

Another thing we need to track is setTimeout and requestAnimationFrame as a way to tell us what events get fired as a group. This can be useful in possibly simplifying code.

For example, if compositionEnd is unreliable to tell us when the edit has been commited to DOM (which it is for some verisons of Android) and in some cases the keyDown fires after it which is when the change is commited to DOM, we can't use compositionEnd. But if the keyDown is grouped with the compositionEnd, we could potentially use setTimeout or requestAnimationFrame from inside the compositionEnd with the minimum timeout and then fire the reconciliation code after that. So in order to test this, whenever an event fires, we run a setTimeout and a requestAnimationFrame. This gets logged. All the subsequent events that get fired do not launch a new setTimeout or requestAnimationFrame. When the callback is fired, we log that. This way we know which events get fired as a continuous group without space for a setTimeout.

13. Holy Grail

I think there is a possibility that there is a holy grail like using setTimeout/requestAnimationFrame. Like maybe we can wait for a compositionEnd, fire a setTimeout and wait for the callback before reconiling and maybe it works in 90% of the edge cases in all Android versions and then we only handle the other cases; however, we can't know without running all the test cases against all the Android versions so it seems likely we need to record all the android events with setTimeout/requestAnimationFrame for grouping at a minimum.

14. Why I haven't made PRs so far

Well, to put it simply, I wouldn't accept my own PRs. Yes, it's mostly working but the code is a mess. Because of how long it takes to retest a change, I kept refactoring to a minimum because it could inadvertently stop code from working which would in turn take a long time to fix and retest.

However, you can peek into my Slate repo.

15. Separating DOM Reconciliation and making it more efficient

I have a version of DOM reconciliation which is basically the code already in Slate extracted into a separate module. It works, but it is somewhat nieve in that it replaces the entire contents of the DOM node in Slate's State even if only one character changed.

A more efficient algorithm would look for characters that match at the beginning and match at the end and only the differences in the middle would make up the actual changes. I had a version of this working in my prototype Editor. I don't think this is necessary for a version 1 of Android support but it wasn't that much work in my prototype to build.

16. Modularize Composition Handling

Right now, Slate's before plugin handles preprocessing of events which includes handling of compositions including when they start and end. This has made it difficult for to build in the different methods that Android needs to handle compositions.

We need a way to separate composition handling into it's own component which can be customized per browser and possibly per API version. The alternative is to have potentially a large number of if/else/switch statements in the before plugin itself.

Steps to Move Forward

So here's a plan for working on Slate Android Support.

Nantris commented 5 years ago

@thesunny amazing write-up (from what I've gotten to read so far) - Have you tried using Expo as a way to bypass the nightmares of Android Studio? Not to nitpick at all, but I wanted to let you know you missed bolding items 15 & 16.

thesunny commented 5 years ago

Thanks @Slapbox and fixed.

Also, I've updated with a Steps to Move Forward.

ianstormtaylor commented 5 years ago

Hey @thesunny thanks for the writeup!

As for version support, can you reconcile what you said with what @jayohms mentioned above? I'm unclear about how the API xx versions reconcile with the x.x.x versions. Specifically it sounds like 7.x+ would guarantee we're working with the latest version of Chrome always.

Supporting back 6 versions sounds like a huge amount of work, and it doesn't sound like it should be necessary for the first stab at Android support. It would be more useful to get it working with the latest version or two, and then see what can be done about backwards compatibility.

javan commented 5 years ago

Build an event logging system that can save data into a permanent format like JSON or into a database. May be ideal if we can store this in a database or something so that others can contribute use cases

FWIW, that's exactly what I built: https://github.com/ianstormtaylor/slate/issues/2062#issuecomment-438436799

thesunny commented 5 years ago

@ianstormtaylor

7.0+ includes Nugat and Oreo or about 50% of the ecosystem which is pretty good.

image

However, what may not have been clear is that this already encompasses 5 API versions which is one less than my proposal. Nougat and Oreo each had 2 API changes each. Adding one more API version (API 23) adds the biggest number of supported users. API 23 (which is Marshmallow or Android 6.0) has the largest market share out of all API versions with 21.3% of all users using it.

image

I think what @jayohms is talking about is the chrome version in relation to Android versions but I think, most likely, we are seeing different behaviors in the event firing and their order due to GBoard. I'm guessing this because the Samsung keyboard I believe doesn't have the GBoard issues.

I think that the approach I'm taking will work almost identically between each Android version with the differences being when a composition starts, when it ends, and which events we need to cancel. In other words, I think the big work is actually getting any 2+ version of Android to work since it has to be abstracted. At that point, we just need to configure the handling and cancelling of events.

As I mentioned in an another post, I had Slate working in 2 versions of Android with (if I remember correctly) one issue. But it was getting really hard to add the second version because I was shoehorning each version on top of the other instead of building a proper framework around it like the before before plugin which I mentioned. I think it will get deblitating going to 5+ versions! Also, I kept getting regressions which I wasn't aware of which is why I was thinking of the event capturing and replay framework (BTW, thanks @javan I was planning on talking to your more about your work in that).

ianstormtaylor commented 5 years ago

@thesunny the benefit to working backwards from the latest versions though is that over time all but 8.x is declining is usage, so depending on how long it takes to get it working, we can eliminate needing to serve versions like 5.x and 6.x.

But I'm still unclear on what the API changes we need to care about are... There's the Chrome version, which is sounds like as of Android 7+ will be the latest version of Chrome. So that shouldn't be an issue.

Are there other things that determine the event ordering? Is it not tied to the Chrome version that is being used?

thesunny commented 5 years ago

@javan

Thank you so much for building that.

There are a few things missing that I need and I wonder if you might be able to help me by adding them. I know I can fork but having some help would be great and move things along faster. I can start another issue to see if we can hash things out if you are able to. We can go deeper if you are able to, but off the top of my head:

thesunny commented 5 years ago

@ianstormtaylor

So, my hope initially was that we wouldn't have to handle so many versions. When I initially started recording the inputs, I was hoping that (a) API versions didn't matter and only Android versions did and (b) I'd find some patterns so that we wouldn't have to handle so many versions.

So for (a) unfortunately, each API versions is basically like handling another browser when it comes to input events. I might be missing something but I don't think we can get away from the fact that there are quite a few API versions we need to support. For example, you can buy brand new Nougat devices today which is still like 4-5 versions of support?

And for (b) there are similarities which is what I'm planning on abstracting against.

Basically, it boils down to this. (1) user starts to make an edit and then we need to lock Slate's document model from changes. (2) as the user is editing, we need to record where the edits took place in th DOM (3) we need to wait until the composition ends, reconcile Slate against the dom changes and unlock the editor .

So we need to write that no matter which version of Android and I think that's actually most of the work to handle all versions of Android. Each Android version only changes the logic of when composition starts and ends and which inputs that we need to blackhole (i.e. the events that need to be ignored lest Slate Documents gets updated which means React re-render the DOM in the middle of composition and puts editor in an incosistent state). But I think to put you at ease, I would probably release with 2 versions of Android first but I'd maybe try to keep working until we got all 6. The idea I was trying to convey is that most of the work has to be done if we are going to support at least 2 versions anyways.

I still have hope though that we can use requestAnimationFrame with an event and it might (hopefully) work for all or almost all versions of Android.

saravanan10393 commented 5 years ago

@thesunny I am really need of this android support. Because of this limitation, i have gone through other libraries likes quills js, proseMirror, but i don't get the comfortable that i got with slate. any ETA for this issue to be get fixed, Since i am building a minimal editor using plain contentEditable div. once this issue is got fixed, i will thrown this code and use the slate instead.

dmitrizzle commented 5 years ago

It may be worth considering building an abstraction library for Android devices' input (with Slate as primary candidate user) and maintaining it as a separate package.

This may make it a little easier to test and could be proven to be useful for other devs. 2¢

JuanValencia commented 5 years ago

I was playing with this today... Not that this works well by any means, but it might be helpful to others:

  componentDidMount () {
    window.addEventListener('input', this.onInput)
  }

  componentWillUnmount () {
    window.removeEventListener('input', this.onInput)
  }

  onInput = e => {
    if (isAndroid && `editor-${this.id}` === e.target.attributes.id.value) {
      e.preventDefault()
      e.stopPropagation()
      const endsWithSpace = /[ ]+$/
      const focusKey = this.state.value.selection.focus.key
      const block = this.state.value.document.getClosestBlock(focusKey)
      const lastText = block.getLastText().text
      const words = lastText ? lastText.split(' ') : ['']
      const lastWord = words[words.length - 1]

      console.log('=========================')
      console.log(e.inputType)
      console.log('key', focusKey)
      console.log('data', e.data)
      console.log('lastText', lastText.length, block.text.length)
      console.log('empty', lastText.length === 0)
      console.log('endsWithSpace:', endsWithSpace.test(lastText))
      console.log('words', words)

      switch (e.inputType) {
        case 'deleteContentBackward':
          if (e.data) { // If the soft keyboard has buffered data
            this.editor
              .deleteBackward(lastWord.length)
              .insertText(e.data)
          } else if (lastWord.length > 1 || words.length > 1) {
            this.editor.deleteBackward(1)
          } else {
            console.log('crash!')
          }
          this.editor.focus()
          break

        case 'insertCompositionText':
          if (lastWord.length > e.data.length) {
            this.editor
              .deleteBackward(lastWord.length)
              .insertText(e.data)
          } else if (lastText.length === 0 || endsWithSpace.test(lastText)) {
            this.editor
              .insertText(e.data)
          } else if (lastWord !== e.data) {
            this.editor
              .insertText(e.data.replace(lastWord, ''))
          }
          break
        default:
          console.log('unhandled: ', e.inputType, e)
      }
    }
  }
....
  render = () => {
  ...
  <Editor
          id={`editor-${this.id}`}
          key={`editor-${this.id}`}
          placeholder='Send a message...'
          ref={this.ref}
          value={this.state.value}
          onChange={this.onChange}
          renderNode={this.renderNode}
          renderMark={this.renderMark}
          plugins={this.plugins}
        />
 ...
Nantris commented 5 years ago

I think @dmitrizzle's idea is worth serious consideration.

@JuanValencia can you comment on the current state of that code and if it improves any of the behavior of Slate as it is?

JuanValencia commented 5 years ago

I kept hacking a bit more. This is the current update:

  onInput = e => {
    if (isAndroid && `editor-${this.id}` === e.target.attributes.id.value && this.editor) {
      e.preventDefault()
      e.stopPropagation()
      const endsWithSpace = /[ ]+$/
      const focusKey = this.state.value.selection.focus.key
      const block = this.state.value.document.getClosestBlock(focusKey)
      const lastText = block && block.getLastText().text
      const words = lastText ? lastText.split(' ') : ['']
      const lastWord = words[words.length - 1]

      console.log('=========================')
      console.log(e.inputType)
      console.log('key', focusKey)
      console.log('data', e.data)
      console.log('lastText', lastText.length, block.text.length)
      console.log('empty', lastText.length === 0)
      console.log('endsWithSpace:', endsWithSpace.test(lastText))
      console.log('words', words)

      switch (e.inputType) {
        case 'deleteContentBackward':
          console.log('---- could crash')
          if (!lastWord.length > 1 && !words.length > 1) {
            // The node with key doesn't exist when you backspace here in the DOM, but does in the value
          }
          break

        case 'insertCompositionText':
          if (lastWord.length > e.data.length) {
            this.editor
              .deleteBackward(lastWord.length)
              .insertText(e.data)
          } else if (lastText.length === 0 || endsWithSpace.test(lastText)) {
            this.editor
              .insertText(e.data)
          } else if (lastWord !== e.data) {
            this.editor
              .insertText(e.data.replace(lastWord, ''))
          }
          break
        default:
          console.log('unhandled: ', e.inputType, e)
      }
    }
  }

This improves the functionality on a nexus 6 and pixel 3 a great deal. There's oddities in edge cases and I really haven't tested it thoroughly. What's more, this is external -- not internal, so there's that.

I don't see why you couldn't do this logic internally -- with several iterations, I'm sure it could catch more and more people's edge cases.

-edit- : I'm not sure how you would abstract this to a separate library. I feel like I want more control over the events not less. What's more, you have to muck with the editor's value since this is more of a compensating solution, rather than a native one like Sunny was suggesting.

thesunny commented 5 years ago

@JuanValencia Nice approach and a great way to test ideas before having to modify Slate code! I think I'll do that during prototyping.

@saravanan10393 I would say estimated time to completion will be January or February realistically.

rbar2 commented 5 years ago

@thesunny I am really hoping to see this as well. Having even "Inserting text" and "Deleting characters backwards" work would be a huge help!

Nantris commented 5 years ago

@JuanValencia I tried to play around with your code but I can't get input events to fire on the editor on desktop or mobile. They fire just fine if I input into a plain <input />, <textarea />, or <div contentEditable />

Any ideas what I might be overlooking? slate@0.43.7 & slate-react@0.20.8

thesunny commented 5 years ago

Just a heads up that I've almost completed an events recorder to properly understand Android in ContentEditable. It was faster to build it from scratch because of some specific requirements I had for it that weren't present in existing events recorders.

I may be asking the community to contribute recordings for various API versions. I will then use that information to build the Slate Android feature.

thesunny commented 5 years ago

Hey everyone, if you have time to make some event recordings from an Android device, please go to this URL with an Android browser:

https://editor-events.herokuapp.com/

The app will identify what API version you're on and also which scenarios don't have recordings for a particular API version. Please follow the instructions on each scenario carefully. Thank you!

Note: "View Recordings" doesn't work on mobile yet. You'll have to view from desktop. I wanted to deploy before I went home and didn't want to delay before the weekend. It shall be working soon enough...

thesunny commented 5 years ago

Featuers:

If you think you can help in designing useful scenarios or editing existing ones as well as potentially curating the recordings (e.g. if a recording is bad) then let me know and I can show you how to enable admin access.

rbar2 commented 5 years ago

@thesunny, when I visit the project on a Google Pixel 2 with Chrome, I see "Note: You are not using an Android browser"🤔. Shall I still proceed and complete recordings, or is there an issue I'm overlooking?

thesunny commented 5 years ago

@rbar2

Ahh... interesting. Do you think you can share your user agent with me? I might be parsing it wrong.

It works for me from Android Studio but I haven't tested it with any real devices yet.

You can get your user agent from here:

https://www.whatsmyua.info/

Note: Recording should be fine as I store the entire User Agent for analysis but you won't be informed which of the tests require a recording for your API version.

rbar2 commented 5 years ago

Sure thing: Mozilla/5.0 (Linux; Android 9; Pixel 2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 Mobile Safari/537.36

Great, I'll do some recording for you👍

birtles commented 5 years ago

It also gives, "Note: You are not using an Android browser" for Firefox for Android (UA: Mozilla/5.0 (Android 9; Mobile; rv:65.0) Gecko/65.0 Firefox/65.0 for current beta).

(And I really hope draft doesn't end up adding UA sniffing here. UA sniffing in libraries like this makes it nearly impossible for us to update Firefox to behave more like Chrome/Safari.)

KrisBraun commented 5 years ago

Thanks so much, @thesunny! I've recorded many using api-27 and the SwiftKey keyboard. I noted in some but not all the comments I was using SwiftKey rather than the default Gboard.

thesunny commented 5 years ago

@KrisBraun Oh, I better add the ability to specify the keyboard. It's Gboard that typically gives us problems. That said, the more recordings we have the better so we know what's going on in all the browsers.

@birtles FYI, this is Slate not Draft but I'll try to use as little UA sniffing as possible. I'm hoping to find some generic solutions here. That said, Firefox will probably be easier than Android. Basically all the browsers except Android/Gboard work similarly. It's only Android and it's multiple API versions that work differently that is problematic. I suggest aiming for compatibility with Chrome/Safari and my guess is Slate will either already be working or it shouldn't take too much work to do so.

@rbar2 Yeah, I just used a bad regexp. Basically I search for /^9[.]0/ after Android when I didn't account for there not being a "." after the "9". I won't be able to fix it immediately but please continue the recordings. :)

rbar2 commented 5 years ago

@thesunny, no problem! I have completed all recordings other than the final, "hard keyboard" one :)