fix editing with "soft keyboards" (eg. Android, IMEs)

ianstormtaylor commented 6 years ago

Do you want to request a feature or report a bug?

Bug.

What's the current behavior?

The issue with using Slate on Android is complex, and due to a difference in how its OS keyboard is designed. On mobile devices, keyboards are starting to move away from the "keys" concepts in many ways...

Autocorrect will make changes without any "key" being press.
Autosuggest will insert words that don't map to keys.
Swipe-to-type will insert entire words in one go, instead of using keys.
etc.

Because of this, it sounds like the Android team (reasonably from their point of view) decided to not reliably fire key-related events.

As soft input methods can use multiple and inventive ways of inputting text, there is no guarantee that any key press on a soft keyboard will generate a key event: this is left to the IME's discretion, and in fact sending such events is discouraged. You should never rely on receiving KeyEvents for any key on a soft input method. —KeyEvent, Android Reference

It sounds like the behavior is:

The keypress event is never triggered, which is fine for us.
Text-based keys like a, b, etc. fire a keydown event but always with a key code of 229, indicating that the key is unidentifiable because the keyboard is still busy processing IME input, which may invalidate the actual key pressed.
Pressing keys like enter fires a keydown event as normal, with an event.key that can be recognized? (This is unclear whether this happens or not.)
Pressing backspace does not fire a proper keydown event.

A few different resources:

Great Android postmortem writeup: https://docs.google.com/document/d/1Hex89Di-r-Wfpo1DLAtxpetoX588ziXVoNyC87Je3Xc/edit#
Chrome "bug": https://bugs.chromium.org/p/chromium/issues/detail?id=118639
React issues: https://github.com/facebook/react/issues/4079 https://github.com/facebook/react/issues/6176 https://github.com/facebook/react/issues/11231
VSCode's IME test docs: https://github.com/Microsoft/vscode/wiki/IME-Test
@danburzo's input methods: https://github.com/danburzo/input-methods

What's the expected behavior?

The fix for this is also complicated. There are a handful of different, overlapping pieces of logic that need to change, to accommodate a handful of different input types...

The first stage is to handle basic insertions, and auto-suggestions...

[ ] Remove the preventDefault in onBeforeInput, so that the DOM is updated, and the onInput logic will trigger, diffing the insertion and then "fixing" it.
[ ] Update the <Leaf> (and other?) components to increment their key such that React properly unmounts and reconciles the DOM, since it has changed out from under it.

This is actually the same starting steps as is required for https://github.com/ianstormtaylor/slate/issues/2060, so I'd recommend we solve that issue in its entirety first, to work from a solid base.

This fixes the actual text insertion pieces, and probably deletions as well. Splitting blocks can still be handled by enter because it still provide proper key codes.

[ ] Check that all other behaviors that aren't text insertions (eg. splitting blocks) are handled properly without access to many keydown events.

And then there's some selection issues, which apparently can mess up Android's IME (and potentially others) if the selection is manually changed during a composition.

[ ] Prevent re-rendering the editor on compositionstart and compositionend.
[ ] Prevent updating the selection while a composition is taking place.

I think this would solve the 90% case for soft keyboard input.

Separately, there's still another question of how to properly handle these kinds of behaviors other plugins. For example if a plugin uses backspace at the start of a block to reset that block, that won't work on Android. So after solving the input issues, we need to step back to an architectural level and solve this plugin handling problem. But that can wait.

ianstormtaylor commented 6 years ago

I've done some input tests in different browsers and devices using Dan Burzo's input tester.

I think the only way we're going to arrive at proper Android support is to use a tool like this, and take extremely detailed recordings of inputing the exact same text across the different platforms. Otherwise the intricacies of the ordering of events, and when compositions do or do not start is going to be too complex to guess.

I've started with a few, we'll need more. I think we'll need:

[x] Inserting text.
[x] Inserting IME text.
[ ] Inserting a word via auto-suggest.
[ ] Inserting a word via gesture typing.
[ ] Auto-correcting a word.
[x] Deleting characters backwards.
[ ] Deleting characters forwards.
[ ] Delete characters backwards, across word boundaries.
[ ] Selecting a word, then deleting it.
[ ] Selecting a range across blocks, then deleting it.
[ ] Moving to the end of a word.
[ ] Moving to the middle of a word.
[ ] Moving to the middle of a word, then inserting text.
[ ] Splitting a block at the end.
[ ] Splitting a block in the middle.
[ ] Joining a block from the start.

Even from just the ones I did, we can already see complex composition behaviors on Android that we're going to have to solve for:

Moving the cursor into or to the edge of a word, starts a new composition immediately (even without any input) with the word as content.
Deleting characters backward doesn't trigger any useful key events, but it does update the composition ticking down the characters in a word.
All text insertions on Android is via composition events, so in solving for its composition handling, we'll probably get desktop IME support in the process.

I think we'll need to store additional information in these tests of what the current window.getSelection() range is, because that is key to understanding how much information we can get from these composition events. Someone might need to build a custom input tester for Slate specifically to make this easier.

thesunny commented 6 years ago

@ianstormtaylor

You are making a lot of progress here. I’ve committed next week to working on mobile support but by a large margin, you are more qualified to solve this than I.

I think we might make more progress if you give me any menial tasks, research, etc. In order to help you. Don’t feel obligated to give me work but if you have any tasks, I will work on them full time for you starting Monday. Let me know.

ianstormtaylor commented 6 years ago

@thesunny thanks! One of the most helpful things would be getting more of those detailed event samplings across devices/browsers for each type of edit. That would allow anyone who works on this in the future to reference them to make sure they're thinking through the event logic correctly.

Without those I think we'd just be fumbling around guessing what logic we need to add to the After plugin to get composition events working.

thesunny commented 6 years ago

Okay, I will start with that Monday

Nantris commented 6 years ago

@ianstormtaylor Dan Burzo's input tester is exactly what I was looking for the other day to send you. Glad you found it/already had it.

@thesunny and @ianstormtaylor how/where are you laying down code for this? I don't think there's much I can contribute in the way of code in the short term, but I'm happy to test ideas on my Android device and help any other way that you two need.

thesunny commented 6 years ago

I created DropBox Paper documents. I set up a testing environment and put together all the tests below

So, a few differences from @ianstormtaylor is that I tested 6 different versions of Android and they all provided different values. There are more differences within the different Android versions than across all browser/OS versions. iOS, Safari and Chrome are very similar.

[x] Inserting text.
[x] Inserting IME text
[x] Inserting a word via auto-suggest.
[x] Inserting a word via gesture typing
[x] Auto-correcting a word
[x] Deleting characters backwards
[x] Deleting characters forwards
[x] Delete characters backwards, across word boundaries
[x] Selecting a word, then deleting it.
[x] Selecting a range across blocks, then deleting it. (1)
[x] Moving to the end of a word, from inside the word
[x] Moving to the start of a word, from inside the word
[x] Moving to the middle of a word, from the left
[x] Moving to the middle of a word, from the left, then inserting text
[x] Splitting a block at the end. (1)
[x] Splitting a block in the middle. (1)
[x] Joining a block from the start. (1)

(1) Forked the @danburzo app to accept HTML tags like <p></p> and can be found here

I'll keep updating this list.

Nantris commented 6 years ago

@thesunny can you list the browser versions for the Android tests? I'm guessing Chrome 68?

Also, we should agree on which field we're using in that input tester, either the Simple DOM contenteditable or the React-managed contenteditable.

I think it's reasonable to expect Slate to only have out of the box mobile support via React. This would simplify the task, I think. What do you think @ianstormtaylor ?

I believe the only events we should be relying on, if we want the widest base of support on Android, are:

compositionStart
compositionUpdate
compositionEnd
deleteContentBackward (maybe)

FYI when typing, each keystroke updates the composition, but choosing a suggestion ends the composition and creates the final word like this:

compositionData: Barnac Suggestion: Barnacles

If you click Barnacles, the final word is constructed like this.

compositionEnd Barnac insertText les

Barnacles

after this word, Google suggests the word, "the" - If you click this word, it is inserted like this:

insertText the (there is a space inserted before the word)

Note, the above data is from the React contentEditable, but appears to be the same in the simple contentEditable.

thesunny commented 6 years ago

The browser version I believe is locked to the Android API version. I'm using whatever Chrome came with the specific Android version. I guess one thing where you can help is to look up which Chrome comes with which Android version.

I'm using the simple DOM contenteditable because that's what Ian used in his examples.

I was considering writing a web App that would store all this information in a database so I wouldn't have to screenshot everything and we'd also have the raw data as JSON somewhere. I'm not sure how long that would take though so for now I'm just manually see if I can get this all done.

I'm surprised how big the differences are between each Android version.

thesunny commented 6 years ago

@ianstormtaylor Just a heads up that I'm going to redo your Input Tests adding the multiple Android versions.

Edit: Completed tests that don't require blocks as the Input Tester doesn't support them.

Edit: I'm going to go back and add tests for Android API 27. It's a minor upgrade Android 8.0 - 8.1 but just to be safe.

Edit: Also missed Android API 24 (which was mistakenly API 23). Fixing with an @v1.1 to identify that it's been fixed.

Edit: Forked input-methods so that the contenteditable contained two <p> blocks with some text in it in order to complete the tests. https://thesunny.github.io/input-methods/index.html

ianstormtaylor commented 6 years ago

@thesunny these are absolutely amazing! Thank you!

Having them for multiple versions of Android is a great idea. And using the non-React input is best too, since we need to know what exact DOM events are fired, regardless of how React chooses to interpret them. Because we'll probably need to bypass React's event handling logic in places, until they catch up.

Another question for us to answer would be what are the usage levels of the different Android versions.

Regardless, starting by trying to get things working with the latest versions that have Chrome's that fire beforeinput seems best, since that makes things easier I think. There's a decent change that adding support for some of the older ones would be much harder.

thesunny commented 6 years ago

@ianstormtaylor popularity of platform can be found here but unfortunately beforeinput support starts at API27 which has only 2% market share.

If we go back to API 23, we can capture 66.4%. The biggest version in distribution is API 23 which has 23.5% distribution.

I'm just looking through some of the charts and looking for some patterns.

In Inserting Text I think we may be able to get away with:

Ignore keydown events. This is because a keydown will happen before a compositionstart so we can't know if a keydown is part of a composition until later. We can't know if we want to process based on the keydown event.
We process all keyup events that don't happen within a compositionstart and compositionend using a DOM diff. In the test, this is always a space (presumably because it doesn't require composition since it's valid in itself).
When there is a compositionstart then we ignore all the keystrokes. When a compositionend event happens, then we do a DOM diff to create a Slate insert operation.

Some analysis I did on other input methods to see if this strategy holds:

Japanese IME: It appears valid for IME in API 28 as well.
Gesture typing: appears valid for gesture typing
Moving to the middle of a word then inserting text. This appears to work but in some API versions, the text is wrapped in a composition and in others they are just simple key events with no composition. Either way, the logic still works.

It seems like the code might be written something like:

For Android only, we allow all regular insertion keypresses to all flow through naturally (not including ENTER, BACKSPACE, etc)
If we find a compositionstart event, we block all operations until a compositionend event materializes. At the time o f a compositionend, we do a diff of the DOM and create an insert op with {isNative: true}.
If we find a keyup event that is not wrapped in a composition we also do a diff and create an insert op with {isNative: true}

If we aren't in Android, we could theoretically do a diff and use the same code base for everything; however, this might be a performance issue. On the other hand, it could potentially help performance if we debounce key events and end up with a bigger diff and one insert operation at time of keyboard rest. I think we should think about this.

Of course we'd still have to handle ENTER, BACKSPACE, etc. individually but we can probably make a short list of operator keys which is probably ENTER, BACKSPACE, SPACE, and PUNCTUATION.

Usually we don't have Plugin actions from typing in Slate unless they are accompanied with one of these keys anyways. For example, a markdown plugin might use * for a bullet or #heading but all of these are punctuation. I also use o for an empty task myself but that has a space in it. So we would have to limit special Plugin handling to be around these keys but that might be acceptable and it's also a good proxy for the special handling to work in Android anyways because Android will always composition normal words so that key handling would fail in Android.

The plugins may need a different API though that exposes something like onInsertText and onNonWordKey. In this API \we can't do something like have the input onetwothree be converted on the fly to 123 as we type it but we could have one two three work because of the spaces calling an onNonWordKey. This may be a feature since onetwothree live converting could never work in Android due to the way it composes.

ianstormtaylor commented 6 years ago

@thesunny great points all around.

It sounds like one thing we need is a list of key presses that don't trigger compositions and instead use keydown to insert directly? It sounds like space is one of these? But we need a full list. (And as you mention, plugins that want to support Android would need to restrict their keydown usage to these keys, or we need to come up with some new abstraction later.)

Do we need to worry about isNative: true for now? Or can we re-render once the composition has finished with compositionend? Being able to punt that (even just for getting it working with later Android versions) would make it a lot easier.

Fair enough for backwards compatibility. I think whoever implements it would likely want to start with API 28, since it's the most standardized. Or even just ignore Android quirks to start and focus purely on the compositionstart/update/end trifecta. And then handle all of the specific quirks around compositions that need to be ignored.

thesunny commented 6 years ago

@ianstormtaylor

I'll do some more research on what keys have special cases with respect to composition like space. I assume enter is one but I'm not sure about punctuation.

That's an interesting idea about not worrying about isNative: true. So I guess the idea would be that we simply would not update the state (and hence not trigger a re-rerender) until compositionend? Feels like this could work...

In my opinion, we should focus on the compositionstart/end events. Everything I've read on this issue across all of these Editors suggests to me that it is impossible to use the events to reconstruct Editor state without doing a DOM diff with reasonable coverage of Android versions. Since we have to go there eventually, why not start with it. I've designed an algorithm for the DOM diff which I think (hope) will be quite simple to implement so it may not be that hard to get this working.

My next goal is to see if I can create a sample contenteditable React-based editor that uses compositionstart, compositionend and DOM diffing to sync a Slate like Editor State. If I can get that working, we can see how that can merge into Slate proper.

Edit: A preliminary check just on Android 28 suggests that punctuation doesn't trigger composition.

thesunny commented 6 years ago

@ianstormtaylor @Slapbox

OMG! I'm super excited!

I created a working prototype. It's a simple React contenteditable editor (one text node in a div) and tested this on three Android versions (including the newest and oldest API versions) and so far it works perfectly!

composition works
IME works
gesture writing works
auto correct works
suggestions work

It's generalized enough that it works on desktop with no code changes.

I created a custom editor because I don't know Slate internals well enough yet but I tried to make it as close to Slate as possible.

It uses insert_text and remove_text ops with the same semantics as Slate (but without path and marks to keep prototype simple)
They are applied to a text state (similar to Slate's editor state but simplified to one text node).

It works like this:

It reconciles on keyup unless the keyup is within compositionstart and compositionend events.
It reconciles on compositionend events
It uses text diffing to generate the operations and applies them to Editor state

I also track selection and restore it after update but Slate already does that so can be ignored.

Here's a screenshot for now...

I'll try and get this published tomorrow to get some feedback. There will probably be issues but it's promising so far!

thesunny commented 6 years ago

@ianstormtaylor

Here is where you can access the Live Composition Editor

This is the GitHub Repo for Composition Editor

If you type in it from most browsers, it is diffing on every keystroke to create the insert operations and then the Editor is being rendered. After render, I set the selection so the cursor doesn't get lost.

If you use it from Android, when you start a composition via the compositionstart event which happens for most input, the Editor waits until compositionend and then does the diff and creates the operation. You can see this in the operations dump as instead of a single character insertion, you'll see an insert_text with multiple characters in it. In some cases you will also see a remove_text.

I dump the Editor state for transparency which includes all the ops or operations that have been applied.

You can reset the editor to the default text and empty the ops by clicking the Reset button on the page.

@Slapbox can you test this on your Android device?

thesunny commented 6 years ago

@ianstormtaylor

I think to implement this, we would have to change the Plugin event handler properties.

Here's my recommendations.

Add an onCompose synthetic event handler property on plugins.
deprecate onKeyDown and onKeyUp because it will be unpredictable in Android and could break it

We could have onCompose be the default handler for insertion of characters even for desktop where they'd always compose 1 character at a time. We could consider calling it something like onInsertText but the issue is that in some cases (like switching a word using a suggestion) we'd actually have a remove_text operation along with the insert_text operation. Maybe onChangeText? But onCompose feels okay to me even if it's a little misleading.

My suggestion is to replace the key handlers with a whitelist of specially named key handlers for all the cases we want to make sure Slate can handle. This is safer than having a catch all like keydown and assuming the user will know, for example, that letters in a composition won't fire the event or that they need to not preventDefault on those events or that they shouldn't modify Slate during a composition.

Here's handlers I'd recommend to replace onKeyDown.

onHotKeyDown which would fire whenever a key combination (i.e. a character in combination with CTRL, ALT, OPT or CMD is pressed) like CMD+B or CMD+SHIFT+2.
onSpace
onEnter
onDeleteBackward
onDeleteForward
Or possible combine the two above into onDelete with an event property that specifies the direction? Not sure but I kind of like them separate so as to avoid confusion...

I will look into punctuation but I'm not hopeful that we can rely on these being 100% handled outside compositions because of contractions like can't and isn't.

ianstormtaylor commented 6 years ago

@thesunny I think it might be best to separate the work of "getting Android working" from "getting the plugin system fully adapted for compositions". Because the latter is going to be more work, and require thinking lots of different API considerations through.

As for how events flow through the editor...

The <Content> component is the one that renders the contenteditable element in the DOM, and attaches events handlers when it does.
For handlers where we need the native DOM semantics, instead of React's synthetic event semantics, we have to attach them in componentDidMount to the DOM node.
The <Content> component itself is rendered in the After plugin's renderEditor hook, and it passes in the <Editor>'s handler callbacks.
The editor takes any event and runs it through the stack which cascades through all of the plugins.

That's the current flow. There's a bit of confusing indirection around how the <Content> element is rendered. But for the most part you can just look directly from <Content> to plugins and ignore it. In the future we may be able to simplify that stuff.

As for how to handle compositions in plugins. I don't think we need to deprecate onKeyDown/Up yet, because Android helpfully fires the keys with 229 codes so that they won't ever be recognized, last I checked. There may be cases where this doesn't work, but those should be researched/documented well before we make a decision.

I'd rather avoid having onSpace, onEnter, etc. all be their own handlers. As for onDeleteBackward and onDeleteForward, see the "commands" concept discussion.

thesunny commented 6 years ago

@ianstormtaylor Thanks for the feedback.

Just working my way through everything now.

thesunny commented 6 years ago

Here is my execution plan for now

[x] disable state updates in Slate from relevant events
[x] allow native updates to flow through to contenteditable div
[ ] generate diff by locating relevant slate node and comparing it to DOM node (attach initially to keyup events)
[ ] use diff to create operations to modify state and rerender
[ ] now do it with composition events
[ ] enable back all the event handlers and such that are necessary to make slate work properly like hitting enter key

Testing For all these tests we need to make sure (a) the changes are in the Editor State (b) the selection is in the right spot

IME

[ ] go to end of a word and type hideki in Japanese
[ ] go to start of a word and type hideki in Japanese
[ ] go to middle of a word and type hideki in Japanese

Gesture Writing

[ ] cursor to end of word then use gesture writing
[ ] cursor to start of word then use gesture writing
[ ] cursor to middle of word then use gesture writing
Regular Writing
[ ] enter a full word then press space
[ ] enter a full word then press enter
[ ] enter 2 letters then select a suggestion
[ ] type "hi" at end of line then hit "enter" (broken in API28)
Suggestions
[ ] click on a word and select an alternate suggestion
[ ] click on a word, type a letter, select an alternate suggestion
Auto-correct
[ ] type cant and let it auto-correct to can't
[ ] type can and cursor away. Move cursor back to the end of can and type t and then let auto-correct fix it to can't
Physical Keyboard
[ ] type "Hello World" pretty quickly. Make sure there are no duplicate characters or incorrectly positioned selection.
[ ] hold the backspace key down

Nantris commented 6 years ago

@thesunny is there anything in particular I can be of help testing? I checked out what you've put together and it looks very useful. All input and output is rendered to the editor as expected.

One note, not a problem of any sort; I'm surprised by how backspacing is handled. Backspacing away the last character of a completed word fires a single remove_text event, and then another one doesn't fire until you have removed the entire word. Compositions are strange.

birtles commented 6 years ago

Thanks for working on this. Please do test with Firefox on Android too. We are looking to update Firefox's behavior (see discussion in https://github.com/w3c/uievents/issues/202) and would appreciate your feedback.

thesunny commented 6 years ago

@birtles You're welcome. I'll test out Firefox once I get Android Chrome working.

As a quick update, I have many things working properly now although I'm not using the diffing algorithm yet and just the replace algorithm that currently exists. It's pretty much working except there are issues with selection placement, the fact that Android doesn't always have an onCompositionEnd event (which makes no sense) and also the composition events are fired before the content is actually in the DOM.

I will add the special cases we need to test for in the execution plan above.

Nantris commented 6 years ago

@thesunny can you give an example of a situation where onCompositionEnd doesn't fire?

thesunny commented 6 years ago

@Slapbox That's a great question.

One example is if you start typing anywhere in a contenteditable and then cursor out of the word to another word or even to another paragraph. There is no compositionend fired in this case even though the word is finished composing.

Technically, editing anywhere in the document could be considered one composition, but my algorithm (and I think most editors out there) depend on a composition happening within a single node. Otherwise we would need to reconcile the entire DOM to Editor State after each composition instead of just one node.

One issue that is slowing me down, probably the thing that is slowing me down the most, is that Android Studio (what I'm using to emulate mobile Chrome) slows down and crashes regularly. Sometimes the browser behaves differently then after a reboot it goes back to normal. I think it may also be behaving differently when I'm using the remote console/debugger with it. This is frustrating as I have 6 instances running in order to test properly. If anybody has any suggestions on making this more stable, please let me know. Maybe I just need to buy 6 Android tablets!

Nantris commented 6 years ago

I've been thinking about how to remedy the issue of difficulty emulating, but I've come up with nothing unfortunately. You're running 6 emulators though? It seems like just 2-3 should do.

Regarding the issue with no compositionend event in those cases, we could could compare the composition on each update of the selection of the editor, and when the composition is updated to have an entirely new word, we can assume we should output the fragment from the old composition before carrying on. Does that remedy this problem and can you see any issues that introduces?

thesunny commented 6 years ago

@Slapbox

Right now I'm just trying to get through 2 emulators at a time but all 6 versions work differently so I need to test on all of them eventually.

Unfortunately, the compositionend issue is complex. Updating on every compositionupdate doesn't solve the problem without introducing new ones.

Right now I'm building a shouldChangeText object that provides different logic on when the text should be changed in the editor state. I'm probably going to write a version for each API version of Android and a default version for everything else.

I was joking before but decided to buy 3 Android tablets for testing. If they seem to work well and we can upgrade/downgrade them to different API levels, I'll probably end up with 8 tablets for testing at different API levels.

Nantris commented 6 years ago

@thesunny, first, thanks again for all your hard work!

Which APIs are you aiming to support currently? Limiting it to 28+ seems like it should cut the number of versions you need to worry about quite a lot, no?

gaearon commented 6 years ago

Hey folks — just a heads up that I'm currently in the process of going through React DOM bugs. If you have something actionable for React that we need to fix on our side please create an issue with suggestions and I'll try to help.

thesunny commented 6 years ago

@gaearon thank you for your offer of help.

Ian listed three React issues in his first post above which I'm sure he'd be grateful to have looked at:

React issues: https://github.com/facebook/react/issues/4079 https://github.com/facebook/react/issues/6176 https://github.com/facebook/react/issues/11231

For all of the contenteditable writers out there, I'm sure we'd all love React to fire a single event which notified the end of a composition reliably and an input method that doesn't fire in the middle of a composition.

I'm working on that logic now but it seems a pity that every contenteditable author will have to figure this out on their own.

I'm not sure how high up this is on your list (I know you have Draft.js at facebook which I have used and contributed to) as it might be too specific to put into a general React release.

@Slapbox

I'm aiming at API 23+ but I'm starting with 28 and 27 right now..

The emulators are killing me so I am buying 3 Android devices to start which I might expand to 8. API 28 adoption is non-existent so not a good target for me personally as I want to support general Android with my web app. API 23+ captures 66% of market share. I can buy a brand new Android Tablet today at Best Buy that comes with API 23 (Marshmallow) so that feels like a good minimum bar.

gaearon commented 6 years ago

Do you think https://github.com/facebook/react/issues/4079 and https://github.com/facebook/react/issues/6176 are closed incorrectly and we have something actionable for React itself there?

I'm not sure how high up this is on your list (I know you have Draft.js at facebook which I have used and contributed to) as it might be too specific to put into a general React release.

I think our general stance is that if it involves a lot of code and is specific to contenteditable it's more likely to end up in Draft (or somewhere else) than in React core because we have constraints on React bundle size.

That said we can take small fixes to React.

thesunny commented 6 years ago

@ianstormtaylor

A few questions I'm hoping you can help me with:

(1) after each render in Content the updateSelection method is called. It seems to set the selection in the browser after every render even if the selection in Editor State hasn't changed. This is problematic because render sometimes happens before I've updated the selection. My questions are (a) is this true that it will set the selection even if it hasn't changed and (b) if it is true, do you have any objection to me writing code that checks if the selection has changed and only updating it if it has. Not sure what the side effects might be. Maybe there is a reason it's this way.

(2) are there any other places other than the event handlers in after where Slate's Editor State is being updated? Related to (1) in that I am getting re-renders but I can't figure out what's causing them. I feel like I intercepted all the relevant after methods. If I don't get any unwanted renders, I may not have to worry about (1).

thesunny commented 6 years ago

Just a heads up to everyone that I won't be able to work on this for about a week so you may not see any (or little) activity here. I'll be back into this issue the week after.

thesunny commented 6 years ago

Just some good news before I go. I have it working as a proof of concept in API28 now with a few selection issues. I removed a bunch of the existing after.js plugin code temporarily so I had less things to think about but text entry including suggestions is working. IME and autocorrect are also working but the cursor is in the wrong place which is what I'll be working on until I see you all next. :)

thesunny commented 6 years ago

@gaearon sorry about that, I thought these were issues that Ian wanted to fix. I mixed it up with the following which is what I should have posted. He mentioned in this Slate issue https://github.com/ianstormtaylor/slate/issues/2060 that he was trying to get this React issue solved to do with using native beforeinput https://github.com/facebook/react/issues/11211

ianstormtaylor commented 6 years ago

@thesunny feel free to make any changes to the selection updating logic. Right now it does update on every render (although I think it checks to see if it needs to, not sure), but there's no reason we have to do that specifically. It just needs to ensure that the selection doesn't get out of date.

Not sure about the other places for re-renders.

@thesunny @Slapbox @gaearon I think it's pretty reasonable for React to not include any contenteditable-related conveniences. This is the domain of Draft.js and Slate to provide these things, and if they happen to be shareable then they should be separate libraries. Trying to paper over compositionend for example is something that would be super complex (if not impossible) for React to manage, and <1% of their users will ever care about it. This definitely doesn't belong in React, and belongs in Slate instead.

@gaearon as for browser bugs, I'm not sure we have bugs specifically. But one of the big issues right now that we face with React is the discrepancy between real and synthetic events. For example...

https://github.com/facebook/react/issues/11211 - The native beforeinput is not accounted for in React right now, which requires us to attach our own listener, which behaves slightly differently. This seems like a fairly easy fix in that it just needs to update the event plugin to check for the real beforeinput event in browsers that support it (Safari, Chrome, Opera, iOS Safari, Chrome for Android). This would be awesome to get support in React core.

https://github.com/facebook/react/issues/5785 - Similarly, the onSelect handler for React has some limitations, and we're forced to attach our own selectionchange listener instead to get around them. It would be nice if React exposed the native selectionchange and selectionstart event handlers I'd think, since these are supported across most browsers going back a long ways. The only complication here might be the high frequency of them being triggered? I'm not sure what the limitations on the synthetic event pool are, but this might run into them.

https://github.com/facebook/react/issues/13104 - This is another one. If React added the beforeinput from above, we'd still need to dip into the nativeEvent right now to check the isComposing flag I think.

thesunny commented 6 years ago

@ianstormtaylor @Slapbox @gaearon

Hi Ian, I understand the reasoning for not including any contenteditable specific fixes; however, I'm not sure composition events shouldn't be normalized in React as they also apply to input elements.

React normalizes other events across browsers in order to make sure that writing an app using React events means your app will more or less work on all platforms.

It feels inconsistent to support composition events in React but them not being portable across browsers like the othe React events. So, in order words, my instinct is that composition events either (a) shouldn't be handled at all in React which indicates developers need to do the work to make them cross platform compatible because React isn't involved or (b) that React normalizes them so developers don't need to worry about them.

At this point, any app that uses composition events to propagate changes will fail on at least one version of Android.

One thing that perhaps should affect the decision on whether to consider fixing/augmenting/creating new synthetic events is how complicated the algorithm actually is. Figuring out how to fix this in Slate has been hard but I think there's a possibility that the solution is actually quite simple and elegant. It may only be a few lines of code and save a lot of headaches.

ianstormtaylor commented 6 years ago

@thesunny that's fair, if there's a simple solution for it. From the threads I've read about Android composition events though it sounded like things weren't super simple, but I don't have the context.

thesunny commented 6 years ago

@ianstormtaylor it’s not clear to me yet whether the solution is simple but there’s nothing I’ve seen so far that prevents it from being so.

The issue I’m seeing with compositionend is that in some implementations it either fires too soon (before the input) or doesn’t fire at all and only fires a compositionupdate.

In the first, I call set timeout but only use the callback if input event doesn’t fire In the same tick. If input does fire then I call the callback from the input event. Logic is tiny and it works.

In the second case a compositionupdate fires but I haven’t yet analyzed the logic to see when that should have been a compositionend. Hopefully it can be solved and is simple (it may require logic to do with selection position).

birtles commented 6 years ago

For what it's worth, we're looking to change the order of compositionend and input in Firefox to match Chrome. (We're just waiting to clarify what value of isComposing would be most useful to Web authors for this series of events.)

ianstormtaylor commented 6 years ago

@birtles do you know what the status is on implementing the beforeinput event from Input Events Level 2 (or even Level 1) is in Firefox? That’s one of the biggest issues.

birtles commented 6 years ago

We're hoping to tackle beforeinput in Q4. From what I understand it's a pretty massive undertaking though. We should get inputType done this quarter, though.

thesunny commented 6 years ago

@ianstormtaylor do you know why there is a a setState({isComposing: false}) in the before.js plugin? I presume it's required for a certain browser?

If I take this code out, my Android code in API28 appears to work. If it's in there, it fails because the selection is not where it's natively supposed to be (presumably because the re-render places the selection where it used to be).

Right now, I'm planning to disable it selectively (i.e. for Android only) but kind of feel icky about it.

  /**
   * On composition end.
   *
   * @param {Event} event
   * @param {Change} change
   * @param {Editor} editor
   */

  function onCompositionEnd(event, change, editor) {
    isStrictComposing = false
    editor.isStrictComposing = false
    const n = compositionCount

    // The `count` check here ensures that if another composition starts
    // before the timeout has closed out this one, we will abort unsetting the
    // `isComposing` flag, since a composition is still in affect.
    window.requestAnimationFrame(() => {
      if (compositionCount > n) return
      isComposing = false

      // HACK: we need to re-render the editor here so that it will update its
      // placeholder in case one is currently rendered. This should be handled
      // differently ideally, in a less invasive way?
      // (apply force re-render if isComposing changes)
      if (editor.state.isComposing) {
        editor.setState({ isComposing: false })
      }
    })

    debug('onCompositionEnd', { event })
  }

ianstormtaylor commented 6 years ago

@thesunny the comment above says why, it's to re-render the placeholder so that it will disappear if there is content in the editor I think. If you search issues there's probably a reference to compositions and placeholders that someone opened.

ianstormtaylor commented 6 years ago

@thesunny searching leads to: https://github.com/ianstormtaylor/slate/blob/36ed4397d880bf75126e9f8f890e999fc69c3b9a/packages/slate-react/src/plugins/after.js#L740

thesunny commented 6 years ago

@ianstormtaylor sorry, I'm not clear what the placeholder is.

thesunny commented 6 years ago

@ianstormtaylor Never mind... I see it now. I thought it was a special Slate construct that was either (a) a placeholder for the cursor in collaboration or (b) a place where in inComposition items were built. But, I see that it's just the DOM placeholder. :)

thesunny commented 6 years ago

Just an update, I have a working version in API28 now. Going to try API27 now...

edit: there were 2 unhandled edge cases with API28. Fixed one. Working on second (compositions that take place across multiple nodes). It's going slower as we started Y Combinator's Startup School which takes up time.

thesunny commented 6 years ago

@ianstormtaylor Can you help me? I've been stuck here for several hours but have narrowed it down to this:

This is document value from change.value:

Then this change.insertTextAtRange operation happens:

And finally we get this:

What happens is the second paragraph gets deleted (set to "") and the text "Hello there" is moved to the end of the first node. The target of the change.insertTextAtRange though is at path [1, 0] which is the second paragraph.

The place where the Hello there was inserted was where the DOM cursor was set at the beginning of the composition but I'm not sure why the cursor position at that time should have an impact on the change.

Nantris commented 6 years ago

I noticed at the time that Slate transitioned to points that there was some weird behavior regarding line-endings. Is there any chance it's related to a bug like the one I reported here? https://github.com/ianstormtaylor/slate/issues/2050

thesunny commented 6 years ago

Thanks for chiming in @Slapbox . It helps when you dive into these issues.

I figured it out but the method behavior is not intuitive and I suggest we change it.

When you insertTextAtRange it deletes at the argument range then inserts text at the value's selection range if the current value's range is in a different node than the argument range. This doesn't make sense. It must be used to solve an edge case somewhere but it's bad behavior.

The method is named insertTextAtRange and it's not what it does.

ianstormtaylor / slate