fix editing with "soft keyboards" (eg. Android, IMEs)

ianstormtaylor commented 6 years ago

Do you want to request a feature or report a bug?

Bug.

What's the current behavior?

The issue with using Slate on Android is complex, and due to a difference in how its OS keyboard is designed. On mobile devices, keyboards are starting to move away from the "keys" concepts in many ways...

Autocorrect will make changes without any "key" being press.
Autosuggest will insert words that don't map to keys.
Swipe-to-type will insert entire words in one go, instead of using keys.
etc.

Because of this, it sounds like the Android team (reasonably from their point of view) decided to not reliably fire key-related events.

As soft input methods can use multiple and inventive ways of inputting text, there is no guarantee that any key press on a soft keyboard will generate a key event: this is left to the IME's discretion, and in fact sending such events is discouraged. You should never rely on receiving KeyEvents for any key on a soft input method. —KeyEvent, Android Reference

It sounds like the behavior is:

The keypress event is never triggered, which is fine for us.
Text-based keys like a, b, etc. fire a keydown event but always with a key code of 229, indicating that the key is unidentifiable because the keyboard is still busy processing IME input, which may invalidate the actual key pressed.
Pressing keys like enter fires a keydown event as normal, with an event.key that can be recognized? (This is unclear whether this happens or not.)
Pressing backspace does not fire a proper keydown event.

A few different resources:

Great Android postmortem writeup: https://docs.google.com/document/d/1Hex89Di-r-Wfpo1DLAtxpetoX588ziXVoNyC87Je3Xc/edit#
Chrome "bug": https://bugs.chromium.org/p/chromium/issues/detail?id=118639
React issues: https://github.com/facebook/react/issues/4079 https://github.com/facebook/react/issues/6176 https://github.com/facebook/react/issues/11231
VSCode's IME test docs: https://github.com/Microsoft/vscode/wiki/IME-Test
@danburzo's input methods: https://github.com/danburzo/input-methods

What's the expected behavior?

The fix for this is also complicated. There are a handful of different, overlapping pieces of logic that need to change, to accommodate a handful of different input types...

The first stage is to handle basic insertions, and auto-suggestions...

[ ] Remove the preventDefault in onBeforeInput, so that the DOM is updated, and the onInput logic will trigger, diffing the insertion and then "fixing" it.
[ ] Update the <Leaf> (and other?) components to increment their key such that React properly unmounts and reconciles the DOM, since it has changed out from under it.

This is actually the same starting steps as is required for https://github.com/ianstormtaylor/slate/issues/2060, so I'd recommend we solve that issue in its entirety first, to work from a solid base.

This fixes the actual text insertion pieces, and probably deletions as well. Splitting blocks can still be handled by enter because it still provide proper key codes.

[ ] Check that all other behaviors that aren't text insertions (eg. splitting blocks) are handled properly without access to many keydown events.

And then there's some selection issues, which apparently can mess up Android's IME (and potentially others) if the selection is manually changed during a composition.

[ ] Prevent re-rendering the editor on compositionstart and compositionend.
[ ] Prevent updating the selection while a composition is taking place.

I think this would solve the 90% case for soft keyboard input.

Separately, there's still another question of how to properly handle these kinds of behaviors other plugins. For example if a plugin uses backspace at the start of a block to reset that block, that won't work on Android. So after solving the input issues, we need to step back to an architectural level and solve this plugin handling problem. But that can wait.

Nantris commented 5 years ago

@thesunny this is excellent!

One issue I'm seeing is on the two recordings that span multiple lines, I can't follow the directions due to my presses getting caught on the word "Instructions," which overlaps with the editable text.

thesunny commented 5 years ago

@Slapbox

Thanks, I'll fix this when I'm in the office. Not sure if I'm going to be in before the new year or not.

thesunny commented 5 years ago

@Slapbox Fixed. @rbar2 Fixed.

I've started work on Android now. With the event recorder, I have a better understanding of what's going on and I'll be able to get through the issues easier. I don't have a set time frame to work on it this time. My plan is to work on it until it's done.

julienR2 commented 5 years ago

Thanks a lot for your work @thesunny ! We are migrating to Slate and hitting Android issues too. I have been catching up during the past week about this and I'm glad to see so much activity on this thread about it ! I have recorded some sessions on your editor-events, if you need some help I'd be more that pleased to give a hand on this one !

thesunny commented 5 years ago

Just a friendly update.

Got parts of Android 8 and 9 working (3 API versions). Virtual keyboard, auto-suggest, auto-correct, gesture writing are working. Haven't checked IME yet (though I expect it will be working). Fixing a bug with splitting blocks with the cursor being in the wrong position.

thesunny commented 5 years ago

I've published an early version of Android support. There are still some bugs but if you'd like to give it a try, please go for it and report back.

https://thesunny.github.io/slate/#/rich-text

I have tested with API 25, 27 and 28. 26 is probably okay. Haven't done any work on 24 or 23.

To report an issue, please mention which Android API version you are on.

If you want to know which API version you are running then visit this URL.

https://editor-events.herokuapp.com/

It may take a bit of time to spool up because it's on the free tier of Heroku. It will say something like "You are using an Android browser with this API version: API-25" in the third paragraph. The API version will be in green.

thesunny commented 5 years ago

Things to mention in a bug report:

[ ] Android API version
[ ] What device or whether you are using an emulator
[ ] Whether you are using gboard (the default) or some other virtual keyboard
[ ] What specific steps are required to reproduce. Please don't post if you can't reproduce.

KrisBraun commented 5 years ago

Played a bit with api 28 with SwiftKey and found none of the issues I've seen previously. 👌💯

Nantris commented 5 years ago

@thesunny this is amazing and really close to fully functional from what I can tell. Here's the bugs I did find.

Key	Value
API Versions	26 & 28
Physical vs Emulator?	Physical devices
Keyboard	Gboard

Note: API 28 and 26 appear to be identical in in their behavior for all of these as best as I can tell.

Single character words

Swipe in This is a - The cursor then sets itself like this: This i|s a

Seems to be the most obvious example of a more subtle bug related to single character words in general, but most prominently in swipe. Another example of the behavior, swipe inThis was a and you'll get This w|as a The single character words can also result in text on the next line down, though I'm not sure how to reproduce that yet.

Cut All/Delete All

Cutting All and Select All/Delete lead to slightly different outcomes. Cutting All seems to truly remove every character, whereas deleting all seems to leave one whitespace character of some sort.

On the History example:

Cutting all contents and then attempting to paste back can sometimes not work because the last character of the placeholder text steals focus. (Backspace in this situation results in the same, the last character of placeholder selected)
- Select All, Copy, then Delete all contents of the undo/redo example, then Select All and finally Paste. When you paste over that whitespace character the first paragraph gets swallowed up.

Enter/Backspace

Variations of bugs related to Enter followed byBackspace and vice versa

Undo/Redo

Some issues in undo/redo behavior, but frankly dramatically less than I would have expected. I haven't identified any useful patterns here yet.

thesunny commented 5 years ago

@gaearon

I was hoping you could help with this or point me in the right direction or to the right person as it's a little deeper than standard React and requires (a) an understanding of what React is doing under the hood (b) what we can expect React to do reliably in the future and (c) a potential new API to help re-stitch or re-sync DOMs with the virtual DOM. Any solution here will likely help resolve DraftJS issues as well.

Below is a before and after scenario.

In Android, we put the cursor in the word "editable". We place the cursor which is denoted by the pipe as follows "edit|able". Then we hit ENTER on the Android virtual keyboard.

It is impossible to detect that ENTER has been pressed in a reliable way because the implementers of Gboard have decided not to deliver that information in the events. There is discussion spanning years where rich text editor developers have been asking for this and the Gboard developers have said no so it's unlikely to arrive and if it does we still have old versions of Android.

So we have to let the operation continue and we end up with the DOM in the before and after state shown below:

<!-- BEFORE -->
<div data-key="1969" style="position: relative;"><span data-key="1970">
    <span data-slate-leaf="true" data-offset-key="1970:0">
      <span data-slate-content="true">This is editable </span>
    </span><span data-slate-leaf="true" data-offset-key="1970:1"><strong data-slate-mark="true"><span data-slate-content="true">rich</span></strong>
    </span><span data-slate-leaf="true" data-offset-key="1970:2"><span data-slate-content="true"> text, </span></span><span data-slate-leaf="true" data-offset-key="1970:3"><em data-slate-mark="true"><span data-slate-content="true">much</span></em>
    </span><span data-slate-leaf="true" data-offset-key="1970:4"><span data-slate-content="true"> better than a </span></span><span data-slate-leaf="true" data-offset-key="1970:5"><code data-slate-mark="true"><span data-slate-content="true">&lt;textarea&gt;</span></code>
    </span><span data-slate-leaf="true" data-offset-key="1970:6"><span data-slate-content="true">!</span></span>
    </span>
</div>

<!-- AFTER -->
<div data-key="1969" style="position: relative;">
  <span data-key="1970">
    <span data-slate-leaf="true" data-offset-key="1970:0">
      <span data-slate-content="true">This is edit</span>
    </span>
  </span>
</div>
<div data-key="1969" style="position: relative;">
  <span data-key="1970">
    <span data-slate-leaf="true" data-offset-key="1970:0">
      <span data-slate-content="true">able&nbsp;</span>
    </span><span data-slate-leaf="true" data-offset-key="1970:1"><strong data-slate-mark="true"><span data-slate-content="true">rich</span></strong>
    </span><span data-slate-leaf="true" data-offset-key="1970:2"><span data-slate-content="true"> text, </span></span><span data-slate-leaf="true" data-offset-key="1970:3"><em data-slate-mark="true"><span data-slate-content="true">much</span></em>
    </span><span data-slate-leaf="true" data-offset-key="1970:4"><span data-slate-content="true"> better than a </span></span><span data-slate-leaf="true" data-offset-key="1970:5"><code data-slate-mark="true"><span data-slate-content="true">&lt;textarea&gt;</span></code>
    </span><span data-slate-leaf="true" data-offset-key="1970:6"><span data-slate-content="true">!</span></span>
    </span>
</div>

What's happened is, from the cursor position, it's cloned all the interim <span> elements up to the enclosing <div> element. Then its moved all of the later <span> elements into the cloned <div>.

So my strategy is to restitch the DOM back into a state where React doesn't end up being confused.

My idea is to delete the second <div data-key="1969"> and move its children <span data-key="1970:1"> through <span data-key="1970:6"> back into the first <div data-key="1969">.

At this point, in the Slate editor, I'd be then making changes to Slate's state to perform the ENTER action within Slate instead of directly in the DOM. This detail isn't necessary to answering the question but provided for context.

So my question is, is this manual dom manipulation enough so that React can sync to the DOM without being confused or do I need to do more? Is there an API to force a resync under the assumption that we can't believe anything that's in the DOM (e.g. like syncing the first time from SSR). Or given your knowledge and background, should I be taking a different approach altogether?

thesunny commented 5 years ago

@gaearon

You asked earlier if there was anything actionable in order to help with this issue. One feature that could enable broken DOM issues related to Android (I believe there may be other cases in Android and potentially in other browsers as well) is to have a function like:

ReactDOM.stitch(element)

Which would fix the given element and all its children in relation to the virtual DOM:

Move any existing children back into the proper positions underneath it if they have moved elsewhere
Recreate any children and their event handlers if they have been removed from the DOM
Remove any children that aren't supposed to be there (i.e. do not exist in the virtual DOM)

I'm assuming that React internally keeps references to elements so that the element can be used to look up the right position in the virtual DOM; however, I'm not familiar with React internals.

hanna-greaves commented 5 years ago

Hey @thesunny! Thanks for working so hard on this, you're doing an awesome job :)

I just wanted to report an issue I had seen with IMEs specifically when testing your current build. I know you're focusing mostly on Android at the moment, but thought it wouldn't help to let you know!

Cursor position incorrect when typing with IME When typing using an IME the cursor does not move correctly with letters, rather it stays are the beginning of the typed fragment.

Repro steps:

Select Japanese or Chinese IME on Windows or Mac
Focus anywhere in the slate editor on your build
Type anything

Expected outcome: The cursor moves along as you type

Actual outcome: The cursor remains static until you complete your input with return

P.S. I have capacity to help out if you would like any extra hands on the code.

thesunny commented 5 years ago

@sgreav I'll keep this in mind but this will likely be addressed separately.

I'm building Android support before the plugin that processes compositions in other browsers and its only added for Android. That said, I feel like in the future, we may be able to merge the composition handling that I'm building with that used by other browsers. This would likely solve the issue since the cursor issue sounds a lot like the issues we are seeing in Android.

jimshute commented 5 years ago

@sgreav @thesunny I got the same problem while typing with IME. It is ok when I typing in the middle of the block, and the cursor goes where I am typing. The problem apears when I type at the end of a block, while the cursor always stay where I was start typing. What's more, the editor will stop working with whitespace at the end. After that I will no longer delete any charactors.

thesunny commented 5 years ago

@jimshute is this on Android or another browser? If it's on Android, please let me know which API version.

Reposting myself but instructions on finding API version:

If you want to know which API version you are running then visit this URL.

https://editor-events.herokuapp.com/

It may take a bit of time to spool up because it's on the free tier of Heroku. It will say something like "You are using an Android browser with this API version: API-25" in the third paragraph. The API version will be in green.

thesunny commented 5 years ago

Important Update and Request for Testing

Hello everybody, this appears to be a small feature but is an important step forward. enter is working on Android API 27.

This is important because Android rearranges the DOM on enter breaking React and we can't stop the event from happening. I was afraid that this was impossible to recover from. After 3 different approaches at reassembling the DOM so that it would work with React again, the third approach is working. Other approaches reconstructed the DOM but React couldn't sync with it. I believe this same code will be able to help resolve the other Android issues.

If you can test to make sure that it is working on your API 27 (Android 8.1) devices, the latest version can be found online at:

https://thesunny.github.io/slate

Preliminary tests show that this is broken on API 26 (Android 8.0) and probably every other API version as well. I'll start working towards fixing those.

thesunny commented 5 years ago

Actually, with a small fix this is working on API 26 in addition to API 27 now.

thesunny commented 5 years ago

Also working on API 28 and ready for testing.

thesunny commented 5 years ago

Enter should be working on API 24 through API 28 (Android 7.0 through to Android 9.0)

I think I will actually not support API 23 because the Editor doesn't even render even before we start trying to add Android support. API 23 appears to be missing some JavaScript features like Array.includes so it would be a fair bit of work to get it going.

adjourn commented 5 years ago

@thesunny What are the conditions where API 23 doesn't work for you?

Just tested it, works fine with API 23 tag, tried Chrome and Samsung browser.

thesunny commented 5 years ago

@adjourn Hmm... I'm using Android Studio. What device are you using specifically?

adjourn commented 5 years ago

@thesunny My old Samsung S5 (Android 6.0.1).

thesunny commented 5 years ago

@adjourn

Hmm... I am running Android 6.0.0 instead of 6.0.1 so I wonder if that's the difference.

I'm getting this error:

which I presume is referring to the Array.includes method

@Slapbox

Can you tell me which of those errors you are still seeing? Please be very specific in how I can recreate the error.

API version
Make sure to reload the editor from scratch
Where in the editor do you first touch
What specific actions do you do (type on virtual keyboard, use hardware keyboard, use gesture writing, etc.)

Many of these issues I'm having trouble recreating because they aren't specific enough.

jimshute commented 5 years ago

@jimshute is this on Android or another browser? If it's on Android, please let me know which API version.

Reposting myself but instructions on finding API version:

If you want to know which API version you are running then visit this URL.

https://editor-events.herokuapp.com/

It may take a bit of time to spool up because it's on the free tier of Heroku. It will say something like "You are using an Android browser with this API version: API-25" in the third paragraph. The API version will be in green.

@thesunny I got the API version on my Android 9.0，which is API-28. But I find that it is a general problem of Chrome on any plateform, including windows10, MacOSX, and Android. I've tried on Chrome/Firefox/MS Edge. When using Firefox/Edge，I can type the words on the end of a paragraph(block) even though the cursor is not in the correct position. But chrome will cause to the problem of typing and deleting disability.

thesunny commented 5 years ago

More bug fixes deployed to

https://thesunny.github.io/slate

API 26/27 Enter Bugs and Fixes

Note: These are all separate issues that have to be fixed separately. Please test all cases and more. It's possible that I may have broken one when I fixed another.

[x] Hit enter in the middle of a word
[x] Hit enter at the beginning of a word
[x] Hit enter at the end of a word
[x] Hit enter at the end of a word before a bold word
[x] Hit enter in the middle of a style word
[x] Hit enter at the end of a line (2019/01/10)
[x] Hit enter at start of line
[x] Hit enter two times (cursor ends up on wrong line... possible React sync breakage) (2019/01/10)
[x] Hit enter on an empty line
[ ] Hit enter at beginning of line
[ ] Hit enter at beginning of document
[ ] Later: Hit enter at end of document (after resolving end of line issue)

thesunny commented 5 years ago

API 26/27 Delete Bugs

[x] Hit backspace from start of a paragraph to join the paragraph before
[x] Hit backspace in an empty paragraph
[x] Hit backspace into an empty paragraph

signalwerk commented 5 years ago

@thesunny I'm reading this thread since a couple days. I'm not even having an android but I just wanted to say; thank you! The work you do is amazing! Thanks a lot!

thesunny commented 5 years ago

@signalwerk and thumb-uppers, thanks for the encouragement! :) It means a lot.

In turn, I'm thankful to @ianstormtaylor and his organization for providing this amazing piece of open source software.

alanchrt commented 5 years ago

If helpful, it looks like it's possible to automate keyboard interactions using monkeyrunner.

Here's a quick proof of concept: https://imgur.com/a/4OXJI2W

The scripting is pretty basic (python):

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

device = MonkeyRunner.waitForConnection()
device.startActivity(
    action='android.intent.action.VIEW',
    data='https://thesunny.github.io/slate/#/smoke',
)

MonkeyRunner.sleep(7)
device.touch(50, 1000, MonkeyDevice.DOWN_AND_UP)
MonkeyRunner.sleep(1)
device.drag((674, 1370), (820, 1200), 0.3)
MonkeyRunner.sleep(5)

Monkeyrunner can apparently screenshot as well, so theoretically a test suite could be built up using visual diffs for failures. I don't know much about the Android dev world and whether or not there are good ways to orchestrate that-- ideally running against VMs/cloud services, not on physical devices over USB.

Nantris commented 5 years ago

@thesunny hopefully this helps with recreating the single character word related bugs. Demonstration - recorded this on API 28 - I cleared all the site data and reloaded the page before recording (which I did just moments ago, so it includes all of your newest changes.)

I won't be able to test if this still occurs on API 26 for at least a few more days, but I assume it does.

Recap of the "Single Character Words" bug:

In a sequence of swiped words, a single character word causes the caret's focus and anchor to move to the first character of the preceding word.

BTW: I meant to type, "This is a" not "if" but had already done a number of takes and this was good enough to demonstrate the issue. I just wanted to clarify that that bad English is due to my sloppy swipes and not relevant to the issue.

I used DU Recorder to make the video and then uploaded to Imgur who automatically convert the video to a GIF. Hope this helps streamline things for some other folks here.

Nantris commented 5 years ago

Enter/Backspace bug:

Demonstration

Steps to recreate:

At the end of a line, press Enter.
Press Backspace to erase the newly created line.
Press Enter to add that line back in and you'll get a crash

Nantris commented 5 years ago

Cut All vs Delete All behavior

This one is definitely not a show-stopper, but the odd behavior here might be helpful in understanding the nature of some other bug.

Cut All and Paste Behavior (When editor has placeholder at least)

Focus gets stuck on last character of placeholder

Delete All and Paste Behavior

Pasted paragraph is somehow swallowed by a whitespace character left behind after deleting all selected contents.

thesunny commented 5 years ago

@Slapbox Thanks for the demo of the single word bug.

I'm working on all the backspace bugs right now. Found a solution requiring fixing the DOM (like the enter bug but encompassing more elements to fix) and will post when it's ready.

Thanks also for the cut demo.

thesunny commented 5 years ago

Backspace bugs appear to be fixed and is at https://thesunny.github.io/slate now.

Entry and Reconciliation Bugs

[x] Click in the middle of a word and select an alternate suggestion without errors
[x] Click in the middle of a word and select an alternate suggestion and make sure React Value is resolved: Note this is due to the logic when you cancel the delete (suggestion detection informs that delete is actually a suggestion) that cancels the reconcile from the compositionEnd you have to reinstate the reconcile.
[x] Fix multi-edit. Start inserting text in a block. Then click in a span after a different mark (e.g. bold). Insert more text. Click outside the editor. Both the changes should be committed to the Editor not just the last one. This is the multi-edit problem and it exists in API 27/28.

thesunny commented 5 years ago

Almost done for the day and taking off for the weekend.

Feels like API 26/27 is almost done. It almost never gets into a bad state now. Hitting enter at the beginning of a line and doing multiple edits in a row are the two issues. The latter should be easy (solved it before) and the former is of unknown difficulty.

Thought some of those following this issue might be interested in the backspace and enter handling notes. They will be left in the code so that others will know what's going on.

Backspace Handling

Save the state using a snapshot during a keydown event as it may end up being a delete. The DOM is in a good before state at this time.
Look at the input event to see if event.nativeEvent.inputType is equal to deleteContentBackward. If it is, then we are going to have to handle a delete but the DOM is already damaged.
If we are handling a delete then:
- stop the reconciler which was started at compositionEnd because we don't need to reconcile anything as we will be reverting dom then deleting.
- start the deleter which will revert to the snapshot then execute the delete command within Slate after a requestAnimationFrame time.
- HOWEVER! if an onBeforeInput is called before the deleter handler is executed, we now know that it wasn't a delete after all and instead it was responding to a text change from a suggestion. In this case:
- cancel the deleter
- resume the reconciler

Enter Handling

Save the state using a snapshot during a compositionEnd event as it may end up being an enter. The DOM is in a good before state at this time.
Look at the native version of the beforeInput event (two will fire a native and a React). Note: We manually forced Android to handle the native beforeInput by adding it to the content code.
If the event.nativeEvent.inputType is equal to insertParagraph or insertLineBreak then we have determined that the user pressed enter at the end of a block (and only at the end of a block).
If `enter is detected then:
- preventDefault
- set Slate's selection using the DOM
- call splitBlock
- Put some code in to make sure React's version of beforeInput doesn't fire by setting a variable. React's version will fire as it can't be cancelled from the native version even though we told it to stop.
During React's version of beforeInput, if the data property which is a string ends in a linefeed (character code 10) then we know that it was an enter anywhere other than the end of block. At this point the DOM is already damaged.
If we are handling an enter then:
- cancel the reconciler which was started from the compositionEnd event because we don't want reconciliation from the DOM to happen.
- wait until next animation frame
- revert to the last good state
- splitBlock using Slate

thesunny commented 5 years ago

Update and a deploy. I think all of the enter handling and backspace bugs have been fixed in API 26/27 (Android 8.0 and 8.1). Please test.

https://thesunny.github.io/slate

The one issue that I haven't resolved yet and may not attempt to until after an initial release is holding down the backspace key or hitting backspace repeatedly quickly. I think when this is happening, a second backspace is able to manipulate the DOM before the first backspace handler has fixed the DOM and then committed the enter handling from Slate to the DOM.

Please report any bugs for API 26/27 here. I will be working on API 28 edge cases so if you have an API 28 device (Android 9), please check here once in a while if you'd like to help in the testing process.

Thank you everyone that has contributed a bug report, either here or via Slack.

thesunny commented 5 years ago

Merged in the new navigation from master that came from this PR: https://github.com/ianstormtaylor/slate/pull/2450 (thanks @ericedem ).

I then added subpages support for each example so there can be more than one set of content per example. This feature is so that I could add manual tests for Slate compositions which are available now here:

https://thesunny.github.io/slate/#/composition/split-join

It's a start but the goal is to have a suite of tests that we can have users run through that will validate that compositions are working on a specific device.

I developed this before continuing work on API 28 because it was easy to miss specific cases to test. As we add more API versions and to check compatibility for other devices like iOS or desktop IME, a set of tests will have incrasing value.

thesunny commented 5 years ago

Fixed issues related to splitting/joining blocks and inserting text in API28 also known as Android 9.0. It may be fairly stable now.

Anybody who is able to help, please test these changes on your device or on Android studio by running it through the two sets of tests on this page:

https://thesunny.github.io/slate/#/composition

The tests are designed to trigger as many edge case bugs that I know of but I expect there are many that I haven't run into yet. Please see if you can get it to break in other ways here:

https://thesunny.github.io/slate/#/rich-text

Please note your API version when submitting an issue here. Thanks.

I will be starting work on API 25 and earlier next.

rbar2 commented 5 years ago

Device: Google Pixel 2 API: 28

Following the instructions on "composition" - WORKS!

Playing around in "rich-text" - if I click between the "c" and "h" in "rich", then hit space, then hit delete.. it deletes the space and the "c"

thesunny commented 5 years ago

@rbar2 Bug should be fixed in API28.

I've added a manual test for it. Didn't seem to fit with the other tests so I created a subpage named "special" for it. You can test it here:

https://thesunny.github.io/slate/#/composition/special

thesunny commented 5 years ago

@adjourn Can you verify that you can get Slate displaying (doesn't have to be working) on Android with an API 23? To clarify, this would not be the editor in my editor-events recorder but in Slate at:

https://www.slatejs.org/

adjourn commented 5 years ago

Both https://thesunny.github.io/slate/ and https://www.slatejs.org/ are rendering and all the regular actions that I can think of right now seem to work (splitting, cut, copy, paste, new block, etc).

Except I sometimes get Unable to find dom node <id>. This is of often because of forgetting to add props.attributes to custom component. when I press backspace in empty block, Im not sure why it happens (both editors).

thesunny commented 5 years ago

Fixed important but obscure bugs around periods. Added a test for this.

If you are following, please test edge cases as I'm hoping to wrap up API 26-28 support.

https://thesunny.github.io/slate/#/composition

As a note, there have been edge cases that appear to do with punctuation. It would be great to discover any more. Sometimes you need two of them. For example, It me. no. would trigger on the second . but not if there wasn't a first one.

Also, editor behaves differently if you click into a blank paragraph directly from editor start (you won't see capitalization in the keyboard in API 26/27) or if you click somewhere else first then click in the blank paragraph (you will see capitalization in the keyboard then).

So if you find a bug and can't immediately replicate it, that doesn't mean it wasn't a bug. I'd ask that you take the time to try to replicate it. If you can lock it down, there's a good chance I can fix it. It's possible I won't be returning to Android code for a while once we get a stable-ish release.

My next step will likely be to package what I have so far as a PR. Clean up the code, match Slate style guide, etc.

There are some big challenges with API 24/25 so I may do those afterwards. The major issue with them is that there is no way to tell from the event whether the user did an Enter or a Backspace. The challenge with 26-28 is that I could tell in some obscure ways through the events, but it was too late to cancel. But in 24/25 I just can't tell. So I have to inspect the DOM after the event and try to infer based on what changed in the DOM to determine that they hit Enter or Backspace.

Nantris commented 5 years ago

@thesunny I think you've fixed every issue I found except the Cut vs Delete All discrepancy and I'm in awe! I did discover one new issue though.

All reports are on API 28

Bug: Backspace after selection change

Seriousness: Critical Likelihood user will hit issue: Likely to be experienced by some users sometimes

I'm not sure exactly how to create this, but a key is focusing a different block and then coming back to where you had been typing and then backspacing. You can see towards the end of the video that when backspacing, a few character deletions before the editor crashes We becomes WeW.

Bug: Backspace on expanded selection

Seriousness: Minor Likelihood user will hit issue: High Backspace on expanded selections only removes the last character in the selection rather than the whole thing. The behavior is identical whether it's a single word or multiple paragraphs. Pasting over an expanded selection works properly so it's definitely related specifically to backspace.

Edit: I noticed this seems to behave differently in different scenarios. There may be more to this.

Reproductions:

Bug: Enter and Backspace in middle of words

Seriousness: Trivial Likelihood user will hit issue: Likely to be experienced by some users sometimes

Pressing enter and then backspace while the caret is in the middle of a word will duplicate portions of the word that follow the caret.

Bug: Unfinished compositions do not suggest words when selection changes then returns

Seriousness: Minor Likelihood user will hit issue: Likely to be experienced by some users sometimes

This is such a trivial bug it's almost more of a feature request. I'm guessing this isn't easily fixable and is probably not worth the time to fix, but for the sake of completeness.

Edit: It turns out this might be related to/might cause a slightly worse bug you can see here: https://imgur.com/a/ivWOL93

thesunny commented 5 years ago

@Slapbox Thank you for the reports. Would be really helpful if you could make an attempt at reproducing from a clean example with specific steps on the "Bug: Backspace after selection change".

It does take some work to isolate but without the steps, it's difficult to diagnose.

thesunny commented 5 years ago

@Slapbox For the "Enter and Backspace in middle of words" I've just tested this and can't replicate. One thing that might help is (a) a video showing this and (b) the specific device you are using which I'm sure you've mentioned earlier but since I can see you editing above maybe I can just ask. :)

Also, "Unfinished compositions" I'm not sure I get what you mean. I tried to do what I think you said but wasn't experiencing the issue. Perhaps specific steps and/or video?

Nantris commented 5 years ago

@thesunny thanks for being so on top of this issue!

Device is a Pixel 2 on Android 9.

Regarding unfinished compositions, I assume that inserting spacebar, enter, etc is required for a composition to go from compositionUpdate to compositionEnd. It seems like failure to press one of those keys after inserting some characters causes weird behavior when you unfocus a word and then return to is (no suggestions in the suggestion bar for that word/no underline for that word indicating that it's in composition.)

rbar2 commented 5 years ago

@thesunny https://thesunny.github.io/slate/#/composition/special works when I follow the instructions exactly, but if I put the cursor in the middle of "middle", then hit space, then hit delete, then hit space (again😅), then hit delete (again 😅)... I end up with "The mid word" (in other words, "dle" is deleted)

thesunny commented 5 years ago

Update: Fixed a few more bugs including enter/backspace. However, there is an off by 1 error but this error is an Android bug and happens in non-slate editors as well.

@Slapbox thank you for the bug testing and all the help on Slack diagnosing issues on your device.

There are a number of edge case bugs that present themselves inconsistently. In one case I was able to remove this inconsistency which happens when the cursor in a blank block sometimes ends up before and sometimes after a slate created non-breaking invisible unicode space.

In others, it appears that the logic in order to determine what the editor is doing is not able to complete its analysis before the next requestAnimationFrame and this causes unusual behavior. Furthermore, the length of the events that make up one action (e.g. some deletes or enters) comprise dozens of events.

For this reason, I will create another abstraction which I'm calling a Combo. One of the features of the combo is that whenever an event is fed to it, it resets the requestAnimationFrame and this usually gives us a bit more time to do the analysis (at least in initial testing). Since all the events fire rapidly together, by always extending the callback, I think we won't have reached the callback until we are truly done analyzing.

This should also simplify the increasingly complicated logic which is currently spread across the Android Plugin. For example, part of the enter handler is spread across many event handlers which expect things to happen in a specific way. With combos, we should be able to define them relatively simply in one place with a callback.

That said, we have some pretty good usability now and I'm pretty much done recommenting the existing code. I might put a PR out next week.

thesunny commented 5 years ago

Okay, put in a pull request for Android support.

https://github.com/ianstormtaylor/slate/pull/2553

There are bugs but I don't think any will crash the browser. Some edge cases will cause the wrong text to end up in the Editor but it can be fixed without crashing. This usually seem to occur around combinations of enter/backspace sometimes in conjunction with spaces and punctuation.

More details in the PR.

ianstormtaylor / slate