Autocorrect tooltips don't appear on iOS

[jwmcglynn]

Steps to Reproduce

When using a native iOS app without text prediction enabled, there is a pop-up that indicates when autocorrections are going to occur, and when backspacing on a correction a pop-up appears that gives alternative choices for the word.

In flutter, the autocorrections happen silently and require backspacing into the word to make the correction. See the gif below for a comparison of the behavior.

Flutter should either integrate with the iOS autocorrection system to provide the popups or emulate them. This is a pretty big usability concern which makes it more difficult to enter text because the corrections are harder to notice and harder to fix.

Logs

N/A, this is a design issue.

Flutter Doctor

$ flutter doctor
Building flutter tool...
[✓] Flutter (on Mac OS X 10.12.6 16G29, locale en-US, channel master)
    • Flutter at /Users/jwm/Projects/flutter
    • Framework revision 337150308c (2 days ago), 2017-11-05 19:39:49 -0800
    • Engine revision e059cc0258
    • Tools Dart version 1.25.0-dev.11.0

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).
      If Android SDK has been installed to a custom location, set $ANDROID_HOME to that location.

[✓] iOS toolchain - develop for iOS devices (Xcode 9.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 9.0.1, Build version 9A1004
    • ios-deploy 1.9.2
    • CocoaPods version 1.3.1

[✗] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).

[✓] IntelliJ IDEA Community Edition (version 2017.2.5)
    • Flutter plugin version 18.2
    • Dart plugin version 172.4343.25

[✓] Connected devices
    • iPhone 7 Plus • DF665BBF-F7D6-4B75-8FCA-E092AD4AD6DC • ios • iOS 11.0 (simulator)

[eseidelGoogle]

MD doesn't seem to talk about this, so this is likely going to be cupertino specific. https://material.io/guidelines/components/text-fields.html Thoughts @cbracken @xster?

[lukepighetti]

@eseidelGoogle iOS doesn't have autocorrect on the keyboard "modal" like Android does, so this is not MD vs Cupertino, but more like we just need it in all apps on iOS otherwise text input is very difficult.

[xster]

Thanks for the feedback. We're addressing a few non-cupertino iOS text entry issues at the moment like https://github.com/flutter/flutter/issues/9507, https://github.com/flutter/flutter/issues/20693 and https://github.com/flutter/flutter/issues/9288. Hopefully we'll have the bandwidth to address this one too after.

[lukepighetti]

You know what's kind of embarrassing? I was using an iPhone 5c today and autocorrect was coming up on the keyboard. Literally never seen or noticed it before. So my apologies for my incorrect statement earlier. I'm not sure when that changed or when it displays. This was on iOS 10 btw.

[ethras]

What's the status on this?

[zoechi]

@Ethras Using "add reaction" on the initial comment would increase priority while +1 comments are rather pointless and cause lots of people to get notified for no reason. To get notified yourself use the [Subscribe] button to the top right instead.

[jslavitz]

This is currently in progress!

[jakub-bacic]

Any ETA regarding the fix?

[xster]

There's a solution in work but our work allotment was reshuffled recently. I would realistically expect this to be done closer to June.

[LongCatIsLooong]

According to UIKit documentation, some of the behaviors described above come from spellchecking rather than autocorrection (Although when I tested on iOS 12 & 13 if autocorrection is on turning off spellchecking only hides the red underline).

Created #34688.

[dvxf]

Please note that the popup shown in the original post is only shown if "Predictive" is turned to OFF in the General->Keyboard settings. By turning off this setting, you don't get 3 auto-correct choices at the top of the keyboard, but just one right next to the word you're typing in, exactly as shown in the original post, and a cross to dismiss the correction.

It was the default behaviour of iOS until the top-of-the-keyboard thing was included, so personally I keep this setting off on my phone as I'm used to it. My guess it that the percentage of users with this setting off is small as it's not the default behaviour anymore, but I'm surely not alone.

Unfortunately, Flutter textfields are almost unusable in this configuration, as you have no idea what word will be replaced by the auto-correct.

[lukepighetti]

Thanks for the explanation @dvxf, could never figure out why others had the top-of-the-keyboard thing and I don't. Flutter should be designed to work without the top-of-the-keyboard thing without a doubt.

[LongCatIsLooong]

Autocorrected words are now highlighted on iOS. Closing.

[jakub-bacic]

@dvxf Just a little comment from my side - it's also a default behavior when... there's no prediction implemented for the given language (e.g. Polish).

Anyway, I'm glad it's fixed now 🎉

[dvxf]

@LongCatIsLooong Thank you, it's definitely an improvement that autocorrected words are now highlighted on iOS! However, it only fixes half of the issue: as far as I can tell, the autocorrect tooltip still doesn't appear on iOS.

So it improves the behaviour when Predictive Keyboard is turned on, but not when Predictive Keyboard is turned off (or in Polish as @jakub-bacic mentioned).

As I mentioned before, unfortunately Flutter textfields are almost unusable in this configuration, as you have no idea what word will be put by the autocorrect in place of the highlighted word.

[lukepighetti]

@LongCatIsLooong it appears that there was a misunderstanding about this issue. This issue is about tooltips, not squiggly line highlighting. Requesting reopening of this issue.

[lukepighetti]

@dvxf do you happen to know if there is any way to enable the Predictive Keyboard in one app? We could shim this issue with that for the time being if it's possible to tell iOS to show Predictive Keyboard when using a Flutter app.

I'm trying to imagine a scenario where Flutter could actually reproduce the case above and I don't see how it could be possible to overlay on top of the system keyboard in a Flutter view.

[LongCatIsLooong]

Oh my apologies. I didn't realize the tooltip can still appear in some cases, reopening. It seems the tooltip will be difficult to implement at the moment: we can't draw the tooltip in flutter, and currently there doesn't seem to be a good way to return the accurate CGRect in func firstRect(for range: UITextRange) -> CGRect in the text input plugin.

Related: https://github.com/flutter/flutter/issues/9344

[lukepighetti]

Even if we relocated it to be above the text input instead of below, that would be fine.

[xster]

@LongCatIsLooong is the remaining issue still a customer critical?

[LongCatIsLooong]

@xster I would think it's still blocking at least for apps that target Polish users?

@lukepighetti I think anchoring the tooltip to the text input's bounds would confuse the users, especially when the text field is multiline.

[lukepighetti]

I'm confused by the lack of priority on this issue. I am an English iOS user and I have no controls for autocorrect in any of my Flutter apps on iOS, and I never have. I build all my apps on dev channel.

[Hixie]

I recommend filing separate bugs for the remaining work so that we can track each specific item separately. Having one issue track a large number of separate work items doesn't lend itself well to clear prioritisation.

For the new bugs, I recommend marking them customer critical, etc, as per: https://github.com/flutter/flutter/wiki/Issue-hygiene#prioritization

[Hixie]

(Please list the new bugs filed here, so that people who are following this one can follow up with the relevant bugs.)

[lukepighetti]

This issue tracks the remaining work precisely, it was just misunderstood.

[Hixie]

Got it, thanks. Sorry, I misread the bug earlier.

[Hixie]

@LongCatIsLooong Any updates?

[LongCatIsLooong]

@Hixie Currently working on 55613. I will start looking into this more closely after that.

[LongCatIsLooong]

I spoke with @GaryQian and @HansMuller about this.

Proposed solutions so far generally fall into these 2 categories:

• Blocking execution

Delegate the func firstRect(for range: UITextRange) -> CGRect computation to the framework using platform channels, block the platform thread until the result comes back. See https://github.com/flutter/engine/pull/4358 for potential problems with blocking the platform thread.

• Asynchronous prefetching

Keep a copy of Paragraph data (or whatever information it takes) in the iOS text input plugin, and keep it up to date with the framework using platform channels. Implement func firstRect(for range: UITextRange) -> CGRect using the engine copy of information.

This introduces a race condition: when text is entered via the software keyboard, the iOS text input plugin sends the update back to the framework (which in turn triggers an update in Paragraph data that needs to be synced back to the engine side), and at the same time the keyboard calls func firstRect(for range: UITextRange) -> CGRect. There's no guarantee the updated Paragraph data will arrive in the engine before func firstRect(for range: UITextRange) -> CGRect returns. And there's no way to tell iOS to update the rect should the Paragraph data arrive late.

[matehat]

@LongCatIsLooong I don't know if your comments was meant to indicate that this issue was blocked or reconsidered, but I wanted to add some weight to this.

We're building a secure messaging app for healthcare, and 50% of the user experience is around typing. Currently on iOS, unless you set your iPhone to have the 3 autosuggestions above the keyboard, the typing experience is simply broken. Some replacements that are usually shown using the white tooltip before being applied do not appear, creating confusion and straight inability to prevent said replacement.

Below, on the left is a reference working example, using Notion (probably react-native or something like this), and on the right, our app using Flutter.

I want to emphasize this: the experience is currently broken. I cannot tell our users to go in the settings and change their keyboard specifically for our app. It is high priority, not a minor visual glitch.

[lukepighetti]

What's especially alarming to me is the lack of priority on the team side and the lack of severity on the community side. We have started recommending React Native for any heavily text-editing based apps that are going on the iOS App Store because of this broken experience.

[matehat]

@lukepighetti that's understandable, and unfortunate since Flutter is such a great framework.

On our end, being a bootstrapped startup with limited resources we simply can't afford to switch to another tech at this point. We made the leap and invested heavily in Flutter. I'm just really hoping the team fixes it soon, since our app really looks clumsy and buggy because of our tech choices. We're pitching to big players in healthcare that they should rely on us for their critical internal processes, but buggy behavior like these do not help our case.

/cc @Hixie @LongCatIsLooong please, please increase the bug priority. It's a pretty big deal.

[lukepighetti]

That's a very difficult position to be in. Maybe someone from the flutter team can give us a high level strategy for implementing it. I don't have the bandwidth to work on this feature ATM but if they already have an idea of how to implement then someone may have the time to achieve it.

I wish there was a way to turn on the autocorrect section on the keyboard when in a Flutter app. That would be a totally acceptable shim for the time being IMHO. Maybe there is a way? I have no idea.

[LongCatIsLooong]

@matehat the gif on the right confuses me a bit, at least the background highlight should have shown up when the text was autocorrected. (I suppose you're using the latest stable or something newer) Could you share your iOS keyboard configuration used in that clip?

For autocorrect as previously mentioned in func firstRect(for range: UITextRange) -> CGRect a correct CGRect needs to be returned by that method for iOS to correctly position the text background highlight and the tooltip. Currently we don't have an easy way to get the correct CGRect synchronously, so we have 2 options:

1. Return CGRectZero to prevent iOS from drawing the highlight and the tooltip, and send the UITextRange provided in the call to the framework to let Flutter draw the highlight. This is the current approach, it should work well for the highlight but we lose the tooltip.

2. Try our best to predict the correct Rect and let iOS draw the highlight & the tooltip. I made a POC that seems to work well enough for #9344, but autocorrect is a bit different:

   • For positioning the IME tooltip(#9344), we just need the bottom left point (or the bottom right point). For autocorrect the entire Rect needs to be on point, so the background highlight iOS shows indicates the correct range of the text that will be autocorrected.
   • For positioning the IME tooltip(#9344), iOS marks the text before calling func firstRect(for range: UITextRange) -> CGRect, so we have some idea about what the range is going to be before func firstRect(for range: UITextRange) -> CGRect gets called. iOS doesn't mark the text that needs to be autocorrected before calling func firstRect(for range: UITextRange) -> CGRect, so we have to entirely rely on the range parameter, which we don't have access to before func firstRect(for range: UITextRange) -> CGRect (at which point it's already too late to request the rect information from the framework). Using the current word works in some cases, but as shown in @matehat's gif clip on the left, iOS may autocorrect multiple words.

It seems the same Rect returned is used for both the highlight and the tooltip, and returning a Rect that has 0 width (or height) would also prevent both the tooltip and the highlight from showing up, so we can't just tell iOS to hide the highlight and show the tooltip.

I guess turning off autocorrect in TextField could be a workaround for the time being, if you would rather not have autocorrect on the text field if there's no tooltip.

[matehat]

@LongCatIsLooong I'm using the latest stable:

Flutter 1.17.5 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 8af6b2f038 (2 weeks ago) • 2020-06-30 12:53:55 -0700
Engine • revision ee76268252
Tools • Dart 2.8.4

Here are the exact keyboard settings:

The whole puzzle is also because we're trying to do all of this with asynchronous callbacks between the host platform and the framework if I understand correctly, right?

[LongCatIsLooong]

The whole puzzle is also because we're trying to do all of this with asynchronous callbacks between the host platform and the framework if I understand correctly, right?

Yes, we'd like to avoid doing synchronous communication: https://flutter.dev/go/flutter-platform-idling.

I noticed in the second gif nothing got highlighted (the correction rectangle didn't appear) when autocorrect took place. That looks like a different bug that needs fixing? In my test flutter app I can make the text highlight show up with the same keyboard configuration, by following these steps:

1. turn predictive off
2. in the flutter app, switch keyboard to French
3. type "ca va bien" ("ca" is highlighted, and if I continue to type it's autocorrected)

[lukepighetti]

What do you mean by highlighted? The squiggly line below text?

[LongCatIsLooong]

@lukepighetti No I was referring to the surrounding light gray correction rectangle. Maybe there's a way to change its color to transparent to make it invisible, so we don't have to worry about having the precise rectangle.

[lukepighetti]

To the best of my knowledge I have never seen the light gray correction rectangle in a Flutter app on iOS. Can you show us a screenshot of what you're talking about on Flutter?

[LongCatIsLooong]

Sorry I forgot it's light blue by default:

[lukepighetti]

Gotcha. Yeah, this issue is about the autocorrect suggestions that appear next to the iOS cursor while you're typing regardless of the styling of the app.

Currently it works as expected if you have Predictive setting turned on in General => Keyboards.

However, if that's turned off these tooltips do not appear when typing.

Taken from iPhone 11 Simulator, iOS 13.6

There are two solutions that I can think of.

1) Implement the autocorrect tooltips for when Predictive is off.

or

2) Find a way to force Predictive setting whenever in a Flutter app.

[matehat]

It doesn't seem like option 2 is possible if I look at all the documentation and questions around predictive UI.

I think showing the tooltip is more important than showing the exact highlighted region, since the tooltip gives you the ability to accept or refuse a correction.

Is there a way to show the tooltip and position it approximately right?

[matehat]

Sorry I just saw this part of @LongCatIsLooong comment:

It seems the same Rect returned is used for both the highlight and the tooltip, and returning a Rect that has 0 width (or height) would also prevent both the tooltip and the highlight from showing up, so we can't just tell iOS to hide the highlight and show the tooltip.

Maybe a minimal but non-zero width, and a height deducted from the text's known line height would do?

[LongCatIsLooong]

I found an approach that seems to work well for https://github.com/flutter/flutter/issues/9344, save for a few edge cases. I'll see if that works for the autocorrect tooltip too. The highlighted region must accurately indicate the range of the autocorrect otherwise it can be misleading. Maybe we can override addSubview to prevent the highlight from being added.

Update: looked into this again recently. The autocorrect highlight view is added to a different window. There doesn't seem to be a good way to prevent the highlight from being added, or make the highlight perfectly invisible.

[lukepighetti]

@matehat did you end up finding any immediate relief? I just had a UX testing session with someone entering tags and they were having a really hard time getting by without tooltips. They noticed it within 3 seconds of typing and it was a serious usability issue.

Do we know if there's a way to at least detect if predictive keyboard is enabled? We could then show them a message suggesting they turn it on.

What I've noticed is that people who are long time iPhone users have this feature disabled by default and don't even know it exists. If you're using an iPhone on an account that isn't many years old then you might not realize that iOS had this tooltip feature. It seems that when this feature was added it was disabled by default and saved to iCloud settings. But new accounts have it enabled by default.

[lukepighetti]

There's another option. If we can get autocorrect options programmatically we can turn off autocorrect on the field and display our own predictive text options in Flutter. That would also allow us to add in our own actions which is a value add over the native controls (afaik).

There's also this method for detecting if Predictive is enabled. https://stackoverflow.com/questions/25993949/check-if-predictive-text-is-enabled

[henryleunghk]

I encountered the same situation like you guys. I have been searching for a period of time looking for a solution to have same text input experience as other iOS apps built with either native Swift/Objective-C/React-Native. Finally, I resorted to writing my own package to export UITextView from iOS for use in Flutter: https://pub.dev/packages/flutter_native_text_input

I hope you guys find it useful. Please feel free to report any issues or create PRs for any missing features. Cheers🍻

[joniaiuser]

Any chance for the iOS experience to be incorporated into Flutter natively?

[joniaiuser]

@LongCatIsLooong Any updates for this?

[FloLecoeuche]

@LongCatIsLooong Can you add this feature to the Flutter roadmap ?