Using Flutter In An iMessage Extension shows an empty view

[kellyiscute]

I was recently trying to create an imessage extension embeded into my app. I tried using FlutterEngine to show a FlutterViewController. However it shows an empty screen and there appears two duplicate warnings Creating MTLCommandBuffer while in background. but when I tried creating a same thing in native code, it shows no warning.
So is there any possibilities that Flutter would work in an imessage extension?

[huycozy]

Hi @guo40020, thanks for filing the issue. Please provide a completed and minimal reproducible code sample so that we may verify this. Also, please provide the output of flutter doctor -v as well. Thank you!

Refer this issue to the related/meta issue: #16092

[kellyiscute]

Thanks for reply.

code sample is in the attachment here

flutter doctor -v

[✓] Flutter (Channel master, 3.1.0-0.0.pre.1757, on macOS 12.5 21G72 darwin-x64, locale zh-Hans-CN)
    • Flutter version 3.1.0-0.0.pre.1757 on channel master at /usr/local/Caskroom/flutter/3.0.5/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 922eea8763 (4 weeks ago), 2022-07-20 01:51:04 -0700
    • Engine revision 1340e57ffb
    • Dart version 2.19.0 (build 2.19.0-13.0.dev)
    • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at /Users/kelly/Library/Android/sdk
    • Platform android-31, build-tools 33.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 13F100
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Ultimate Edition (version 2021.3)
    • IntelliJ at /Users/kelly/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] IntelliJ IDEA Ultimate Edition (version 2021.3)
    • IntelliJ at /Users/kelly/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/213.5744.223/IntelliJ IDEA.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.67.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.44.0

[✓] Proxy Configuration
    • HTTP_PROXY is set
    • NO_PROXY is 127.0.0.1,localhost,::1
    • NO_PROXY contains localhost
    • NO_PROXY contains 127.0.0.1
    • NO_PROXY contains ::1

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 12.5 21G72 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 104.0.5112.79

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[huycozy]

Hi @guo40020, thanks for providing the sample code. But it looks like the project is missing ios/Runner.xcworkspace. Could you please provide it? Also, could you please provide the steps to reproduce this issue? (A such sample is the repository in the issue #96165, see README.md file)

[kellyiscute]

Here is the sample code with previously missing file

Steps To Reproduce

1. Select iMsg target and run
2. Wait until it loads, and there would be a button says show flutter.
3. Click on it, and two warning will appear in xcode output panel
4. An empty transparent screen shows up in the app. If you tries to drag it down, it will be dismissed just like normal views

[danagbemava-nc]

Hi @guo40020, sorry for the delay.

I'm not entirely sure that there is support for this at the moment. I do see the blank screen and the warnings in my xcode log. I'll label this so someone with more knowledge than myself on this can take a look.

Code sample can be found in https://github.com/flutter/flutter/issues/109530#issuecomment-1217429241

xcode log

``` 2022-08-19 06:37:35.829420+0000 iMsg[81340:5446145] Successfully load keyboard extensions 2022-08-19 06:37:35.922555+0000 iMsg[81340:5446145] [PPT] Error creating the CFMessagePort needed to communicate with PPT. 2022-08-19 06:37:39.682853+0000 iMsg[81340:5446145] GrMtlCommandBuffer: WARNING: Creating MTLCommandBuffer while in background. 2022-08-19 06:37:39.684499+0000 iMsg[81340:5446145] GrMtlCommandBuffer: WARNING: Creating MTLCommandBuffer while in background. ```

recording

https://user-images.githubusercontent.com/88313112/185559211-d3ed8fde-4461-42fb-a298-bd907f5faadc.MP4

flutter doctor -v

``` [✓] Flutter (Channel stable, 3.0.5, on macOS 12.5 21G72 darwin-arm, locale en-GB) • Flutter version 3.0.5 at /Users/nexus/dev/sdks/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision f1875d570e (5 weeks ago), 2022-07-13 11:24:16 -0700 • Engine revision e85ea0e79c • Dart version 2.17.6 • DevTools version 2.12.2 [✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0) • Android SDK at /Users/nexus/Library/Android/sdk • Platform android-33, build-tools 33.0.0 • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 13.4.1) • Xcode at /Applications/Xcode.app/Contents/Developer • CocoaPods version 1.11.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2021.2) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) [✓] VS Code (version 1.70.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.46.0 [✓] Connected device (5 available) • M2007J20CG (mobile) • 5dd3be00 • android-arm64 • Android 11 (API 30) • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64 • Android 13 (API 33) (emulator) • Nexus (mobile) • 00008020-001875E83A38002E • ios • iOS 15.6 19G71 • macOS (desktop) • macos • darwin-arm64 • macOS 12.5 21G72 darwin-arm • Chrome (web) • chrome • web-javascript • Google Chrome 104.0.5112.79 [✓] HTTP Host Availability • All required HTTP hosts are available • No issues found! ``` ``` [✓] Flutter (Channel master, 3.1.0-0.0.pre.2346, on macOS 12.5 21G72 darwin-arm, locale en-GB) • Flutter version 3.1.0-0.0.pre.2346 on channel master at /Users/nexus/dev/sdks/flutters • Upstream repository https://github.com/flutter/flutter.git • Framework revision adda088bcc (4 hours ago), 2022-08-17 22:59:07 -0400 • Engine revision a77ead45a8 • Dart version 2.19.0 (build 2.19.0-108.0.dev) • DevTools version 2.16.0 [✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0) • Android SDK at /Users/nexus/Library/Android/sdk • Platform android-33, build-tools 33.0.0 • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 13.4.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 13F100 • CocoaPods version 1.11.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2021.2) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) [✓] VS Code (version 1.70.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.46.0 [✓] Connected device (5 available) • M2007J20CG (mobile) • 5dd3be00 • android-arm64 • Android 11 (API 30) • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64 • Android 13 (API 33) (emulator) • Nexus (mobile) • 00008020-001875E83A38002E • ios • iOS 15.6 19G71 • macOS (desktop) • macos • darwin-arm64 • macOS 12.5 21G72 darwin-arm • Chrome (web) • chrome • web-javascript • Google Chrome 104.0.5112.79 [✓] HTTP Host Availability • All required HTTP hosts are available • No issues found! ```

[jmagman]

There is a FlutterView showing, it's just empty.

That Creating MTLCommandBuffer while in background warning is probably because Skia is checking ([UIApplication sharedApplication].applicationState == UIApplicationStateBackground), but sharedApplication is forbidden in an app extension https://developer.apple.com/library/archive/documentation/General/Conceptual/ExtensibilityPG/ExtensionOverview.html#//apple_ref/doc/uid/TP40014214-CH2-SW6

I'm not sure if the empty window behavior is because of Skia, or if something else in the engine that's making a similar assumption about not running in an extension.

The Flutter.framework isn't marked as safe for running in extensions, but I'm not sure if it actually would be if APPLICATION_EXTENSION_API_ONLY was set, or if it's fundamentally linking on disallowed APIs.

warning build: Linking against a dylib which is not safe for use in application extensions: Build/Products/Debug-iphonesimulator/Flutter.framework/Flutter

Running Flutter in an extension isn't supported as of right now.

[jmagman]

This also reproduced with impeller.

Using the Impeller rendering backend.

[chinmaygarde]

This isn't a supported use case at the moment. Though if we can create a Metal layer, we ought to be able to support it.

[tomduncalf]

@chinmaygarde I wonder if you might have any pointers to what you think might be causing Flutter to not work in extensions right now, or where one might start to look?

I'm interested in trying to use Flutter for an AUV3 extension as described in e.g. https://github.com/flutter/flutter/issues/16092 and am willing to do some digging to try to make it work.

No worries if you have no ideas, but any pointers to likely places in the engine code or whatever would be useful!

[tomduncalf]

@chinmaygarde Never mind, I think I have this figured out! https://github.com/flutter/flutter/issues/16092#issuecomment-1288192913

[jtkeyva]

@chinmaygarde @jmagman @guo40020

any luck or any news on this? imessage support would be awesome. thanks