'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!'

[0xchase]

Background

I am attempting to extend my flutter application to load VST plugins and I'm unsure if there's a workaround for this exception on MacOS.

In case the reader is unaware, it is common for music production software to load plugins (distributed as a dll/so/dylib) that perform various kinds of audio processing (like emulating a synthesizer), into the address space of your host software. These plugins will process audio passed to them by the host software, and can render their own GUIs. One common plugin specification is the VST format, which I'm attempting to load into my application.

Steps to Reproduce

A loader for the VST format has already been implemented in the JUCE framework, so I wrote a small C++ wrapper around JUCE, that builds as a dynamic library, and exports a few functions like ffi_create_plugin, ffi_plugin_process_audio and ffi_plugin_show_gui. My flutter application can then load this dynamic library, and call the exported functions, which will in turn load a selected VST plugin.

I can successfully load a headless VST plugin this way. The problem is when I call ffi_plugin_show_gui, (when a user presses a button). At this point the application crashes with an unhandled exception: 'NSInternalInconsistencyException', reason: 'NSWindow drag regions should only be invalidated on the Main Thread!'.

The Cause

UIKit is not thread safe, so all calls to it must be made by the main thread. Most VST plugins will use UIKit. So the exception is caused by triggering the GUI creation on a non-main thread. Therefore, I need a way in flutter to call the export that creates the GUI from the main thread, instead of whatever thread it is currently being called from.

Potential Solutions

• Create the GUI from the main thread: I don't have a great understanding of Flutter's threading model, but perhaps there's a way to call ffi_plugin_show_gui from the main thread. I've dug through the docs and have not found a documented way to do this.
• Disable Main Thread Checker: The apple developer docs show it is possible to disable the check that causes this exception. Perhaps because each window will be be calling UIKit on a single thread, it doesn't matter that it isn't called on the main thread. I'm not sure if doing this would be safe, and I'm not sure how to configure this in flutter.
• Something else: Perhaps someone who understands Flutter better than me can propose a better solution.

I'm worried that Flutter's architecture will make this impossible, which would pose a significant problem for my project.

Thanks in advance for any assistance or advice.

[exaby73]

Hello @0xchase. Thank for you filing this issue. Could you provide a complete and minimal, reproducible example if possible?

[dcharkes]

• Create the GUI from the main thread: I don't have a great understanding of Flutter's threading model, but perhaps there's a way to call ffi_plugin_show_gui from the main thread. I've dug through the docs and have not found a documented way to do this.

Flutter's UI thread (the "main Dart thread") not being the system thread is a design decision from flutter and a known issue with interop.

Another issue that also needs to be solved besides being able to run something on the system thread is thread-pinning in general, so that subsequent FFI calls are called on the same thread. https://github.com/dart-lang/sdk/issues/46943

The third threading issue is async callbacks (because Dart isolates can only have a single mutator thread). https://github.com/dart-lang/sdk/issues/37022 (Not sure if you would run into this as well, but listing it here for completeness sake.)

@mkustermann is looking into the threading issues with dart:ffi on Flutter and Dart standalone.

[0xchase]

Hello @0xchase. Thank for you filing this issue. Could you provide a complete and minimal, reproducible example if possible?

I think this request might misunderstand the nature of this issue. This problem is caused by intentional design decisions in the flutter architecture, rather than a bug. I can still create a code sample if you or someone else wants it for testing, but as best I can tell solving this issue requires changing the flutter threading model or exposing new APIs to run code on the main thread (assuming this doesn't already exist), rather than a simple bug fix.

[exaby73]

@0xchase I have found a similar issue below. Before I label this, could you confirm if the below mentioned request is the same as yours or related to yours?

• https://github.com/flutter/flutter/issues/104144

[0xchase]

@exaby73 These issues are vaguely related because they both deal with VSTs, but the technical problems are different. The linked issue involves building a VST with flutter, while this one is about hosting a VST in a flutter application. The former requires changes to the flutter build system, while this one requires changes to the flutter architecture.

[exaby73]

@0xchase Thanks for the clarification. Labelling the issue

[gspencergoog]

cc @gaaclarke @stuartmorgan

[gaaclarke]

I'd use platform channels instead of dart:ffi. They have built into them already the threadhop that will place you on the main thread. https://docs.flutter.dev/development/platform-integration/platform-channels

Good luck, sounds cool.

[stuartmorgan]

• perhaps there's a way to call ffi_plugin_show_gui from the main thread.

This is definitely not possible.

• Disable Main Thread Checker: The apple developer docs show it is possible to disable the check that causes this exception.

That's a debug option that is unrelated to actual production UIKit code that does thread assertions, which is what you are likely hitting.

Perhaps because each window will be be calling UIKit on a single thread, it doesn't matter that it isn't called on the main thread.

That is not how UIKit works.

• Perhaps someone who understands Flutter better than me can propose a better solution.

I'm worried that Flutter's architecture will make this impossible, which would pose a significant problem for my project.

It's unclear to me that this needs a new Flutter-level solution at all.

• As pointed out above, platform channels already provide the threading model you want.
• If you really want to use FFI, since you are already writing your own wrapper anyway you can just use GCD to jump to the main thread in your wrapper before calling the relevant API (as suggested here).

[liamappelbe]

@0xchase I'm investigating use cases for Platform Isolates. What does the API surface of your VST plugin look like? There are 3 broad categories I'm interested in.

1. APIs that don't use callbacks. Just call an API method and get a result. You never have to pass a function pointer to an API method.
2. APIs that only use callbacks to send events. You do pass callbacks to the API, but they're only used to notify the caller about some event. They all return void, and the API doesn't care if the callback is executed synchronously or asynchronously.
3. APIs that use callbacks to interact synchronously with the caller. You pass callbacks to the API that are expected to execute synchronously. They might return a result directly, or they might be expected to do some other action synchronously (eg fill an audio buffer).

[0xchase]

@liamappelbe Thanks for the ping. With respect to this issue, the API calls in question are primarily used for triggering the creation of a UI window created and drawn by the VST plugin. So for this use case callbacks aren't needed.

In my application, all the audio is handled outside of the main thread.

[flutter-triage-bot[bot]]

The triaged-desktop label is irrelevant if there is no team-desktop label or fyi-desktop label.