Investigate Web/JS-only support

[nmfisher]

Web support is still proving to be a PITA for the following reasons:

• lack of FFI support in both WASM/JS
• Flutter WASM support still not stable
• relatively large download sizes
• general WASM threading issues

A light(er)weight, Dart-only (?) version for Web would go some way to addressing these deficiencies.

First step would be to separate the existing components into Flutter- and Dart- specific packages, the former containing the widgets and platform channel code, the latter containing the controller, bindings, native code and libs.

Then we'd need to either: 1) Write a separate FilamentController for JS that glues to the native Filament bindings 2) See if we can work around the FFI issues - I think these were mostly (?) caused by the absence of shared memory so if we can compile to a single WASM module, this might be addressable.

The end-goal is to have a lightweight (< 2mb) library that can: 1) render models with some kind of ubershader (even if it's a stripped-down version) 2) render animations 3) hook into some browser UI components

[PetterGs]

This is a fantastic project. I'm curious about the FFI difficulties, I haven't looked over the code yet but can the WASM interface be improved by this repo? Sorry if this is an ignorant comment, and I have overlooked something.

https://github.com/DeMille/wasm-ffi

[nmfisher]

Thanks for your comments, I genuinely appreciate it. I've gone into a bit more detail below, partially to help clarify some things in my head and partially to document things for anyone else who wishes to contribute in future.

As a quick overview of how the package is structured, there's three main components:

1) FilamentController is the Dart class that exposes the main API for users 2) When you call createViewer on FilamentController requests the platform channel to set up a rendering context/surface 3) FilamentController invokes certain methods via FFI (FlutterFilamentFFI) to spawn a "render" thread and create a Filament engine. A Filament engine requires its own thread, so most tasks (loading entities, adding lights, etc) will be dispatched to a task runner that runs them on that "render" thread.

Filament itself is precompiled and stored in this repository via Git LFS, and, on desktop or mobile, the C++ code in this package (FlutterFilamentFFI etc) are compiled via Flutter's CMake integration (on Android/Windows) or Xcode (MacOS/iOS). On the experimental web implementation, I was compiling via a custom CMake script that used emscripten under the hood to compile to WASM with some accompanying JS glue code.

I chose this structure because (a) it requires the absolute minimum platform-specific code, and (b) doesn't block the main Flutter UI thread.

With that in mind, there's probably two big issues with a full/stable WASM implementation right now. Keep in mind this was my first time working with WASM/emscripten, so there may be simpler solutions that I'm not aware of.

One is concurrency. Flutter web apps run on the main browser "thread", so if you want your Flutter app to wait for some work to complete on a pthread, you either need to block and wait, or use some emscripten magic to unwind the stack and return to the main thread.

When I did the original web implementation, I was waiting for std::future to complete in FlutterFilamentFFI from the task runner - this was fine on mobile/desktop, but would always block the main thread (and thus regularly cause the whole browser) to hang. However, I've since changed this to be asynchronous on the C++ side and used a Future on the Dart instead. I haven't revisited the web branch since I made that change, but now that I think about it, that may help solve the issue.

The other issue is about shared memory. Quick example, loading an asset from a path like asset://suzanne.glb goes like this: 1) When you create a viewer via FilamentController, in addition to the Metal context/render target/etc, we also request the platform channel to provide a function pointer that is used to loading binary assets. This
2) FilamentController converts the Dart string to an FFI-compatible char* and passes to FlutterFilamentFFI 3) FlutterFilamentFFI submits a task to the render thread 4) FilamentViewer (or SceneManager, depending on which branch you're on) invokes the function pointer with the path you provided and does what it needs to do with the returned binary resource.

That works well on desktop/mobile, where the Flutter app is linked directly with our C++ code (and Filament), so everything is working in the same memory space. If I allocate a pointer in Flutter with FFI, I can freely pass that around to any of my C++ code (and on any thread) and the pointer will point to the same location in memory.

There were some major issues with WASM though.

First, Dart's FFI implementation still has some bugs/missing features (e.g. see here, but at least I could work around that). More importantly, Dart didn't support allocating memory directly on WASM, so I couldn't do something like final ptr = calloc<Float>(1);. It would throw an exception due to not being supported. I needed to write a custom allocator that called into C++ to allocate the memory, then I could only write to it by calling another function and passing values via the stack (i.e. one byte at a time), which as you can imagine was interminably slow.

Additionally, Flutter's experimental WASM support didn't allow linking directly with an emscripten-compiled C++ library, nor could I ever successfully instantiate both modules with SharedMemory. Even if I could allocate on the Dart side, there was no way to pass the pointers around properly (either function pointers or data pointers).

However, this will be ironed out eventually - it seems like there's a lot of support for WASM across both the Dart and the Flutter teams. It's been a few months since I checked, so some of these may have even been addressed already. Flutter WASM requires WasmGC which is pretty bleeding-edge anyway, so I figured I'd just wait for the Flutter/Dart team to fill in the gaps around FFI. The package you linked probably won't help that much unless I wanted to write an additional JS-only FilamentController, which is something I really wanted to avoid (see my earlier comment about having as little platform-specific code as possible).

However, what would be possible right now is to separate the repository into two - a Dart-only package on one hand, and a Flutter-only package on the other. The Flutter package (which would depend on the Dart package) wouldn't work until the issues above are addressed, but the Dart package could be used standalone to build a web app with the same underlying 3D API. Any UI elements would then just be regular HTML/JS on top of the target canvas.

I actually like this idea, and it's not a huge amount of work (probably a day or so). Unfortunately, right now I need to stick to paying work, so unless someone was willing to sponsor it, it's probably not going to happen any time soon.

I'm very much open to feedback/thoughts/etc, though, so please feel free to comment.

[PetterGs]

Thank you very much for the rundown. I'm unfamiliar with both the inner workings of Flutter and Filament at the moment. Just searching for the easiest way to create an iOS and Android 3D cross platform instant app. Not possible to use something like Unity for that.

[nmfisher]

Out of interest, what type of 3D app were you looking to create? I've actually added some basic game engine components to let me make a simple 3D game for the Flutter game competition, which is currently in the top 20 submissions:

https://devpost.com/software/escape-from-heat-island

P.S. a vote for the game project would really help get more exposure for this repository, and might help get some more resources to add further functionality!

[PetterGs]

I would love to create a geospatial ARCore app for both Android and iOS, without resorting to Unity. It's a bit of a tall order right now mostly due to Apple. Actually writing a Java OpenGL App for Android is a breeze. The problem is making a multiplatform App from one codebase.

Great project. Voted naturally. Looks great.

[nmfisher]

Thanks for that, much appreciated!

I'm not sure how whether this repository would be a good fit for a cross-platform ARCore app (I've never worked with it before so I'm not exactly sure how much ARCore does itself, and how much it defers to external rendering libraries). I'll open an issue to remind myself to dig into a bit more detail.

[PetterGs]

Sure it's just about passing matrices and buffers to a rendering library, the ARCore library itself doesn't do any rendering. The problem is iOS. You have to use objective-c, or swift, to use it. Which is a pain.

[nmfisher]

Quick update, I've started work to decouple the Flutter elements from the the Dart/FFI elements. This means there will be two packages: 1) a Dart package encapsulating all Filament libraries and rendering/scene logic 2) a Flutter package for widgets and platform channels

I've been using native-assets for building/linking the Dart package; while this is still considered experimental, I've been using it across a number of other packages and it seems quite stable to me.

This will make it easier to bundle a non-Flutter version for web/WASM (one of the ideas mentioned when I first created this issue).

I also took the opportunity to clean up a lot of the code, and this decoupling will have quite a few other benefits:

1) testability - I can now write tests in Dart that render directly into a texture, rather than dealing with Flutter tests which are mostly designed to handle testing UI events. 2) scripting - it's now easier to render PNGs directly to filesystem via a script 3) multiple Filament viewport widgets (https://github.com/nmfisher/flutter_filament/issues/18) (I haven't actually gone ahead with this, though, because I need to decide how to handle Windows where we don't actually render into a texture, so there's no easy way to handle multiple viewports). 4) embedding - the current job system/background render thread isn't ideal for performance (the only reason I do this is because Flutter is running on the main thread and we don't want to block the UI when processing intensive tasks). It would be great to manage our own main thread that itself embeds Dart and/or Flutter. 5) easier build process (we don't need to worry about podspecs or syncing MacOS/iOS code any more).