Closed CicadaCinema closed 1 year ago
Hi! Thanks for brining this up. I probably should've clarified that part in the guide. There are two things to note here:
To run Dart-only tests, always run cargo build -r
beforehand. This is your issue here; you are missing the binaries locally. If you look in my Dart unit test source code, you will see that it references the binaries in the /target/release
folder (or something like that), which doesn't exist (or might be out of date) unless you run cargo build -r
right before running the Dart tests.
To run Flutter-specific tests, you will need to create the platform specific binaries, which is what you are mentioning. That can be done through melos run build
(for all platform binaries, only possible on macOS, and takes ages to finish) or melos run build:other
(windows/linux) or melos run build:android
(android). Please note that after you run that build command, you will need to copy the binaries over to the correct part of your /packages/flutter_library_name
directory. That is detailed here.
I will admit the current build system is somewhat of a pain right now, but unfortunately you can't do too much better until Dart supports Native Assets.
To run Dart-only tests, always run
cargo build -r
beforehand. This is your issue here; you are missing the binaries locally. If you look in my Dart unit test source code, you will see that it references the binaries in the/target/release
folder (or something like that), which doesn't exist (or might be out of date) unless you runcargo build -r
right before running the Dart tests.
Perfect, this works for me!
Please note that after you run that build command, you will need to copy the binaries over to the correct part of your
/packages/flutter_library_name
directory.
Could you explain why builds need to be run locally in this case? In the guide, you write:
When the Dart tooling builds our library (such as when an application consuming our library is built), it invokes the platform specific build process. We hijack this build process by downloading a copy of the binaries for the needed platform, if not already present on the filesystem. This last part is the key; it allows us to run integration tests locally and in CI by providing our own copy of the binaries instead of forcing our build process to always fetch the binaries from GitHub releases.
So only projects which depend on the Flutter library can take advantage of this? For example, if I wrote an example which depends on package:flutter_mimir
and then wrote some tests for that example, would I not have to worry about platform-specific code or compiling these binaries at all?
Could you explain why builds need to be run locally in this case?
Sure. That is because you want any consumers of the library to automatically fetch the corresponding release binaries from online (GitHub releases). However, if you are developing locally and make changes on the Rust side, that behavior is incorrect; you would instead want to use the binaries you built locally instead of fetching them from online. The CMake & podspec modifications detailed in the guide only download binaries if not already present, which takes care of this behavior.
So only projects which depend on the Flutter library can take advantage of this? For example, if I wrote an example which depends on package:flutter_mimir and then wrote some tests for that example, would I not have to worry about platform-specific code or compiling these binaries at all?
Correct, by having the CMake/podspec scripts auto-download the binaries if they don't find them on the filesystem in the correct places, users of the library don't have to worry at all about it.
Great - thanks for clarifying!
It looks like the tests run fine after I run cargo build -r
and melos run build:other
, and then copy the generated archive to the relevant folder.
By the way, it looks like that build script has a little hiccup towards the end - I think mkdir
runs on an existing directory
which leads to the warnings towards the end of the following log:
If memory serves, that mkdir failing shouldn't be an issue. If the directory is already created, it doesn't need to be created again (and can just fail). However, the:
cp: cannot stat '../target/x86_64-pc-windows-msvc/release/embedded_milli.dll': No such file or directory
Does look like an issue. For some reason the .dll
is not being created; would you mind pasting the entire output here? Not sure what is going on there.
Or perhaps the .dll
is in a different location that I expected. Maybe it is created in a different spot when running on WSL?
would you mind pasting the entire output here? Not sure what is going on there.
The snipped text is just repeated "Compiling" logs:
It looks like no Windows libraries are present at all:
atom@Dell-Desktop:~/mimir$ tree -L 2 target
target
├── CACHEDIR.TAG
├── aarch64-unknown-linux-gnu
│ ├── CACHEDIR.TAG
│ └── release
├── release
│ ├── build
│ ├── deps
│ ├── examples
│ └── incremental
└── x86_64-unknown-linux-gnu
├── CACHEDIR.TAG
└── release
9 directories, 3 files
In fact, it was cargo-xwin which produced this 'os error 17'. Not sure what's going on here:
atom@Dell-Desktop:~/mimir$ rustup target add aarch64-pc-windows-msvc
info: component 'rust-std' for target 'aarch64-pc-windows-msvc' is up to date
atom@Dell-Desktop:~/mimir$ cargo xwin build --target aarch64-pc-windows-msvc -r
Error: File exists (os error 17)
atom@Dell-Desktop:~/mimir$ cargo xwin --version
cargo-xwin-xwin 0.13.3
Ahh! Thanks for looking into that. Assuming you have LLVM installed, probably an issue with cargo-xwin then. Seeing as everything needed by mimir builds in CI I probably won't look into it any further but if that is blocking for you, you may want to file an issue over at that project. But, seeing as you are developing on Windows, you could also just probably build the binaries locally and not have to interface thru cargo-xwin.
Yeah, I agree.
@CicadaCinema I just realized an issue when running in iOS/macOS in release mode. Binaries were getting stripped as-is. Going to file a PR to fix the docs I made in FRB but since you have already gone thru the guide, switch out your EnforceBundling.swift files with the following:
public func dummyMethodToEnforceBundling() -> Int64 {
return dummy_method_to_enforce_bundling()
}
let dummyVar = dummyMethodToEnforceBundling();
Annoying that the previous version didn't work so you will need to change to that.
@CicadaCinema I just realized an issue when running in iOS/macOS in release mode. Binaries were getting stripped as-is. Going to file a PR to fix the docs I made in FRB but since you have already gone thru the guide, switch out your EnforceBundling.swift files with the following:
public func dummyMethodToEnforceBundling() -> Int64 { return dummy_method_to_enforce_bundling() } let dummyVar = dummyMethodToEnforceBundling();
Annoying that the previous version didn't work so you will need to change to that.
This is unrelated to the issue with cargo-xwin, right?
Thanks for the heads-up! For now I think I'll stick with targeting Linux and Windows though - at least until Native Assets releases.
This is unrelated to the issue with cargo-xwin, right?
correct; just figured I’d let you know since it seemed as though you got through a substantial part of the guide already.
For now I think I'll stick with targeting Linux and Windows though - at least until Native Assets releases.
and that’s a good decision! It’s really a pain to deal with all the platform specific setup as-is.
@GregoryConrad Hi, sorry about bothering you again - I've attempted to write a simple template following your guide, and I'm running into similar issues again.
The corresponding repository is https://github.com/CicadaCinema/flutter_dart_rust_template and I've decided to use a slightly different naming convention to yours, to be able to more easily identify what's what and to be able to replace these strings when using this template in a real project: bbb_project_bbb
is the pure-Dart project and ccc_project_ccc
is the Flutter project which wraps it; these can both be found in packages/
. aaa_project_aaa
is the name of the native Rust library which can be found in packages/bbb_project_bbb/native/
.
In this commit I have created a simple adder function and exported it through bbb_project_bbb
and ccc_project_ccc
. I have also added some Dart tests for these wrappers in the same commit. After building the native library (and moving/renaming the resulting archive to the relevant folder - see logs), the Dart test passes, but the Flutter test (in packages/ccc_project_ccc/
) fails!
Do you think you could take a look?
Invalid argument(s): Failed to load dynamic library 'libaaa_project_aaa.so': libaaa_project_aaa.so: cannot open shared object file: No such file or directory
Looks like your tests can't find the .so
. Did you copy the .tag.gz
archive to the correct place (/packages/bbb_project_bbb/linux/
)?
I haven't tried running Flutter unit tests like that. It may be worth trying to run/use Flutter integration tests, assuming you have copied over the .tar.gz
with the correct name based on your pubspec.yaml.
Did you copy the
.tag.gz
archive to the correct place (/packages/bbb_project_bbb/linux/
)?
Yep! Excerpt from the logs:
atom@Dell-Desktop:~/flutter_dart_rust_template$ mv platform-build/other.tar.gz packages/ccc_project_ccc/linux/aaa_project_aaa-v0.0.0.tar.gz
atom@Dell-Desktop:~/flutter_dart_rust_template$ ls -lah packages/ccc_project_ccc/linux
total 2.3M
drwxr-xr-x 2 atom atom 4.0K Dec 25 17:22 .
drwxr-xr-x 11 atom atom 4.0K Dec 25 17:21 ..
-rw-r--r-- 1 atom atom 1.3K Dec 25 17:21 CMakeLists.txt
-rw-r--r-- 1 atom atom 2.3M Dec 25 17:22 aaa_project_aaa-v0.0.0.tar.gz
Based on your linux/CMakeLists.txt, looks like that should be working. I never tried doing Flutter unit tests, which is why I didn't write much on it in the guide. Looks like you probably won't be able to use Rust in Flutter unit tests, but integration tests should work.
Oh! You could also do something similar to the Dart-only unit tests for the Flutter unit tests, by referencing /target/release/...so
! Forgot about that for a second.
I.e. you will need to get your Dynamic Library instance manually by using DynamicLibrary.open() and point it to your /target/release/...
built file. You won't be able to rely upon the FFI convenience I made (createLib()
), at least in Flutter unit tests.
Looks like you probably won't be able to use Rust in Flutter unit tests, but integration tests should work.
Is this true even for packages depending on ccc_project_ccc
? I removed the unit tests in ccc_project_ccc
, leaving the unit tests in ccc_project_ccc_example
. These fail too:
Does this mean that it is fundamentally impossible to write passing unit tests which rely on the Rust native library, even if the library is bundled with a dependency of the project being tested?
That is to say, the native functionality will be available to the project when built and in integration tests, but not in unit tests?
Is this true even for packages depending on ccc_project_ccc? I removed the unit tests in ccc_project_ccc, leaving the unit tests in ccc_project_ccc_example. These fail too:
Correct, any unit tests depending on ccc will fail unless they use the binary or mock the implementation.
Does this mean that it is fundamentally impossible to write passing unit tests which rely on the Rust native library, even if the library is bundled with a dependency of the project being tested?
Not necessarily impossible; you have a few options:
ccc
can use the binary for their platform to run their unit tests (although I wouldn't necessarily recommend this approach unless the user of ccc
is another package in your mono repo).ccc
can provide a stub implementation for testing that does not actually call/use FRB, but I would suggest only allowing this internally in the mono repo, and it could be risky.ccc
mock the ccc
functions. IMHO, any unit testing relying on ccc
should be doing this anyways, as they are unit tests. Any functionality actually interacting with the outside environment should be an integration test anyways.All that being said, integration tests should all pass as expected, based on what I see in your repo. I have a feeling you are really looking for integration tests here, not unit tests.
Thank you for clarifying! I've added an integration test and it does pass, as expected!
Hi, I'm trying to use mimir to better understand the guide you wrote for flutter_rust_bridge.
I am facing an issue which I can replicate with mimir:
As you can see, this test fails because the shared library cannot be found.
According to the guide you wrote, the shared library binaries are distributed through GitHub Releases, so I did not build them before running the tests. If I do build the shared library however, the result is similar:
Click to expand
``` atom@Dell-Desktop:~$ git clone https://github.com/GregoryConrad/mimir && cd mimir && melos clean && melos bootstrap && bash scripts/build-other.sh && melos run test Cloning into 'mimir'... remote: Enumerating objects: 2609, done. remote: Counting objects: 100% (542/542), done. remote: Compressing objects: 100% (311/311), done. remote: Total 2609 (delta 315), reused 295 (delta 230), pack-reused 2067 Receiving objects: 100% (2609/2609), 3.42 MiB | 11.60 MiB/s, done. Resolving deltas: 100% (1358/1358), done. Cleaning workspace... Workspace cleaned. You will need to run the bootstrap command again to use this workspace. melos bootstrap └> /home/atom/mimir Running "flutter pub get" in workspace packages... ✓ example └> packages/mimir/example ✓ mimir └> packages/mimir ✓ flutter_mimir └> packages/flutter_mimir ✓ flutter_example └> packages/flutter_mimir/example > SUCCESS Generating IntelliJ IDE files... > SUCCESS -> 4 packages bootstrapped Updating crates.io index Ignored package `cargo-zigbuild v0.14.2` is already installed, use --force to override Updating crates.io index Ignored package `cargo-xwin v0.13.3` is already installed, use --force to override info: component 'rust-std' for target 'aarch64-unknown-linux-gnu' is up to date Updating crates.io index Updating git repository `https://github.com/GregoryConrad/milli` Updating git repository `https://github.com/meilisearch/heed` Updating git repository `https://github.com/meilisearch/lmdb-rs` Compiling proc-macro2 v1.0.49 Compiling quote v1.0.23 Compiling unicode-ident v1.0.6 ... SNIP ... Compiling lindera v0.14.0 Compiling charabia v0.6.0 Compiling milli v0.37.0 (https://github.com/GregoryConrad/milli?branch=filter-parser-convenience#50954d31) Finished release [optimized] target(s) in 2m 11s info: component 'rust-std' for target 'x86_64-unknown-linux-gnu' is up to date Compiling libc v0.2.139 Compiling cfg-if v1.0.0 Compiling serde v1.0.151 ... SNIP ... Compiling lindera v0.14.0 Compiling charabia v0.6.0 Compiling milli v0.37.0 (https://github.com/GregoryConrad/milli?branch=filter-parser-convenience#50954d31) Finished release [optimized] target(s) in 1m 22s info: component 'rust-std' for target 'aarch64-pc-windows-msvc' is up to date Compiling cfg-if v1.0.0 Compiling serde v1.0.151 Compiling winapi v0.3.9 ... SNIP ... Compiling lindera v0.14.0 Compiling charabia v0.6.0 Compiling milli v0.37.0 (https://github.com/GregoryConrad/milli?branch=filter-parser-convenience#50954d31) Finished release [optimized] target(s) in 1m 27s info: component 'rust-std' for target 'x86_64-pc-windows-msvc' is up to date Compiling cfg-if v1.0.0 Compiling serde v1.0.151 Compiling winapi v0.3.9 ... SNIP ... Compiling lindera v0.14.0 Compiling charabia v0.6.0 Compiling milli v0.37.0 (https://github.com/GregoryConrad/milli?branch=filter-parser-convenience#50954d31) Finished release [optimized] target(s) in 1m 27s linux-arm64/ linux-arm64/libembedded_milli.so linux-x64/ linux-x64/libembedded_milli.so windows-arm64/ windows-arm64/embedded_milli.dll windows-x64/ windows-x64/embedded_milli.dll melos run test └> melos run test:dart --no-select && melos run test:flutter --no-select └> RUNNING melos run test:dart └> melos exec -c 1 --fail-fast -- "dart test test" └> RUNNING $ melos exec └> dart test test └> RUNNING (in 2 packages) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- example: 00:01 +0 -1: test/example_test.dart: Main function runs correctly [E] Invalid argument(s): Failed to load dynamic library '../../../target/release/libembedded_milli.so': ../../../target/release/libembedded_milli.so: cannot open shared object file: No such file or directory dart:ffi new DynamicLibrary.open package:example/example.dart 136:25 getLibrary package:example/example.dart 16:28 main To run this test again: /home/atom/snap/flutter/common/flutter/bin/cache/dart-sdk/bin/dart test test/example_test.dart -p vm --plain-name 'Main function runs correctly' 00:02 +0 -1: Some tests failed. Consider enabling the flag chain-stack-traces to receive more detailed exceptions. For example, 'dart test --chain-stack-traces'. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- $ melos exec └> dart test test └> FAILED (in 1 packages) └> example (with exit code 1) melos run test:dart └> melos exec -c 1 --fail-fast -- "dart test test" └> FAILED ScriptException: The script test:dart failed to execute melos run test └> melos run test:dart --no-select && melos run test:flutter --no-select └> FAILED ScriptException: The script test failed to execute ```I would appreciate it if you could give any advice on any debugging steps I could take to set up my system to run passing tests from this repository.
System information
Click to expand
**I am running Ubuntu Linux via WSL.** #### Windows ``` C:\Users\Atom>ver Microsoft Windows [Version 10.0.19045.2364] ``` #### WSL ``` C:\Users\Atom>wsl --version WSL version: 1.0.3.0 Kernel version: 5.15.79.1 WSLg version: 1.0.47 MSRDC version: 1.2.3575 Direct3D version: 1.606.4 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.19045.2364 ``` #### Linux Kernel ``` atom@Dell-Desktop:~$ uname -a Linux Dell-Desktop 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ``` #### Flutter ``` atom@Dell-Desktop:~$ flutter doctor -v [✓] Flutter (Channel stable, 3.3.10, on Ubuntu 22.04.1 LTS 5.15.79.1-microsoft-standard-WSL2, locale C.UTF-8) • Flutter version 3.3.10 on channel stable at /home/atom/snap/flutter/common/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision 135454af32 (9 days ago), 2022-12-15 07:36:55 -0800 • Engine revision 3316dd8728 • Dart version 2.18.6 • DevTools version 2.15.0 [✗] Android toolchain - develop for Android devices ✗ Unable to locate Android SDK. Install Android Studio from: https://developer.android.com/studio/index.html On first launch it will assist you in installing the Android SDK components. (or visit https://flutter.dev/docs/get-started/install/linux#android-setup for detailed instructions). If the Android SDK has been installed to a custom location, please use `flutter config --android-sdk` to update to that location. [✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome) ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable. [✓] Linux toolchain - develop for Linux desktop • clang version 10.0.0-4ubuntu1 • cmake version 3.16.3 • ninja version 1.10.0 • pkg-config version 0.29.1 [!] Android Studio (not installed) • Android Studio not found; download from https://developer.android.com/studio/index.html (or visit https://flutter.dev/docs/get-started/install/linux#android-setup for detailed instructions). [✓] Connected device (1 available) • Linux (desktop) • linux • linux-x64 • Ubuntu 22.04.1 LTS 5.15.79.1-microsoft-standard-WSL2 [✓] HTTP Host Availability • All required HTTP hosts are available ! Doctor found issues in 3 categories. ``` #### Flutter Snap (this is how Flutter and Dart are installed on my system) ``` atom@Dell-Desktop:~$ snap info flutter name: flutter summary: Flutter SDK publisher: Flutter Team✓ store-url: https://snapcraft.io/flutter contact: https://github.com/flutter/flutter/issues license: unset description: | Flutter is Google’s UI toolkit for building beautiful, natively compiled applications for mobile, web, and desktop from a single codebase. commands: - flutter.dart - flutter - flutter.openurl snap-id: YO0Adf4fqgxL7i3SaCn00oxEfd5CNQ63 tracking: latest/stable refresh-date: yesterday at 17:27 GMT channels: latest/stable: 0+git.ccbc698 2022-10-20 (130) 215MB classic latest/candidate: ↑ latest/beta: ↑ latest/edge: 0+git.74565fa 2022-11-29 (133) 216MB classic installed: 0+git.ccbc698 (130) 215MB classic ``` #### Rust ``` atom@Dell-Desktop:~$ rustup --version rustup 1.25.1 (bb60b1e89 2022-07-12) info: This is the version for the rustup toolchain manager, not the rustc compiler. info: The currently active `rustc` version is `rustc 1.66.0 (69f9c33d7 2022-12-12)` ```