[Bug] flutter_rust_bridge_codegen can't understand enums/struct defined by macros

JustSimplyKyle commented 1 year ago

Describe the bug

It failed to generate types defined by macros.

Codegen logs with `RUST_LOG=debug` environment variable

2023/07/31 18:31:57 [DEBUG] configs=[Opts { rust_input_path: "/home/kyle/coding/era-connect/app/native/src/api.rs", dart_output_path: "/home/kyle/coding/era-connect/app/lib/api/gen/bridge_generated.dart", dart_decl_output_path: Some("/home/kyle/coding/era-connect/app/lib/api/gen/bridge_definitions.dart"), c_output_path: ["/home/kyle/coding/era-connect/app/macos/Runner/bridge_generated.h"], rust_crate_dir: "/home/kyle/coding/era-connect/app/native", rust_output_path: "/home/kyle/coding/era-connect/app/native/src/bridge_generated.rs", class_name: "Native", dart_format_line_length: 80, dart_enums_style: false, skip_add_mod_to_lib: false, llvm_path: ["/opt/homebrew/opt/llvm", "/usr/local/opt/llvm", "/usr/lib/llvm-9", "/usr/lib/llvm-10", "/usr/lib/llvm-11", "/usr/lib/llvm-12", "/usr/lib/llvm-13", "/usr/lib/llvm-14", "/usr/lib/", "/usr/lib64/", "C:/Program Files/llvm", "C:/msys64/mingw64"], llvm_compiler_opts: "", manifest_path: "/home/kyle/coding/era-connect/app/native/Cargo.toml", dart_root: Some("/home/kyle/coding/era-connect/app"), build_runner: true, block_index: BlockIndex(0), skip_deps_check: false, wasm_enabled: false, inline_rust: false, bridge_in_method: true, extra_headers: "", dart3: true }]
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/config/mod.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/config/config_loader.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/config/config_state.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/config/ui_layout.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/download/mod.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/download/assets.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/download/library.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/download/rules.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/download/util.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/logger.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/quilt.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/api/vanilla.rs"
2023/07/31 18:31:57 [DEBUG] Trying to parse "/home/kyle/coding/era-connect/app/native/src/bridge_generated.rs"
2023/07/31 18:31:57 [WARN] Skipping unresolvable module io (tried /home/kyle/coding/era-connect/app/native/src/io.rs, /home/kyle/coding/era-connect/app/native/src/io/mod.rs, /home/kyle/coding/era-connect/app/native/src/bridge_generated/io.rs, /home/kyle/coding/era-connect/app/native/src/bridge_generated/io/mod.rs, )
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(get_logger)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(download_vanilla)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(launch_game)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(download_quilt)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(dummy)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(fetch_state)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(write_state)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(get_ui_layout_config)
2023/07/31 18:31:57 [DEBUG] parse_function function name: Ident(set_ui_layout_config)
2023/07/31 18:31:57 [ERROR] panicked at 'not yet implemented: generate code for type "TestEnum"', /home/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/flutter_rust_bridge_codegen-1.79.0/src/ir/ty_unencodable.rs:59:9
thread 'main' panicked at 'not yet implemented: generate code for type "TestEnum"', /home/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/flutter_rust_bridge_codegen-1.79.0/src/ir/ty_unencodable.rs:59:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: Recipe `gen` failed on line 9 with exit code 101

To Reproduce

#[macro_export]
macro_rules! test {
    () => {
        pub enum TestEnum {
            Fail,
        }
    };
}
test!();

pub fn dummy(a: TestEnum) {}

Expected behavior

To work!

Generated binding code

No response

OS

Arch Linux 6.3.7

Version of `flutter_rust_bridge_codegen`

1.79.0

Flutter info

[!] Flutter (Channel stable, 3.10.6, on Arch Linux 6.3.7-273-tkg-pds, locale en_US.UTF-8)
    • Flutter version 3.10.6 on channel stable at /opt/flutter
    ! Warning: `dart` on your path resolves to /opt/dart-sdk/bin/dart, which is not inside your current Flutter SDK checkout at /opt/flutter. Consider adding /opt/flutter/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision f468f3366c (3 weeks ago), 2023-07-12 15:19:05 -0700
    • Engine revision cdbeda788a
    • Dart version 3.0.6
    • DevTools version 2.23.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✗] Android toolchain - develop for Android devices
    • Android SDK at /opt/android-sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.

[✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.

[✓] Linux toolchain - develop for Linux desktop
    • clang version 15.0.7
    • cmake version 3.27.1
    • ninja version 1.11.1
    • pkg-config version 1.8.1

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/linux#android-setup for detailed instructions).

[✓] Connected device (1 available)
    • Linux (desktop) • linux • linux-x64 • Arch Linux 6.3.7-273-tkg-pds

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 4 categories.

Version of `clang++`

15.0.7

Version of `ffigen`

8.0.2

Additional context

No response

fzyzcjy commented 1 year ago

Yes that's expected in the current version. Macros can create arbitrary things, while our parser executes before macros are run currently.

However, feel free to PR! One naive way may be that, we may call rust compiler to expand the macro, and run our frb only on that expanded code.

In addition, another way is that, you can check other packages that parses rust code (e.g. rust-to-c code generator?). If some packages somehow understand this, it would be great to learn from it.

JustSimplyKyle commented 1 year ago

Rust2C? I can only find C2rust https://github.com/dtolnay/cargo-expand#disclaimer There is this, so the second option isn't that great

JustSimplyKyle commented 1 year ago

Currently my "workaround" is to have a module, containing only that macro. and then use cargo expand to expand that macro. Kinda like a code generation, within a code generation. I don't think this workaround can be written within frb, it might be out of the scope of the project. A separate utility would be much suitble

fzyzcjy commented 1 year ago

Oh I mean something like https://github.com/mozilla/cbindgen or https://github.com/PyO3/pyo3 - check whether their current version support things inside macro

Yes I do expect something like cargo-expand.

flutter_rust_bridge already calls a lot of utilities inside it (e.g. cbindgen, ffigen, etc), so it is not a problem to let it call cargo-expand :)

MnlPhlp commented 1 year ago

I tried experimenting with cargo expand to get this working but it creates a few problems.

All the marker macros are gone after expanding (e.g. mirror)
The build scrips in the tests have to be changed because they create a deadlock
- cargo build calls the generator
- generator calls cargo expand
- cargo expand calls cargo build (or something similar that executes the build-script)
- infinite loop ...

I got some stuff working without the build-scripts and by using doc-comments as mirror markers. Might be worth to create a pull request and use it as a starting point and for discussion.

fzyzcjy commented 1 year ago

All the marker macros are gone after expanding (e.g. mirror) I got some stuff working without the build-scripts and by using doc-comments as mirror markers

Hmm, one naive way may be, we parse and translate frb marker macros into a comment. For example, we can parse#[frb(apple, orange)] into a internal struct, and use serde to serialize it into a json and become a comment like // FRB: {"apple": true, "orange": whatever} etc

The build scrips in the tests have to be changed because they create a deadlock

One more naive idea for brainstorming: Maybe we can use environment variable to control the behavior of the build script. In other words, our frb build script may like:

fn main_function_in_build_script() {
  if (env("FRB_IS_DOING_CARGO_EXPAND") == "yes!!!") {
    println!("do nothing when doing cargo expand to avoid deadlock");
  } else {
    run_frb_codegen();
  }
}

fn run_frb_codegen() {
  execute_shell_command("cargo expand", environment_variables={"FRB_IS_DOING_CARGO_EXPAND": true});
  ...
}

MnlPhlp commented 1 year ago

I will try this next week. But I think it still doesn't work because just running cargo-expand from the buildscript already creates a deadlock. The cargo expand will hang at "waiting for file lock on build directory" or something like that. Maybe there is a way to ignore that lock.

fzyzcjy commented 1 year ago

Hmm that looks like a problem. More ideas for brainstorm:

Is it possible to use something in a level lower than cargo expand, which ignores the lock?
Maybe we should disable cargo expand (and warn the user) when in buildscript mode, and only enable cargo expand when executed as a standalone binary?

MnlPhlp commented 1 year ago

@JustSimplyKyle This should be fixed by #1320 So you can try again after the next release

JustSimplyKyle commented 1 year ago

Thanks for the pr! can confirm works in my use case

fzyzcjy commented 1 year ago

Oops I forget to execute the release script, will do it now

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.

fzyzcjy / flutter_rust_bridge