Cross-compiling for Android from Linux

[sneurlax]

We are using cargokit here: https://github.com/cypherstack/tor/tree/no_scripts

We were having issues with building for Android so updated cargokit: https://github.com/cypherstack/tor/tree/no_scripts_updated_cargokit

However, this broke running for Linux, too. (error here, verbose log here, flutter doctor output here) So we reverted to the older cargokit.

Then we found this comment:

// Right now we don't support cross-compiling on Linux. So we just return // the host target.

So, we are wondering if there's a good direction to go in in order to enable cross-compiling for Android from Linux. Got any pointers? We'd love to help get this working better

We would also like to help fix running on Linux with the updated cargokit instead of using the older version from https://github.com/irondash/hello_rust_ffi_plugin

Thanks!

[knopp]

Hey! Cross-compiling on linux means cross compiling for x86_64 linux to arm64 and vice versa. That is not supported because it's complicated to setup dependencies right now.

Cross compiling to android should always work.

But from the logs it looks like a possible bug in pub? The error is not directly from cargo kit, it seems like an unexpected exception when running pub on the build tool script.

[knopp]

Can you clean the /home/user/.pub-cache/log/pub_log.txt file, run the build again, and let me know if there is something interesting in the pub_log?

[sneurlax]

I cleaned the file and reran the build but it's essentially the same, ERR being

ERR : Bad state: No element
FINE: Exception type: StateError
ERR : dart:collection                                      ListQueue.first
    | package:pub/src/io.dart 149:31                       canonicalize
    | package:pub/src/source/path.dart 250:9               PathDescription.==

We're having this issue on linux for linux* on two separate Ubuntu 20.04 machines and on mac for iOS and macos, as well (same Bad state: no element error). We've tried using various different Flutter and Rust versions. Mac can build for Android fine, maybe our setup is incorrect. Work ongoing

[knopp]

We're building precompiled libraries for super_native_extensions for android on linux github action, I also periodically test this on linux. So it seems like it is something specific to your setup. Would be great to find out what and either fix this in pub or have a workaround for cargokit.

[knopp]

I am able to reproduce the problem locally. That's a start :)

[knopp]

Reported here: https://github.com/dart-lang/pub/issues/4010

[sneurlax]

Reported here: dart-lang/pub#4010

@knopp this having been merged I'm just trying to figure out how to build it all locally to test this out!

[knopp]

Reported here: dart-lang/pub#4010

@knopp this having been merged I'm just trying to figure out how to build it all locally to test this out!

That might be tricky unless you want to build your own engine with updated dart-sdk. I'm looking into adding workaround to cargokit where the path would be resolved before pub gets to it, which might be necessary - i'm not sure how long it will take for the fix to roll into stable flutter.

[sneurlax]

I'm willing to :D just looking for pointers

[sneurlax]

Would following https://github.com/dart-lang/sdk/wiki/Building work?

[sneurlax]

PS, I confirmed this works for Android on a Linux host, thanks!

We are just still having issues for macOS now.

[knopp]

PS, I confirmed this works for Android on a Linux host, thanks!

We are just still having issues for macOS now.

I looked at the code but the scirpt phase seems to be commented out in macos podspec? Does it fail when you uncomment it macOS and iOS both use cocoa pods so if it works in one it should in the other. In fact you might be able to get away with same podspec for both platforms. For example the hello_rust_ffi_plugin ios folder is just symlink to macos.

[julian-CStack]

I believe there is an issue with the rust code compilation and not cargo kit itself. This is probably out of the scope of cargo kit specifically but the error is here:

Launching lib/main.dart on macOS in debug mode...
Running pod install...
Building macOS application...
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00006021-0010303C2EEBC01E }
{ platform:macOS, arch:x86_64, id:00006021-0010303C2EEBC01E }
Undefined symbols for architecture arm64:
  "_lzma_alone_decoder", referenced from:
      xz2::stream::Stream::new_lzma_decoder::h2ddcd255e5327597 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_alone_encoder", referenced from:
      xz2::stream::Stream::new_lzma_encoder::h407225d8c42fe2ee in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_auto_decoder", referenced from:
      xz2::stream::Stream::new_auto_decoder::hcf8f292248d82111 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_check_is_supported", referenced from:
      xz2::stream::Check::is_supported::h3256ca4a2b4fda16 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_code", referenced from:
      xz2::stream::Stream::process::h02ad63d15520a45a in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_easy_encoder", referenced from:
      xz2::stream::Stream::new_easy_encoder::h443f2e6110e1e6be in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_end", referenced from:
      _$LT$xz2..stream..Stream$u20$as$u20$core..ops..drop..Drop$GT$::drop::hc71dd9ed27d79a26 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_lzma_preset", referenced from:
      xz2::stream::LzmaOptions::new_preset::h0b1faee695ad43e3 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_memlimit_get", referenced from:
      xz2::stream::Stream::memlimit::h818027012714fd95 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_memlimit_set", referenced from:
      xz2::stream::Stream::set_memlimit::hbe9c47f5ecc7a176 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_mf_is_supported", referenced from:
      xz2::stream::MatchFinder::is_supported::hd8afe1df57a683e7 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_stream_decoder", referenced from:
      xz2::stream::Stream::new_stream_decoder::h6eea11fde4a7c890 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_stream_encoder", referenced from:
      xz2::stream::Stream::new_stream_encoder::hf4af5c4ed3fbe287 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_stream_encoder_mt", referenced from:
      xz2::stream::MtStreamBuilder::encoder::h66248d216a1a9a63 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_stream_encoder_mt_memusage", referenced from:
      xz2::stream::MtStreamBuilder::memusage::h4abbb0d41ae10638 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
warning: Run script build phase 'Run Script' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'Flutter Assemble' from project 'Runner')
** BUILD FAILED **

Exception: Build process failed

My quick hack to get https://github.com/cypherstack/tor/tree/staging to work on arm64 macos was to build a dylib for aarch64-apple-darwin and tell cocoapods to use that instead of trying to build via cargo kit.

Like I mentioned above, I'm not very familiar with rust and I have zero familiarity with this specific rust codebase we are compiling so this may be unrelated to cargo kit. I could be wrong about this though

[knopp]

The issue here likely is that you need to link liblzma (-llzma) into the final dylib. Unlike other platforms on iOS and macOS the rust crate is compiled as static library that needs to be linked to cocoapod dylib. In which case all the additional libraries that need to be linked with it, that would otherwise be part of rust cdylib, need to be specified using linker flags inside podspec.

[knopp]

Can you point me to the branch that gives you the error? I can take a look.

[julian-CStack]

~https://github.com/cypherstack/tor/tree/macos_issue~ https://github.com/cypherstack/tor/commit/10fb5c9f456426d7ed139821a9e744f52a89b0f4 gives me the above error.

Adding -llzma to LDFLAGS in the Podspec yields:

Launching lib/main.dart on macOS in debug mode...
Running pod install...
Building macOS application...
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00006021-0010303C2EEBC01E }
{ platform:macOS, arch:x86_64, id:00006021-0010303C2EEBC01E }
Undefined symbols for architecture arm64:
  "_lzma_stream_encoder_mt", referenced from:
      xz2::stream::MtStreamBuilder::encoder::h66248d216a1a9a63 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
  "_lzma_stream_encoder_mt_memusage", referenced from:
      xz2::stream::MtStreamBuilder::memusage::h4abbb0d41ae10638 in libtor_ffi_plugin.a(xz2-9291dd1791ee9b79.xz2.307c095da4d833f1-cgu.0.rcgu.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
warning: Run script build phase 'Run Script' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'Flutter Assemble' from project 'Runner')
** BUILD FAILED **

Exception: Build process failed

edit: changed above link due to temporary branch deletion

[knopp]

I see what the problem is. My suggestion with adding -llzma was not correct. The lzma library is already part of static library. The problem is, that for iOS build liblzma gets build automatically by the lzma-sys crate, while for macos build it attempts to link existing system library, in my case from homebrew.

[knopp]

It seems that we can force building static library by setting the LZMA_API_STATIC environment variable during crate build. But there's no way to set arbitrary environment variable for cargokit build. Let me look into this.

[knopp]

@julian-CStack, try adding the following dependency in Cargo.tom:

lzma-sys = { version = "0.1.20", features = ["static"] }

That should force static library build for lzma-sys on all platforms.

[julian-CStack]

@julian-CStack, try adding the following dependency in Cargo.tom:

lzma-sys = { version = "0.1.20", features = ["static"] }

That should force static library build for lzma-sys on all platforms.

Fantastic! This fixes the issue for me. Thank you for your time on this!