Closed david-allison closed 1 year ago
TarekkMA
commented
3 years ago Do you mean to make a script which:
What does these steps mean:
david-allison
commented
3 years ago Sorry! Was writing the instructions for myself.
Please also feel free to mark where the below needs to be moved in the documentation
4 Android targets
.so for 4 different Android architectures:These need to be installed as targets for Rust:
2/3 targets for testing
In Anki-Android, we run a lot of unit tests under Robolectric: https://github.com/robolectric/robolectric
This runs a simulated Android runtime under the JVM.
The problem with this is that the Rust libraries are built for Android, not for your native PC OS.
To fix this, we have rsdroid-testing, which is a jar, containing the rust code compiled as native libraries for Windows (dll), macOS (dylib), and Linux (.so)
So we need rust targets for these
macOS
To my knowledge, a rust target cannot legally be cross-compiled for macOS unless on a macOS computer, as it requires the Apple SDK, which is protected by a license: https://www.apple.com/legal/sla/docs/xcode.pdf
We're fine with this on GitHub, as we can use macOS on our CI to generate a full build.
So, a warning/mention that you'd only be cross-compiling Windows/Linux targets on Windows/Linux machines - a "release-quality .jar" for the .jar wouldn't be possible (as it'd be missing the .dylib), but you could create one for your local machine for testing.
Nightly for testing
rsdroid-testing currently uses cargo's --out-dir flag, which (for now) requires that a rust nightly build is set up.
https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#out-dir
C compiler
I believe this is Microsoft C++ Build Tools: https://users.rust-lang.org/t/why-do-i-need-microsoft-c-build-tools/18581/2 on Windows. Needed for the linker.
Note: WSL should be the preferred environment for anything on Windows at this point
TarekkMA
commented
3 years ago So we want a script to do all the checks and possibly to install things that can be installed like targets, rust itself, etc.
And to write a dev doc with the exact steps that should be considered, so the environment is ready.
And if WSL will be used, so bash script would be a universal script that doesn't need any dependency
david-allison
commented
3 years ago Yep - sounds good to me!
Feel free to take the design in any way that you feel is best - I have a few scripts which may be useful as reference: https://github.com/david-allison-1/Anki-Android-Backend/tree/rust-wip/.github/scripts
TarekkMA
commented
3 years ago Thank you will work on that, and keep you updated.
mikehardy
commented
3 years ago macOS
To my knowledge, a rust target cannot legally be cross-compiled for macOS unless on a macOS computer, as it requires the Apple SDK, which is protected by a license: https://www.apple.com/legal/sla/docs/xcode.pdf
We're fine with this on GitHub, as we can use macOS on our CI to generate a full build.
So, a warning/mention that you'd only be cross-compiling Windows/Linux targets on Windows/Linux machines - a "release-quality .jar" for the .jar wouldn't be possible (as it'd be missing the
.dylib), but you could create one for your local machine for testing.
It might be useful to mention that for people experimenting with macOS that is possible to virtualize it https://github.com/kholia/OSX-KVM - the license considerations are the same but it is (for instance) possible to run linux on mac hardware so it's fine to mention. You can even run OSX in Docker nowadays https://github.com/sickcodes/Docker-OSX
Note: WSL should be the preferred environment for anything on Windows at this point
Yes yes yes
TarekkMA
commented
3 years ago This is nice, there is a rust project named cross, which aims to make cross compilation easier using dockers. https://github.com/rust-embedded/cross
And also if we run on the CI on macOS we can compile for the 3 platforms at once without docker. I have manged to compile for windows from macOS
david-allison
commented
3 years ago This is nice, there is a rust project named cross, which aims to make cross compilation easier using dockers. https://github.com/rust-embedded/cross
And also if we run on the CI on macOS we can compile for the 3 platforms at once without docker. I have manged to compile for windows from macOS
CI was already set up for this: https://github.com/david-allison-1/Anki-Android-Backend/actions/runs/556385657 and does exactly that. Note: I'll need to look into the CI failures - GitHub Actions changed. Shouldn't be too hard.
AFAIK, cross doesn't support x86_64-apple-darwin
I had a full build working with https://github.com/tpoechtrager/osxcross on Windows (WSL), but ran across: https://github.com/tpoechtrager/osxcross#packaging-the-sdk
TarekkMA
commented
3 years ago Yes I didn't find it, but thought it could be extended with custom platform-docker image
TarekkMA
commented
3 years ago I think we will also need to add all PC target nightly tool chains.
I had issues with error: could not compilecfg-if` andcan't find crate for core` solved using
rustup toolchain install nightly-x86_64-unknown-linux-gnu #in gradle comments
rustup target add x86_64-unknown-linux-gnu --toolchain nightly #but after this the issue did go awayAnd for macOS I had to install this
rustup component add rustfmt --toolchain nightly-x86_64-apple-darwinLinker for Linux, do you have an idea how to solve the linker issue?
# cc is clang macOS [FAIL]
sudo ln -s /usr/bin/cc /usr/local/bin/x86_64-unknown-linux-gnu-gcc
# Also fail
sudo ln -s /usr/local/bin/gcc-10 /usr/local/bin/x86_64-unknown-linux-gnu-gcc
The environment of this project is so hard to set up
Yesterday I faced the issue of being unable to install protobuf-compiler becase I needed to have an older version of python (3.7), tried to install pyenv as done in scripts but i faced issues and it seems like other users have these same issues also.
I did install python 3.7 via homebrew and it worked but still fails on gradle.
I couldn't run grade from the command line so changed this line in gradle-wapper.properties
- distributionUrl=https\://services.gradle.org/distributions/gradle-6.1.1-all.zip
+ distributionUrl=https\://services.gradle.org/distributions/gradle-6.3-all.zipAnd still figuring out the rest of the issues
david-allison
commented
3 years ago This isn't going to be an easy one to solve at all. There's a lot of moving parts.
One of the main intentions of having this in Anki-Android-Backend is that contributors to Anki-Android don't get discouraged by the complexity and can immediately contribute.
I /really/ appreciate the help with this, feel free to PR any incremental work/commits, as it will save a lot of hassle for others.
TarekkMA
commented
3 years ago Currently, the pain points of setting the environment is not the rust or NDK, but it is the rsdriod-testing module and the nature of cross compilation. And also generating the protobuf classes (on macOS at least).
I was thinking if it's okay to utilize cross pakage. And we don't cross compile osx binaries anyway ?
As they say it's
“Zero setup” cross compilation and “cross testing” of Rust crates
And I think we really need that 😄
And I think we can integrate https://hub.docker.com/r/sickcodes/docker-osx to work with corss via a custom docker setup. I didn't look into the details, but I think it's doable. The only bad part about it is the image is about 20 GB, so if we get to enable it, we should make a flag on local.properties for example to enable it. So anyone who wishes to generate a full "release-quality .jar" can.
I will leave the protobuf issue for now. Will tackle it after I get the cross compilation sorted
mikehardy
commented
3 years ago Food for thought, in other projects where I have a lot of environment difficulty, I maintain a 'doctor' script that examines the environment for expected versions of all toolchain items, and knows how to install them or upgrade them in order to confirm that the environment is ready to go at the end
I think maintaining a Docker image may be useful but that implies that development will be done in the Docker image yes? Will that be high performance enough? I think it might be so this could be a good direction
At that point the doctor.sh script I'm attaching - as an idea - would be implemented instead as a Dockerfile right? Or a set of Dockerfiles if needed (unix, mac, windows) if possible.
TarekkMA
commented
3 years ago I wanted to do this at first, build a doctor utility script in bash or in gradle, but I found out that instillation of some components will be cumbersome and will be different from user to user even if they have same os.
And the only part that we need docker(cross package) for is the rsdriod-testing module. Where cross compilation is required.
If the user is running macOS for example he will use his OS for compiling macOS and other 2 OSs with docker.
I don't know if it will affect the performance drastically as it is only for rsdriod-testing, @david-allison-1 may know if that is the case
But if t I think we can make 2 versions of the build, one where the user only compile his OS binaries for use during development, and the other for the full "release-quality.jar"
Using cross we wouldn't need to maintain any Dockerfiles. Maybe we would need it for the osx one, but we will leave that as the last thing we think about.
We might still need doctor script to check NDK, Rust, Java, Docker, etc and those are very straight forward to install on any system
TarekkMA
commented
3 years ago I don't know why cross giving me this error about linker not found. Cross did work on my toy projects just fine with no extra setup.
Oh, I forgot about the symbolic link trick.
Using a custom docker and making the link seems to make it work.
FROM rustembedded/cross:x86_64-unknown-linux-gnu
RUN ln -s /usr/bin/cc /usr/local/bin/x86_64-unknown-linux-gnu-gccBut the issue with anki/rslib still exists.
TarekkMA
commented
3 years ago The above error about the linker does only happen when compiling rsdriod-bridge
In rslib it does fetch and compile all packages under cross docker (win/linux) but it gives me this error.
When I do compile it natively it does work.
BTW I moved anki to be in the same directory as rsdriod-bridge, since this is a limitation in cross being unable to get local crates outside the project directory.
Caused by:
process didn't exit successfully: `/target/release/build/anki-127f4a73d84ce9fa/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-changed=./ftl/card-stats.ftl
cargo:rerun-if-changed=./ftl/card-template-rendering.ftl
cargo:rerun-if-changed=./ftl/card-templates.ftl
cargo:rerun-if-changed=./ftl/custom-study.ftl
cargo:rerun-if-changed=./ftl/database-check.ftl
cargo:rerun-if-changed=./ftl/deck-config.ftl
cargo:rerun-if-changed=./ftl/empty-cards.ftl
cargo:rerun-if-changed=./ftl/filtering.ftl
cargo:rerun-if-changed=./ftl/findreplace.ftl
cargo:rerun-if-changed=./ftl/importing.ftl
cargo:rerun-if-changed=./ftl/media-check.ftl
cargo:rerun-if-changed=./ftl/network.ftl
cargo:rerun-if-changed=./ftl/notetypes.ftl
cargo:rerun-if-changed=./ftl/scheduling.ftl
cargo:rerun-if-changed=./ftl/search.ftl
cargo:rerun-if-changed=./ftl/statistics.ftl
cargo:rerun-if-changed=./ftl/sync.ftl
cargo:rerun-if-changed=../proto/backend.proto
cargo:rerun-if-changed=../proto/sqlite.proto
--- stderr
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', anki/rslib/build.rs:217:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceit seems to be diffrent one.
david-allison
commented
3 years ago This implies that rslib\ftl\anki-core-i18n\core doesn't exist - error message could be improved (I thought I'd already done this, apologies)
Creating anki-core-i18n was handled by the Makefile in Anki, I changed anki-core-i18n to refer to a submodule (to allow for reproducible builds)
The issues are going to be associated with the files: anki\rslib\build.rs or rslib-bridge\build.rs
TarekkMA
commented
3 years ago I checked from within docker and the rslib\ftl\anki-core-i18n\core was there
the issue on rslib-bridge\build.rs is with this code
https://github.com/david-allison-1/Anki-Android-Backend/blob/6c16f621080382923f287dff089e86a235e8026b/rslib-bridge/build.rs#L96-L100
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }'But I wrote some debug code before the faling code in build.rs to see if cross for example missed something up. But all files were there and noting was missing
println!("{:?}",Path::new("src/backend_proto.rs").canonicalize()?);
let pwd_out = Command::new("pwd").output().unwrap();
println!("pwd status: {}", pwd_out.status);
println!("stdout: {}", String::from_utf8_lossy(&pwd_out.stdout));
println!("stderr: {}", String::from_utf8_lossy(&pwd_out.stderr));
for entry in WalkDir::new("./src") {
println!("DD {}", entry?.path().display());
} "/project/src/backend_proto.rs"
pwd status: exit code: 0
stdout: /project
stderr:
DD ./src
DD ./src/.gitignore
DD ./src/ankidroid.rs
DD ./src/backend_proto.rs
DD ./src/dbcommand.rs
DD ./src/lib.rs
DD ./src/sqlite.rsrustfmtTarekkMA
commented
3 years ago The issue was because I think cross didn't include cargo bin in docker. and the 2 build.rs needed to execute the rustfmt command.
After I removed the lines for formating. Cross worked with no issue for both Linux and windows.
I tried using Cross.toml to use mount the bin directory but no luck, it doesn't read it at all.
TarekkMA
commented
3 years ago ./gradlew clean assembleRelease is now working with no issues at alland it will be surprisingly easy to set up the environment in the future, I will upload a PR with the changes I made.
Only needed steps are:
Python 3.7 (macOS)
Without adding compilers/linkers or anything else
The only trade off we did is not using rustfmt, but I guess it can be fixed.
Figured out the issue. The feature of assigning volumes is not yet published.
So I pulled cross from git directly, and it does work not for --out argument of cargo
cargo install cross --git https://github.com/rust-embedded/cross/The current implementation of cross does not allow the change of the volume name like src:dest it will be src:src. But this won't be an issue for --out argument of cargo
"-v" "/Users/tarekkma/Developer/Projects/Anki-Android-Backend/rsdroid-testing/assets:/Users/tarekkma/Developer/Projects/Anki-Android-Backend/rsdroid-testing/assets"I might be able to make a PR to cross to enable selecting the mounting destination, would be a cool first rust PR 😃 Edit: https://github.com/rust-embedded/cross/pull/540 Hope to be merged soon.
TarekkMA
commented
3 years ago I have added steps to get the environment ready. Please let me know if we need to change anything.
david-allison
commented
3 years ago I have added steps to get the environment ready. Please let me know if we need to change anything.
Looks great! Will need to get a VM out and test that I can do this.
rsdroid.gradle to the fileTarekkMA
commented
3 years ago BackendProto.Backend is about 100K lines of code and the IDE seems to ignore it and not index it. Will that be a problem?
Edit: adding this line to the IDE config should fix this issue, but I don't know if AnkiDroid devs will also have to edit this value or not.
idea.max.intellisense.filesize=999999
BackendProto.Backendis about 100K lines of code and the IDE seems to ignore it and not index it. Will that be a problem?
It's in /build/generated anyway, there's very little reason to view the file.
RustBackendImpl is the one which may be viewed, and that's only 1.4k
Consider a
doctor.shscript