search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.13k stars 4.7k forks source link

Android Build Needs To Be Updated - Doc And Termex #66562

Open vindicatorr opened 2 years ago

vindicatorr commented 2 years ago

Description

The documentation and script for cross-compiling Android seems to be out of date.

Reproduction Steps

$ NDK_DIR="/23.1.7779620/" \ __lldb_Dir="/23.1.7779620/toolchains/llvm" \ PATH="/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/python3/bin/python3.9:$PATH" \ ./eng/common/cross/build-android-rootfs.sh arm64 31

Expected behavior

Build on Arch Linux

Actual behavior

Target API level: 31 Target architecture: arm64 NDK location: /23.1.7779620/ Target Toolchain location: /runtime/.tools/android-rootfs/android-ndk-r21 Download dependencies... cp: cannot stat '/runtime/.tools/android-rootfs/tmp/arm64//data/data/com.termux/files/usr/*': No such file or directory

Regression?

No response

Known Workarounds

No response

Configuration

x64 Arch Linux

Other information

https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/docs/workflow/building/coreclr/android.md?plain=1#L17 cross/build-android-rootfs.sh

https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 Is this not part of ndk (23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/bin/lldb)? If this is the same kind of lldb, then I think it should just reference this if NDK_DIR is set.

https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L110 https://termux.net/dists/stable/main/binary-aarch64/Packages results in {"code":"PAGE_NOT_FOUND","message":"Page not found"} If the only packages that are needed are:

__AndroidPackages="libicu"
__AndroidPackages+=" libandroid-glob"
__AndroidPackages+=" liblzma"
__AndroidPackages+=" krb5"
__AndroidPackages+=" openssl"

... is there any reason why the sources can't be downloaded and just built along with the rest of it? What does Termex end up offering instead?

https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L115 With dpkg, I'm guessing the script isn't meant for Arch Linux users. It would be good to have a "check" at the start of the script to determine if it should even continue.

EDIT0: https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 I also commented out this line since I set the environment variable for it (IF it's supposed to work like that). I'm now going to go about manually doing the sysroot stuff, and then run the build.sh that the script shows at the bottom.

EDIT1: https://github.com/termux/termux-packages/wiki/Package-Management So maybe https://packages.termux.org/apt/termux-main/dists/stable/main/binary-aarch64/Packages now?

EDIT2: I'm seeing https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L17 seem to counter with https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L58 cross vs .tools In addition to https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L8 where it looks like it should be echo . //Note the space.

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 2 years ago

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details
### Description The documentation and script for cross-compiling Android seems to be out of date. ### Reproduction Steps $ NDK_DIR="/23.1.7779620/" \\ __lldb_Dir="/23.1.7779620/toolchains/llvm" \\ PATH="/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/python3/bin/python3.9:$PATH" \\ ./eng/common/cross/build-android-rootfs.sh arm64 31 ### Expected behavior Build on Arch Linux ### Actual behavior Target API level: 31 Target architecture: arm64 NDK location: /23.1.7779620/ Target Toolchain location: /runtime/.tools/android-rootfs/android-ndk-r21 Download dependencies... cp: cannot stat '/runtime/.tools/android-rootfs/tmp/arm64//data/data/com.termux/files/usr/*': No such file or directory ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration x64 Arch Linux ### Other information https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/docs/workflow/building/coreclr/android.md?plain=1#L17 `cross/build-android-rootfs.sh` https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 Is this not part of ndk (23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/bin/lldb)? If this is the same kind of lldb, then I think it should just reference this if `NDK_DIR` is set. https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L110 `https://termux.net/dists/stable/main/binary-aarch64/Packages` results in `{"code":"PAGE_NOT_FOUND","message":"Page not found"}` If the only packages that are needed are: ``` __AndroidPackages="libicu" __AndroidPackages+=" libandroid-glob" __AndroidPackages+=" liblzma" __AndroidPackages+=" krb5" __AndroidPackages+=" openssl" ``` ... is there any reason why the sources can't be downloaded and just built along with the rest of it? What does Termex end up offering instead? https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L115 With `dpkg`, I'm guessing the script isn't meant for Arch Linux users. It would be good to have a "check" at the start of the script to determine if it should even continue. EDIT0: https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 I also commented out this line since I set the environment variable for it (IF it's supposed to work like that). I'm now going to go about manually doing the `sysroot` stuff, and then run the `build.sh` that the script shows at the bottom. EDIT1: https://github.com/termux/termux-packages/wiki/Package-Management So maybe `https://packages.termux.org/apt/termux-main/dists/stable/main/binary-aarch64/Packages` now? EDIT2: I'm seeing https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L17 seem to counter with https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L58 `cross` vs `.tools` In addition to https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L8 where it looks like it should be `echo .` //Note the space.
Author: vindicatorr
Assignees: -
Labels: `area-Infrastructure-coreclr`, `untriaged`
Milestone: -
janvorli commented 2 years ago

@vindicatorr the Android rootfs generation and runtime build was a best effort thing at a time in the past. It is neither maintained nor supported in any way. The android package URLs has been changing a lot in the past, so it keeps breaking from time to time. But fixes from the community are welcome.

Also, in general, the script is meant to be used on Ubuntu or Debian only.

ghost commented 2 years ago

Tagging subscribers to this area: @directhex See info in area-owners.md if you want to be subscribed.

Issue Details
### Description The documentation and script for cross-compiling Android seems to be out of date. ### Reproduction Steps $ NDK_DIR="/23.1.7779620/" \\ __lldb_Dir="/23.1.7779620/toolchains/llvm" \\ PATH="/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/python3/bin/python3.9:$PATH" \\ ./eng/common/cross/build-android-rootfs.sh arm64 31 ### Expected behavior Build on Arch Linux ### Actual behavior Target API level: 31 Target architecture: arm64 NDK location: /23.1.7779620/ Target Toolchain location: /runtime/.tools/android-rootfs/android-ndk-r21 Download dependencies... cp: cannot stat '/runtime/.tools/android-rootfs/tmp/arm64//data/data/com.termux/files/usr/*': No such file or directory ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration x64 Arch Linux ### Other information https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/docs/workflow/building/coreclr/android.md?plain=1#L17 `cross/build-android-rootfs.sh` https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 Is this not part of ndk (23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/bin/lldb)? If this is the same kind of lldb, then I think it should just reference this if `NDK_DIR` is set. https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L110 `https://termux.net/dists/stable/main/binary-aarch64/Packages` results in `{"code":"PAGE_NOT_FOUND","message":"Page not found"}` If the only packages that are needed are: ``` __AndroidPackages="libicu" __AndroidPackages+=" libandroid-glob" __AndroidPackages+=" liblzma" __AndroidPackages+=" krb5" __AndroidPackages+=" openssl" ``` ... is there any reason why the sources can't be downloaded and just built along with the rest of it? What does Termex end up offering instead? https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L115 With `dpkg`, I'm guessing the script isn't meant for Arch Linux users. It would be good to have a "check" at the start of the script to determine if it should even continue. EDIT0: https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 I also commented out this line since I set the environment variable for it (IF it's supposed to work like that). I'm now going to go about manually doing the `sysroot` stuff, and then run the `build.sh` that the script shows at the bottom. EDIT1: https://github.com/termux/termux-packages/wiki/Package-Management So maybe `https://packages.termux.org/apt/termux-main/dists/stable/main/binary-aarch64/Packages` now? EDIT2: I'm seeing https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L17 seem to counter with https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L58 `cross` vs `.tools` In addition to https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L8 where it looks like it should be `echo .` //Note the space.
Author: vindicatorr
Assignees: -
Labels: `untriaged`, `area-Infrastructure-mono`
Milestone: -
directhex commented 2 years ago

Okay, this is an older community effort to produce a CoreCLR JIT on Android. I think @am11 may have views on it, but I don't know much about it.

janvorli commented 2 years ago

@directhex this was not just JIT, but the whole runtime running on Android under Termux. As I've mentioned above, it was brought to functional state for both arm and arm64 long time ago (long time before mono was in the runtime repo) by several community members with my help. But it has never been officially supported. Things mostly just worked due to the fact that the Termux basically mostly behaves like a regular Linux. The problem with the Termux has always been that the native libraries package URLs format / subpath change occasionally. So the rootfs building script eng/common/cross/build-android-rootfs.sh needs to be updated each time someone wants to build the rootfs and runtime on top of it.

am11 commented 2 years ago

The setup in build-android-rootfs.sh is indeed something which can be improved and incorporated in unified build-rootfs.sh. This setup goes hand-in-hand with cmake toolschain file (eng/common/cross/toolchain.cmake) with rest of our cross build matrix (./build.sh -cross -os ...). When it was last updated, I was able to build clr+mono+host+libs subsets with it. Things have changed a lot since, as @janvorli mentioned, termux URLs changed as well as the fact that we now explicitly set -p:CrossBuild=false for Android and disregard -cross argument because of the competing (manual) NDK setup used by mono.

It would be nice to consolidate these two parallel setups for Android. My personal preference is to update build-rootfs.sh and use -cross with Android, so it is easier to maintain (as we already have multiple platforms on that plan, including linux-bioinic). I opened https://github.com/dotnet/runtime/issues/56622 for discussion.

ghost commented 2 years ago

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details
### Description The documentation and script for cross-compiling Android seems to be out of date. ### Reproduction Steps $ NDK_DIR="/23.1.7779620/" \\ __lldb_Dir="/23.1.7779620/toolchains/llvm" \\ PATH="/23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/python3/bin/python3.9:$PATH" \\ ./eng/common/cross/build-android-rootfs.sh arm64 31 ### Expected behavior Build on Arch Linux ### Actual behavior Target API level: 31 Target architecture: arm64 NDK location: /23.1.7779620/ Target Toolchain location: /runtime/.tools/android-rootfs/android-ndk-r21 Download dependencies... cp: cannot stat '/runtime/.tools/android-rootfs/tmp/arm64//data/data/com.termux/files/usr/*': No such file or directory ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration x64 Arch Linux ### Other information https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/docs/workflow/building/coreclr/android.md?plain=1#L17 `cross/build-android-rootfs.sh` https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 Is this not part of ndk (23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/bin/lldb)? If this is the same kind of lldb, then I think it should just reference this if `NDK_DIR` is set. https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L110 `https://termux.net/dists/stable/main/binary-aarch64/Packages` results in `{"code":"PAGE_NOT_FOUND","message":"Page not found"}` If the only packages that are needed are: ``` __AndroidPackages="libicu" __AndroidPackages+=" libandroid-glob" __AndroidPackages+=" liblzma" __AndroidPackages+=" krb5" __AndroidPackages+=" openssl" ``` ... is there any reason why the sources can't be downloaded and just built along with the rest of it? What does Termex end up offering instead? https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L115 With `dpkg`, I'm guessing the script isn't meant for Arch Linux users. It would be good to have a "check" at the start of the script to determine if it should even continue. EDIT0: https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L68 I also commented out this line since I set the environment variable for it (IF it's supposed to work like that). I'm now going to go about manually doing the `sysroot` stuff, and then run the `build.sh` that the script shows at the bottom. EDIT1: https://github.com/termux/termux-packages/wiki/Package-Management So maybe `https://packages.termux.org/apt/termux-main/dists/stable/main/binary-aarch64/Packages` now? EDIT2: I'm seeing https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L17 seem to counter with https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L58 `cross` vs `.tools` In addition to https://github.com/dotnet/runtime/blob/51b90cc60b8528c77829ef18481b0f58db812776/eng/common/cross/build-android-rootfs.sh#L8 where it looks like it should be `echo .` //Note the space.
Author: vindicatorr
Assignees: -
Labels: `area-Infrastructure-coreclr`, `area-Infrastructure-mono`
Milestone: 7.0.0