search

open-quantum-safe / liboqs

C library for prototyping and experimenting with quantum-resistant cryptography
https://openquantumsafe.org/
Other
1.92k stars 469 forks source link

Add options for iOS and Android cross-compilation #594

Closed nicholasfulton closed 10 months ago

nicholasfulton commented 4 years ago

I'm attempting to build liboqs for both iOS and Android for a Xamarin.Forms project, and haven't been able to manipulate the included build and configuration files to make that work. If it would be possible, add configuration or documentation to help with cross-compiling the library for iOS and Android, ideally targeting multiple processor configurations. Thanks!

dstebila commented 4 years ago

Hi Nicholas, a while ago someone created a cross-compilation script for Android (https://github.com/open-quantum-safe/liboqs/blob/master/configure-android) but it is not something we actively maintain since we don't have anyone in the project who uses it nor do we have any continuous integration testing set up for Android. We don't have much expertise in cross-compilation. If you are able to get it working, please let us know what you've done so we can incorporate it.

nicholasfulton commented 4 years ago

I believe I've made good progress with developing a build script for Android. However, developing a more general solution runs into an issue with building using Clang. It seems that the build scripts specifically require GCC and disallow Clang. That's problematic because Clang is necessary to cross-compile for iOS and Android, as GCC doesn't include toolchains for those platforms. Is there a reason that liboqs requires GCC? Otherwise, I'll work on a more elegant way to allow building with Clang as well.

EDIT: It appears the existing "configure-android" script is built for a previous version of the Android ndk which supported GCC, and will no longer work because of that.

dstebila commented 4 years ago

Hi @nicholasfulton, in general liboqs should compile just fine with clang -- our macOS continuous integration builds use the default compiler on macOS, which is clang.

The now-unmaintained configure-android script does explicitly refer to gcc, and that was just because that was what the person putting that script together was able to get working, not intentionally made to restrict use to gcc-only.

zugaldia commented 4 years ago

I've got basic Java bindings working, including Android, on https://github.com/zugaldia/javacpp-presets. It uses the JavaCPP infrastructure to build the bindings. A few notes for those willing to give it a try:

It's interesting that, while the Linux bindings build rather smoothly, Android (via NDK) throws multiple compilations warnings and errors for some header files. If someone more familiar with the project wants to check out the setup, this is the main script to review.

xvzcf commented 4 years ago

We are in the process of moving the build system to CMake and dropping support for autotools (PR https://github.com/open-quantum-safe/liboqs/pull/556).

@zugaldia would it be possible to update cppbuild.sh to use CMake?

zugaldia commented 4 years ago

@xvzcf yup - that should not be a problem, in fact other JavaCPP Presents projects (example) use CMake. Feel free to ping me here once that switch is ready to be tested and I'll happily update cppbuild.sh.

dstebila commented 4 years ago

Should be okay to start looking at the iOS and Android cross-compilation again since our change to CMake has landed. FYI I've started a PR for cross-compilation on Raspberry Pi #618.

nicholasfulton commented 4 years ago

I believe I have a fully functioning Android build script, which I've tested on the Windows Subsystem for Linux with Ubuntu running. I was able to compile for all major ABI's with no issue. In the 'add-cpu-extensions-flag.cmake' file, I had to comment out the 'try_run' command in order to compile for Android x86 architectures. It seems that it was trying to run tests without awareness that the build machine and the host machine are completely different. My app is developed in Xamarin Forms, so I'm working through some issues with the shared library not being found. I'll update when I know whether the libraries produced by the script actually function. Also, once I finish that, I'll work on the iOS build. CMake definitely made the Android compilation way more straightforward, and it seems it'll do the same for iOS.

dstebila commented 4 years ago

Thanks for the update @nicholasfulton!

Hatzen commented 4 years ago

What will the next steps look like? Will this issue be handled as is or are there plans to make a demo App or at some documentation for creating an aar (android library)? I m asking as i am looking for using this awesome library collection for my android app, but i am not very familiar with using native libs. So i would like to avoid spending time with getting into it :)
The timetable looks like 17th April will be a release?

nicholasfulton commented 4 years ago

At the moment, I'm working on setting up a CI/CD container for the Android compilation, and my next step is to develop a working build script for iOS (the quarantine has messed with my access to a Mac which was essential to that). The Android build script should be fully function at this time, but it only works on Linux (I used Ubuntu Windows Subsystem to build and test). To use it you'll need to run './build-android.sh --help' (from the scripts directory) and input the required options. The root directory of the Android NDK is required and best practice is to build for all 4 supported Android ABI's, which can be any of "armeabi-v7a," "arm64-v8a," "x86," or "x86_64." Optionally, you can specify an sdk version as well. If you have any further questions about how the compilation works, feel free to ask (once I get CI/CD working, I'll definitely need to make it a priority to include the build instructions in README).

Hatzen commented 4 years ago

@nicholasfulton thanks for your fast response. The last days i tried it a few times but could not get it working. Probably it is my bad but i cant get any useful informations from the logs or readme files. I tried around using different versions and installing different packages without any effect. Do you have any thoughts what i am doing wrong?

Detailed setup of ubuntu subsystem and error logs Using a clean installed ubuntu 18.04 LTS on Windows subsystem ```shell pwd >/home/hatzen sudo su apt-get update mkdir liboqs cd liboqs git clone https://github.com/open-quantum-safe/liboqs.git mkdir ndk cd ndk wget https://dl.google.com/android/repository/android-ndk-r21-linux-x86_64.zip apt-get install unzip unzip android-ndk-r21-linux-x86_64.zip apt-get install -y build-essential cmake libssl-dev cd /home/hatzen/liboqs/liboqs/scripts bash build-android.sh /home/hatzen/liboqs/ndk/android-ndk-r21 Valid directory for NDK at /home/hatzen/liboqs/ndk/android-ndk-r21 Compiling for ABI armeabi-v7a Compiling for SDK 21 Cleaning up previous build Building in directory build -- The ASM compiler identification is Clang -- Found assembler: /home/hatzen/liboqs/ndk/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- Check for working C compiler: /home/hatzen/liboqs/ndk/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- Check for working C compiler: /home/hatzen/liboqs/ndk/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - found -- Found Threads: TRUE CMake Error at .CMake/gcc_clang_intrinsics.cmake:6 (message): .CMake/detect_gcc_clang_intrinsics.c returned exit code: Call Stack (most recent call first): .CMake/compiler_opts.cmake:13 (include) CMakeLists.txt:40 (include) -- Configuring incomplete, errors occurred! See also "/home/hatzen/liboqs/liboqs/build/CMakeFiles/CMakeOutput.log". ``` A Copy of /home/hatzen/liboqs/liboqs/build/CMakeFiles/CMakeOutput.log [CMakeOutput.log](https://github.com/open-quantum-safe/liboqs/files/4497603/CMakeOutput.log) Maybe other useful version informations ``` cmake --version cmake version 3.10.2 openssl version OpenSSL 1.1.1 11 Sep 2018 gcc --version gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 ```
xvzcf commented 4 years ago

The error is due to this commit introduced by me. I can fix it in an upcoming PR, or @nicholasfulton perhaps you could fix it as part of your work on setting up a CI workflow for Android?

greebie commented 4 years ago

I have successfully built portions of the C code from liboqs into a Flutter/Dart project (which is currently private, but will be open for viewing once I have a reasonable MVP). More than happy to help prevent someone else from pulling their hair out based on my learning.

Some challenges I found for cross-compilation via the Flutter framework:

nicholasfulton commented 4 years ago

I'll continue researching CI/CD integration for Android and iOS. I started but got sidetracked and quit working on it for a while. It does seem like the biggest obstacle is some algorithms' issues on ARM. I'll look into the errors, because it'd be ideal to be able to build the full set of algorithms for ARM instead of needing to skip specific ones.

dstebila commented 4 years ago

Just checking in to see if there are any updates to this work.

nicholasfulton commented 4 years ago

I think I'm pretty close to a Docker container for CI/CD on Android, working with the configuration for some other build profiles/targets.

dstebila commented 4 years ago

Thanks for the update Nicholas. Looking forward to it!

nicholasfulton commented 3 years ago

Sorry for the delay. I created a draft pull request (#875) which I believe should test building the library. I'm not sure of format or whether it works correctly with the CircleCI workflows. I will be looking into it more, but I published the draft PR for potential feedback.

SWilson4 commented 10 months ago

Given that we now have CI builds for both Android and iOS, I think this issue can be safely closed. @dstebila @baentsch Feel free to reopen if you have objections.

MewanCdot commented 1 month ago

I have successfully built portions of the C code from liboqs into a Flutter/Dart project (which is currently private, but will be open for viewing once I have a reasonable MVP). More than happy to help prevent someone else from pulling their hair out based on my learning.

Some challenges I found for cross-compilation via the Flutter framework:

* iOS requires that added C or C++ code be signed using XCode.

* Documentation for inclusion of .so and .a binaries is slim and I have not been able to get it to work.

* Documentation about how Android and iOS handle CMakeLists.txt in subdirectories is not clear.

* Compiling the source code is possible, but it requires some sideways hacking which would be hard to maintain for a project where new algos appear, disappear and re-appear over time.

Hi @greebie , I am working on a flutter app of the strongswan vpn client with liboqs support, liboqs support for the android section is working but I am lost on how to start with the iOS section. Do you have a reference i can follow??

professor-greebie commented 1 month ago

@MewanCdot Sorry I do not have too much in terms of references here. I did get the code to work in iOS but it required that I create a facade over liboqs and compile it in the phone.

This is probably not too helpful, but I did get it to work in Android: https://github.com/greebie/liboqs_flutter. You can check through my commits and see if there's anything helpful there. Since it was just a personal project, it's not fantastic in terms of documentation and design. Spaghetti is delicious though, isn't it? :)

This is also probably not too helpful, but I relied a lot on this to get things running: https://docs.flutter.dev/platform-integration/ios/c-interop. If you encounter unit testing issues, do consider this Stack Overflow question: https://stackoverflow.com/questions/66641396/testing-library-functions-in-a-flutter-package-with-native-functions-using-dart/66722572#66722572 (and upvote please if you find it helpful).

MewanCdot commented 1 month ago

@professor-greebie Any help is deeply appreciated, thank you good sir.