bryanpkc
commented
2 years ago What architecture do you want to cross-compile to?
Open meow464 opened 2 years ago
bryanpkc
commented
2 years ago What architecture do you want to cross-compile to?
meow464
commented
2 years ago AArch64 (iOS)
bryanpkc
commented
2 years ago The current code base does not support cross-compilation of the frontend and the libraries, however we have implemented such a mechanism downstream. I'll see if I can clean things up and submit a PR with the change.
meow464
commented
2 years ago That would be amazing!
kaadam
commented
2 years ago @bryanpkc I have a POC for enable cross-compilation in CMakefiles what I did some weeks ago [1], if you and the community are interested in, I can upload that. [1] https://github.com/kaadam/flang/commit/a89ecd1aa5c70ac6b1eb9ad36c1d2e3f5b420379
meow464
commented
2 years ago Thanks, I will take a look at that too!
meow464
commented
2 years ago This is what I'm doing and the error I'm getting.
As you can see CMake keeps trying to build for the triple x86_64-apple-ios (the iphone simulator on a x86 host)
EDIT: To clarify, that's not on kaadam's fork.
meow464
commented
2 years ago I managed to make libpgmath target arch64 by commenting out two lines in flang/runtime/libpgmath/CMakeLists.txt:
# set(LIBPGMATH_SYSTEM_NAME "${CMAKE_SYSTEM_NAME}")
# set(LIBPGMATH_SYSTEM_PROCESSOR "${CMAKE_SYSTEM_PROCESSOR}")Now I'm getting:
[ 0%] Building C object lib/aarch64/tanh/CMakeFiles/aarch64_tanh.dir/gstanh4.c.o
clang-10: error: the clang compiler does not support '-mcpu=aarch64'I've tried setting compiler flags with -DCMAKE_{C,CXX,Fortran}_FLAGS="-target arm64-apple-ios -miphoneos-version-min=9.0 -mcpu=generic" to no avail.
Any help is appreciated.
meow464
commented
2 years ago I fixed the problem above but now there are more problems. Please let me know if something is not clear.
The hypothesis:
mth_128mask.cpp is generated from runtime/libpgmath/tools/mth_mask.awk. The top of mth_128mask.cpp looks like this:
#ifdef __cplusplus
extern "C" {
#endifstatic const vrs4_t Csp1_4={1.0, 1.0, 1.0, 1.0}; static const vrd2_t Cdp1_2={1.0, 1.0}; static const vrs8_t Csp1_8={1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0}; static const vrd4_t Cdp1_4={1.0, 1.0, 1.0, 1.0}; static const vrs16_t Csp1_16={1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0}; static const vrd8_t Cdp1_8={1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0};
}
The top of [`mth_intrinsics.h`](https://github.com/flang-compiler/flang/blob/flang_20211221/runtime/libpgmath/lib/common/mth_intrinsics.h#L8) looks like this:
2. Since `mth_128mask.cpp` is a cpp file, `#ifdef __cplusplus` is also triggered in the included `mth_intrinsics.h` file. We end up including a cpp file `cstdint` inside an `extern C`. This shouldn't happen, of course.
So far so good. If I move `#include "mth_intrinsics.h"` to outside the `extern C` block everything works. But a question arises: Why am I the first one hitting this problem?
Only in the ARM pathway `mth_128mask.cpp` is generated, for x86 and power it's `mth_128mask.c`, meaning `#ifdef __cplusplus` is never triggered. This explains why I didn't find this on MacOS.
But why didn't anyone find it on any other ARM platform? It seems this behaviour should be triggered on any ARM platform. I don't have an ARM machine so I can't test.
The relevant part of [`runtime/libpgmath/lib/common/CMakeLists.txt`](https://github.com/flang-compiler/flang/blob/flang_20211221/runtime/libpgmath/lib/common/CMakeLists.txt#L419) looks like this:elseif(${LIBPGMATH_SYSTEM_PROCESSOR} MATCHES "aarch64")
set(TARGET_NAME "mth_128mask") if(${LIBPGMATH_SYSTEM_NAME} MATCHES "Windows") add_custom_command(OUTPUT ${TARGET_NAME}.cpp DEPENDS ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk PRE_BUILD COMMAND awk -v MAX_VREG_SIZE=128 -v TARGET=ARM64 -v TARGET_OS=WIN -f ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk > ${TARGET_NAME}.cpp) else() add_custom_command(OUTPUT ${TARGET_NAME}.cpp DEPENDS ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk PRE_BUILD COMMAND awk -v MAX_VREG_SIZE=128 -v TARGET=ARM64 -f ${LIBPGMATH_TOOLS_DIR}/mth_mask.awk > ${TARGET_NAME}.cpp) endif() add_custom_target(${TARGET_NAME} ALL DEPENDS "${TARGET_NAME}.cpp") libmath_add_object_library("${TARGET_NAME}.cpp" "${FLAGS}" "${DEFINITIONS}" "${TARGET_NAME}_build") add_dependencies("${TARGET_NAME}_build" ${TARGET_NAME})
set(TARGET_NAME "mth_128generic") add_custom_command(OUTPUT ${TARGET_NAME}.c DEPENDS ${LIBPGMATH_TOOLS_DIR}/mth_generic_frp.awk PRE_BUILD COMMAND awk -v MAX_VREG_SIZE=128 -v TARGET=ARM64 -f ${LIBPGMATH_TOOLS_DIR}/mth_generic_frp.awk > ${TARGET_NAME}.c) add_custom_target(${TARGET_NAME} ALL DEPENDS "${TARGET_NAME}.c") libmath_add_object_library("${TARGET_NAME}.c" "${FLAGS}" "${DEFINITIONS}" "${TARGET_NAME}_build") add_dependencies("${TARGET_NAME}_build" ${TARGET_NAME})
Problem above is "fixed" but now I get:
Here is the relevant #ifdef in fltfenv.c
#else /* #if (defined(TARGET_X8664) || defined(TARGET_X86) || defined(X86)) */
/*
* aarch64 and POWER (not X86-64).
*
* Without loss of generality, use libc's implemenations of floating point
* control/status get/set operations.
*/
#include <fenv.h>But the functions in the error: feenableexcept, fedisableexcept and fegetexcept are GNU extensions and obviously not available on iOS or Windows.
This was taken into account for windows but the #if defined(TARGET_WIN_ARM64) assumes the #else case is always GNU and tries to the libc implementation.
Fortunately FreeBSD implemennts all thoses three functions. freebsd/aarch64/fenv.h freebsd/aarch64/fenv.c
My questions are:
fltfenv.c becoming a bit too platform dependent for something in the common directory? It already implements the x86 version itself.meow464
commented
2 years ago I implemented the missing functions above myself.
When trying to build libpgmath for iOS I'm getting a bunch of warnings that I will fix afterwards and an error.
__io_set_errno is declared in runtime/include/stdioInterf.h and defined in runtime/flangrti/iostdinit.c. It used to be included indirectly by pgstdinit.h but this file has been deleted.
The file fpcvt.c where the error is raised is included for all architectures but but only causing problems here. I'm not able to figure out why.
meow464
commented
2 years ago For now I simply declared the prototype for the function above.
The next error is again a bit weird. It seems it should be triggered on any arm64 platform.
The error is on the file runtime/libpmath/lib/common/sincos/gsincos.cpp. As you can see the function SINCOS(VFLOAT, FLOAT, FLOAT) is missing and nowhere to be found in the code base. But gsincos.cpp does have the suspiciously familiar prototype extern "C" void sincos(double, double*, double*) throw();.
It turns out gsincos.cpp is only used for arm64. x86 and power use the very similar file gsincos.c which does have the same call to SINCOS(VFLOAT, FLOAT, FLOAT) except the familiar prototype from above is extern void SINCOS(FLOAT, FLOAT *, FLOAT *); (on the exact same line number too).
kiranchandramohan
commented
2 years ago Good to see that you are making progress here. Is sincos expected to come from libm?
meow464
commented
2 years ago sincos should be available on math.h, yes. But to be honest I don't fully understand all the code in runtime/libpggmath/lib/common/sincos/ yet.
As for the overall status: I can build libpgmath but: 1. some targets are still missing the arm64 compiler flags and thus built for x86 (the host) and; 2. there are some warnings about long double to double implicit conversion on arm64.
d-parks
commented
2 years ago WRT SINCOS not being defined: The error is on the file runtime/libpmath/lib/common/sincos/gsincos.cpp. As you can see the function SINCOS(VFLOAT, FLOAT, FLOAT) is missing and nowhere to be found in the code base
SINCOS is a CPP object macro that is defined in CMakeLists.txt.
meow464
commented
2 years ago I completely missed that, thanks! I only needed to add the prototype for sincosf(float, float*, float*).
meow464
commented
2 years ago I'm getting a bunch of implicit long double -> double conversion warnings:
It seems that on apple long double and double are the same type even if arm64 theoretically supports float128. Ref1 Ref2. For now I simply disabled TARGET_SUUPPORTS_QUADFP, please let me know if this is not the correct course of action.
meow464
commented
2 years ago I'm still getting quite a few warnings about user defined types in C-linkage functions.
Looking at their code I'm not sure if this is really a problem. They are all structs with operator overloads and constructors, which in principles C code wouldn't even know how to call.
For now I will just move on to flang itself and come back later.
d-parks
commented
2 years ago Hi. I'm away from my desk reading this on my phone - so I might be missing something.
But if I'm following you correctly, you're trying to build for aarch64. The errors you're seeing are from an x86_64 target.
meow464
commented
2 years ago Thanks, I will check if there is any if(aarch64) missing.
The build process does finish and link successfully, what makes you think those are x86 errors?
bryanpkc
commented
2 years ago But the functions in the error: feenableexcept, fedisableexcept and fegetexcept are GNU extensions and obviously not available on iOS or Windows.
We implemented our own AArch64-specific routines in fltfenv.c as well.
__io_set_errno is declared in runtime/include/stdioInterf.h and defined in runtime/flangrti/iostdinit.c. It used to be included indirectly by pgstdinit.h but this file has been deleted.
I remember that I ran into this problem as well. IIRC the use of __io_set_errno can be deleted and replaced with calling the libc version of the function.
michaeldupton
commented
2 years ago is there a set of code I can test on an apple M1? I am hitting lots of the issues mentioned.
meow464
commented
2 years ago It's not working yet but I'm making progress. libpgmath already builds, I'm now tackling the flang runtime itself, which I expect to be easier. After that it should be usable on MacOS. For iOS I still need to automate the build of the Xcode framework and build arm64+x86 fat binaries.
meow464
commented
2 years ago Everything builds statically, I'm now trying to solve some linking problems.
The problem seems to be with -z,defs, this flag does exist on Linux and according to man ld: " Report unresolved symbol references from regular object files. This is done even if the linker is creating a non-symbolic shared library. This option is the inverse of -z undefs.".
But it does not exist on mac, the equivalient seems to be -no_weak_imports: Error if any symbols are weak imports (i.e. allowed to be unresolved (NULL) at runtime). Useful for config based projects that assume they are built and run on the same OS version.
I can't figure out where this flag is being set.
meow464
commented
2 years ago The problem does not happen when -DCMAKE_SYSTEM_NAME=Darwin and I still cannot find where those flags are coming from. This makes me think they were indeed auto inserted by CMake.
Now what would be a good way around this? Should I just set the system name to Darwin instead of iOS? Ideally we should be able to distinct between both. iOS for example doesn't support system() calls and building the compiler itself makes no sense.
kiranchandramohan
commented
2 years ago Now what would be a good way around this? Should I just set the system name to Darwin instead of iOS?
I guess for a local workaround that should be OK. Or are you asking whether this is acceptable in the flang project?
Anyway, great progress here.
meow464
commented
2 years ago I would like to make pull request, yes.
kiranchandramohan
commented
2 years ago The llvm-project cmake sets the System Name to Darwin in iOS, so I think it should be OK.
https://github.com/llvm/llvm-project/blob/db0631096e59c71b6f6b034306327b42d9184184/llvm/cmake/platforms/iOS.cmake#L3
meow464
commented
2 years ago I can see they also have SET(IOS True). Thanks very much!
meow464
commented
2 years ago We are 90% done, now there is only 90% to go. :tada: :tada: :tada:
All the libraries build and link statically and dynamically. But I still need to:
#ifdefs and hopefully it will just work on the M1 (this could be the second item of the list, the more people testing the better).A three years old issue was solved 20 days ago. Talk about luck. I will leave it here for reference.
kiranchandramohan
commented
2 years ago @meow464 This is great. M1 support will be super. Thanks for pushing forward with this.
meow464
commented
2 years ago OpenBLAS compiles and links successfully! But I still have to test it on a real device.
meow464
commented
2 years ago I'm trying to make OpenBLAS work with numpy on iOS. It's not yet ready for a pull request but but I will leave the instructions here for now in case someone else needs to test it.
build-llvm.sh and build-flang.sh in sequence. Don't forget to add -jN to make (N is how many cores you want the build process to use). Everything will be installed under INSTALL_PREFIX=/usr/local/flang-iOS;flang -L/usr/local/flang-iOS/lib -isysroot $(xcrun --sdk iphoneos --show-sdk-path)Things to be aware of:
build-flang-iphoneos.shsetup.sh
INSTALL_PREFIX=/usr/local/flang-iOS
# Targets to build should be one of: X86 PowerPC AArch64
CMAKE_OPTIONS="-DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX \
-DLLVM_CONFIG=$INSTALL_PREFIX/bin/llvm-config \
-DCMAKE_CXX_COMPILER=$INSTALL_PREFIX/bin/clang++ \
-DCMAKE_C_COMPILER=$INSTALL_PREFIX/bin/clang \
-DCMAKE_Fortran_COMPILER=/usr/local/bin/flang \
-DCMAKE_Fortran_COMPILER_ID=Flang \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_TARGETS_TO_BUILD=X86;AArch64"build-llvm.sh
. setup.sh
if [[ ! -d classic-flang-llvm-project ]]; then
git clone -b release_100 https://github.com/flang-compiler/classic-flang-llvm-project.git
fi
cd classic-flang-llvm-project
mkdir -p build && cd build
cmake $CMAKE_OPTIONS -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
-DLLVM_ENABLE_CLASSIC_FLANG=ON -DLLVM_ENABLE_PROJECTS="clang;openmp" ../llvm
make
sudo make installbuild-flang-iphoneos.sh
. setup.sh
# use the branches ios or ios_bitcode
if [[ ! -d flang ]]; then
git clone -b ios https://github.com/meow464/flang.git
fi
CMAKE_OPTIONS_COMMON=$CMAKE_OPTIONS" -DLLVM_FLANG_CPU_TARGET=apple-a9 \
-DCMAKE_SYSTEM_PROCESSOR=arm64 \
-DCMAKE_CROSSCOMPILING=TRUE \
-DCMAKE_SYSTEM_NAME=Darwin \
-DIOS=TRUE \
-DCMAKE_OSX_SYSROOT=$(xcrun --sdk iphoneos --show-sdk-path) \
-DCMAKE_OSX_DEPLOYMENT_TARGET=9.0 \
-DCMAKE_OSX_ARCHITECTURES=arm64"
CMAKE_OPTIONS_FLANG=$CMAKE_OPTIONS_COMMON" -DCMAKE_MACOSX_BUNDLE=OFF"
(cd flang/runtime/libpgmath
mkdir -p build_iphoneos && cd build_iphoneos
cmake $CMAKE_OPTIONS_COMMON ..
make
sudo make install)
cd flang
mkdir -p build_iphoneos && cd build_iphoneos
cmake $CMAKE_OPTIONS_FLANG -DFLANG_LLVM_EXTENSIONS=ON ..
make
sudo make installProblem: I'm running into a EXC_BAD_ACCESS with this call stack (forgive the formatting but the Xcode debugger UI is not copy/paste friendly):
lapacke_dgesdd.c: info = LAPACKE_dgesdd_work( matrix_layout, jobz, m, n, a, lda, s, u, ldu, vt, ldvt, &work_query, lwork, iwork );
lapacke_dgesdd_work.c: LAPACK_dgesdd( &jobz, &m, &n, a, &lda_t, s, u, &ldu_t, vt, &ldvt_t, work, &lwork, iwork, &info );
ilaenv.f -> INTEGER FUNCTION ILAENV( ISPEC, NAME, OPTS, N1, N2, N3, N4 ): SUBNAM = NAME
fntcharsup.c -> void Ftn_str_copy_klen(int n, char *to, int64_t to_len, ...): *to_p++ = *from;
Where Ftn_str_copy_klen is a variable args function and seems to be the implementation of the assignment SUBNAME=NAME.
Going step by step on lldb I can see that from is NULL, in fact it never receives a value here from = va_arg(ap, char *);.
All of this led me to this commit. I was simply using the flang, flang1 and flang2 build for x86 but it seems some code generation code wasn't being triggered?
The files in the commit have changed quite a bit but there is a very similar #ifdef TARGET_LLVM_ARM64. I will try building flang, flang1 and flang2 for x86 with TARGET_LLVM_ARM64 and see how it goes.
Please let me know if this makes sense. I'm missing something obvious or about to shoot myself on the foot?
meow464
commented
2 years ago It didn't work :( I still get a EXC_BAD_ACCESS on this line because *from is NULL.
I don't know how to solve this.
EDIT: Sometimes I get a SIGSEGV but I still don't know how to solve it.
meow464
commented
2 years ago Note: I copied the source of the function below.
If I put a breakpoint at the function declaration I can actually proceed through by going step by step, from is no longer NULL but now I get a EXC_BAD_ACCESS at memcpy(qq->str, from, (size_t)from_len);
The only thing I can think of is one thread stepping on each other's foot?
bryanpkc
commented
2 years ago What does the assembly code (i.e. compiled code for the Fortran function ILAENV) look like before the call to Ftn_str_copy_klen? Is it passing the expected number of arguments?
meow464
commented
2 years ago I'm knowledgeable enough to tell if this is correct. It doesn't look particularly wrong.
0x100a5d3cc <+132>: ldr x4, [sp, #0x68]
0x100a5d3d0 <+136>: mov w8, #0x1
0x100a5d3d4 <+140>: add x1, sp, #0x50 ; =0x50
0x100a5d3d8 <+144>: mov w0, #0x1
0x100a5d3dc <+148>: mov w2, #0x10
0x100a5d3e0 <+152>: str w8, [sp, #0x64]
0x100a5d3e4 <+156>: bl 0x1009dd2b8 ; f90_str_copy_klen at ftncharsup.c:286Inside f90_str_copy_klen, even after va_start(ap, to_len), ap remains \0.
(edited) Note: ap is a string on iOS
Note: This is not compiled with TARGET_LLVM_ARM64 as mentioned 3 comments above.
meow464
commented
2 years ago It also happens when:
-DTAREGT_LLVM_ARM64 -DTARGET_LLVM_ARM -DTARGET_ARM64 -DTARGET_ARM;With obj-c the function at least seems to receive the string parameter but the error still happens on the same line. I got a SIGBUS once. I guess I will keep trying to pinpoint the origin.
Anyone using flang on ARM by any chance?
bryanpkc
commented
2 years ago It seems that a proper port of commit 5d46b686663a6ae0835a87301cdc82db5aed626a for AArch64 Darwin iOS is necessary. Hopefully someone else with hardware access can help look into it.
meow464
commented
2 years ago (I didn't give up I just been busy. I got here, I'm fixing this)
The function can be called by C code compiled with clang-10. This makes me think llvm11 probably wouldn't be helpful. Some interesting things: The C code seems to use an unusual set of registers x8, x9, x10 (only x0-x7 are for argument passing?). The flang code does use x0-x7 but the code for f90_str_copy_klen still tries to manipulate x8, x9, x10. The code for f90_str_copy_klen is not included below.
It's way past midnight now but I will keep investigating.
meow464
commented
2 years ago The darwin aarch64 ABI differs a bit from the default ARM64. It's not implemented in LLVM yet but it is implemented in clang and rustc so I will take inspiration from them.
meow464
commented
2 years ago I'm back.
On the AAPCS arguments for variadic functions are passed on the registers (x0-x7) until they are full and only then added to the stack.
On Apple only the normal arguments are assigned to registers and all variadic arguments are put on the stack.
You can see on my comment above how flang is trying to put all parameters on registers, while clang correctly puts only the first 3 on registers.
The function responsible for deciding what to do with vargs seems to be ll_abi_classify_va_arg_dtype. On aarch64 it's implemented here.
As documented, I set *num_gp=0 and num_fp=0:
I also set the CMakeLists.txt to use the modified aarch/ll_abi.cpp.
But yet, the problem persists and the generated code looks the exact same: parameters on registers and nothing on the stack.
It really looks like I accidentally didn't swap openblas.a for the new version but I sure did. I deleted the old one, ran the test app without it to make sure I would get a missing symbol error and it was not hidden in some cache then added the new version and tried again.
All ideas are welcomed.
meow464
commented
2 years ago Hmm I can't find where ll_abi_classify_va_arg_dtype is being called from...
Is there cross compilation support? I only need the libraries, not the compiler. No matter what parameters I try it keeps building for the host architecture (x86_64).