Dependency on preexisting Flang toolchain to compile

[AndreiLux]

The current build system requires a preexisting Flang installation to exist, and since gfortran isn't compatible, the build fails before its able to make Flang itself to be able to use it.

Besides fixing this dependency, is it possible to provide precompiled binaries so that one can actually bootstrap a new compilation? I'm not aware of any AArch64 package readily available anywhere.

[steleman]

I know about the chicken-and-egg problem with Flang.

AFAIK Ubuntu disco provides binaries for flang for x86_64. Not for AArch64, though:

https://packages.ubuntu.com/search?searchon=sourcenames&keywords=flang https://packages.ubuntu.com/disco/flang-7 https://packages.ubuntu.com/disco/libflang-dev https://packages.ubuntu.com/disco/libflang0d-7

Maybe that would help bootstrapping flang with a cross-compile. Yes, it's cumbersome.

I will ask to see if we can provide pre-built Aarch64 binaries here. It's not a matter of whether we have them (we do), it's a matter of support implications for Marvell. And that decision is way above my pay grade.

[AndreiLux]

I remember the main Flang can't properly cross-compile things due to the hardcoded triple settings, it's an ongoing issue in the main repo and I've already tried that route first. Does the Marvell one fix this issue? I admittedly didn't try it yet with this tree. If it can, any specific build setting changes I need to make besides these defines?

-DLLVM_TARGETS_TO_BUILD=AArch64
-DLLVM_TARGET_ARCH=AArch64
-DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu 

[steleman]

I remember the main Flang can't properly cross-compile things due to the hardcoded triple settings, it's an ongoing issue in the main repo and I've already tried that route first. Does the Marvell one fix this issue?

No, I haven't fixed that brokenness yet. :-)

Please give me a few days to sort this out internally at Marvell. I have two possible solutions:

1. I have a "crippled" build of flang that compiles with gfortran. It's not suitable for a real-world production compiler, but it is suitable for obtaining a flang compiler that can then be used to bootstrap flang. You would build flang with gfortran and pass -DFLANG_GFORTRAN_BOOTSTRAP=1 on compile line. Actually that's how I bootstrapped flang the first time I built it on AArch64 a couple years ago. :-)
2. Pre-built AArch64 binaries. That might take a while to get sorted out internally at Marvell. So it seems to me that option #1 is the fastest.

I'm technically on vacation until Monday Jan 6th, so I might go radio silence until then. :-)

[AndreiLux]

Option 1) sounds great, I don't mind having to compile twice to get to the real thing from scratch.

Happy new year! I'll check back in after CES.

[steleman]

Actually, there's an even simpler solution to this:

Download the evaluation version of the ARM Allinea Studio compiler:

https://www.arm.com/products/development-tools/server-and-hpc/allinea-studio

It's based on LLVM-7, and it has flang in it. So it can be used to bootstrap flang.

[steleman]

Does this help:

https://developer.arm.com/docs/101169/latest/using-arm-licence-server

Option [1] is still on the table, but it's not a top priority on Marvell's list of top priorities.

So, i'll get to it when i can get to it, depending on whatever else is happening on my radar screen.

[AndreiLux]

Sorry for the extremely late response here, I finally managed to get back to this.

I managed to compile using Arm's compiler, and everything seems to work relatively well except for a few oddities on compiling a few SPEC programs:

[1/16] Building Fortran object 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90.o
FAILED: 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90.o 549.fotonik3d_r/power_mod.mod
/data/workfolder/llvm/install/bin/flang  -I../CFP2017/549.fotonik3d_r  -I../CFP2017/549.fotonik3d_r -J549.fotonik3d_r   -static -Ofast -fomit-frame-pointer -DNDEBUG -DSPEC_CPU -DSPEC -DSPEC_CPU_LITTLEENDIAN -DSPEC_AUTO_BYTEORDER=0x12345678 -DSPEC_WORDS_LITTLEENDIAN -DANDROID_LIB -DBIONIC_IOCTL_NO_SIGNEDNESS_OVERLOAD -Wno-macro-redefined -Wno-builtin-macro-redefined -mcpu=cortex-a76 -mtune=cortex-a76 -DSPEC_LINUX_AARCH64 -DSPEC_CPU_LINUX -DSPEC_CPU_LP64 -DSPEC_LINUX -DSPEC_LP64 -w -DSPEC_AUTO_SUPPRESS_OPENMP -c 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90-pp.f90 -o 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90.o
F90-S-0004-Old Module file /data/workfolder/llvm/install/bin/../include/iso_fortran_env.mod (../CFP2017/549.fotonik3d_r/power.F90: 44)
F90-F-0004-Recompile source file  (../CFP2017/549.fotonik3d_r/power.F90: 44)
F90/aarch64 Linux Flang - 1.5 2017-05-01: compilation aborted

Searching around a bit tells me I'm supposed to delete .mod files but since these are include header .mod files that doesn't seem to make sense.

Using the Arm Flang compiler works just fine and the mentioned .mod files are identical except for the dates. The issue is identical on 549.fotonik3d and 507.cactuBSSN_r .

[flang-cavium]

Hi Andrei,

Yup this is weird. I've never seen it when I ran SPEC. And I ran both SPEC2K6 and SPEC2K17 hundreds of times. My internal builds are based on the same exact sources as on Github.

One thing that pops in my mind: how do you do a cleanup when you cleanup the SPEC build directories after a build + run? I'm thinking maybe the error is related to an old *.mod file being kept around in one of the SPEC build directories.

The sure-fire way to do it is to rm -rf all the SPEC build directories before doing another run:

rm -Rf $SPEC/benchspec/C//run rm -Rf $SPEC/benchspec/C//build rm -Rf $SPEC/benchspec/C//exe

More details here:

https://www.spec.org/cpu2017/Docs/runcpu.html#cleaning

Another thing: -O3 might give you better performance than -Ofast with LLVM.

And also:

-fvectorize -fslp-vectorize -finline-functions -funroll-loops

For vectorization, YMMV. The performance result is highly dependent on how many Vector functional units the CPU has.

To use PGI's PGMATH vector library -- for Fortran only:

-fveclib=PGMATH

To use the SLEEF vector library -- for C and C++ only:

-fveclib=SLEEF

Flags for Fortran only:

-freciprocal-math -Mallocatable=03 -Mcache_align

Flags for C, C++ and Fortran:

-ffast-math -ffp-contract=fast

-flto -fuse-ld=gold (use the GNU Gold Linker)

The following benchmarks benefit quite a bit from Polly (polyhedral optimization):

• exchange2
• fotonik3d

Polly Flags (C, C++ and Fortran): -mllvm -polly -mllvm -polly-vectorizer=stripmine -mllvm -polly-codegen-generate-expressions -mllvm -polly-stmt-granularity=scalar-indep -mllvm -polly-ignore-aliasing

If you're running SPECspeed, something to keep in mind: the default OpenMP scheduling for LLVM's OpenMP is 'dynamic', unlike GCC, where the default scheduling is 'static'. Dynamic scheduling is slow. You should set it to 'static' in your SPEC config file, when compiling with LLVM/Flang:

export OMP_SCHEDULE="static"

--Stefan

---

On Wed, Feb 26, 2020 at 12:11 PM Andrei F. notifications@github.com wrote:

Sorry for the extremely late response here, I finally managed to get back to this.

I managed to compile using Arm's compiler, and everything seems to work relatively well except for a few oddities on compiling a few SPEC programs:

[1/16] Building Fortran object 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90.o FAILED: 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90.o 549.fotonik3d_r/power_mod.mod /data/workfolder/llvm/install/bin/flang -I../CFP2017/549.fotonik3d_r -I../CFP2017/549.fotonik3d_r -J549.fotonik3d_r -static -Ofast -fomit-frame-pointer -DNDEBUG -DSPEC_CPU -DSPEC -DSPEC_CPU_LITTLEENDIAN -DSPEC_AUTO_BYTEORDER=0x12345678 -DSPEC_WORDS_LITTLEENDIAN -DANDROID_LIB -DBIONIC_IOCTL_NO_SIGNEDNESS_OVERLOAD -Wno-macro-redefined -Wno-builtin-macro-redefined -mcpu=cortex-a76 -mtune=cortex-a76 -DSPEC_LINUX_AARCH64 -DSPEC_CPU_LINUX -DSPEC_CPU_LP64 -DSPEC_LINUX -DSPEC_LP64 -w -DSPEC_AUTO_SUPPRESS_OPENMP -c 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90-pp.f90 -o 549.fotonik3d_r/CMakeFiles/549.fotonik3d_r.dir/power.F90.o F90-S-0004-Old Module file /data/workfolder/llvm/install/bin/../include/iso_fortran_env.mod (../CFP2017/549.fotonik3d_r/power.F90: 44) F90-F-0004-Recompile source file (../CFP2017/549.fotonik3d_r/power.F90: 44) F90/aarch64 Linux Flang - 1.5 2017-05-01: compilation aborted

Searching around a bit tells me I'm supposed to delete .mod files but since these are include header .mod files that doesn't seem to make sense.

Using the Arm Flang compiler works just fine and the mentioned .mod files are identical except for the dates. The issue is identical on 549.fotonik3d and 507.cactuBSSN_r .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

-- Stefan Teleman stefan.teleman@gmail.com

[AndreiLux]

I'm not using SPEC's harness but have a custom CMake and Ninja system I use across Linux/Windows/Android/iOS.

If you don't have the problem then it must be on my end... it's weird that it happens with this one compiler build but not with the others. I'll have to investigate more.

Thanks for the performance tips, but generally we want to keep things simple for comparisons between platforms and vendors. I will try -O3 instead of -Ofast though.