search

ralna / GALAHAD

A library of modern Fortran modules for nonlinear optimization
https://www.galahad.rl.ac.uk
Other
119 stars 15 forks source link

'omp.h' file not found on Platform("aarch64", "macos") #18

Closed amontoison closed 1 year ago

amontoison commented 2 years ago 
 Compiling guess_topology       guess_topology.cxx:13:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
 => Disabling optimization guess_topology.cxx:13:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
make[2]: *** [/workspace/srcdir/GALAHAD/src/ssids/makemaster:310: /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/libgalahad_spral.a(guess_topology.o)] Error 1
make[1]: *** [/workspace/srcdir/GALAHAD/src/makedefs/intermediaries:560: make_ssids] Error 2
make: *** [/workspace/srcdir/GALAHAD/src/makemaster:187: all_standalone_double] Error 2
dpo commented 2 years ago

What version of gcc?

amontoison commented 2 years ago

It was gcc 11.1.

amontoison commented 2 years ago

https://stackoverflow.com/a/33401294

nimgould commented 2 years ago

The SOMP = -fopenmp compiler flag should provide all the relevant include data here. Could you please go to the $GALAHAD/src/ssids directory, and try

make -f (architecture)

where (architecture) is whatever name you have in $GALAHAD/makefiles/ (e.g., mac64.osx.gfo). This will tell us what flags are being passed

amontoison commented 2 years ago

I have an error when I try make -f binarybuilder.bb.fc:

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f binarybuilder.bb.fc
make: binarybuilder.bb.fc: No such file or directory
make: *** No rule to make target 'binarybuilder.bb.fc'.  Stop.

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # ls
C                        config.h                 fkeep.F90                gpu_subtree_no_cuda.f90  solve.cu
LICENCE                  contrib.f90              gpu                      inform.f90               ssids.f90
NumericSubtree.cxx       contrib.h                gpu_alloc.f90            ldlt_app.cxx             ssidss.data
README                   contrib_free.f90         gpu_datatypes.f90        ldlt_nopiv.cxx           ssidss.f90
SymbolicSubtree.cxx      cpu                      gpu_dense_factor.f90     ldlt_tpp.cxx             subtree.f90
ThreadStats.cxx          cpu_iface.f90            gpu_factor.f90           makemaster               syrk.cu
akeep.f90                cpu_solve.f90            gpu_interfaces.f90       profile.cxx              updata_ssids
anal.f90                 cpu_subtree.f90          gpu_smalloc.f90          profile.hxx              wrappers.cxx
assemble.cu              datatypes.f90            gpu_solve.f90            profile_iface.f90
cholesky.cxx             dense_factor.cu          gpu_subtree.f90          reorder.cu
sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids #
nimgould commented 2 years ago

Sorry, Alexis, I should have said make -f $GALAHAD/makefiles/binary builder.bb.fc

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Alexis @.> Sent: Tuesday, November 8, 2022 6:38:50 PM To: ralna/GALAHAD @.> Cc: Gould, Nick (STFC,RAL,SC) @.>; Comment @.> Subject: Re: [ralna/GALAHAD] 'omp.h' file not found on Platform("aarch64", "macos") (Issue #18)

I have an error when I try make -f binarybuilder.bb.fc:

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f binarybuilder.bb.fc make: binarybuilder.bb.fc: No such file or directory make: *** No rule to make target 'binarybuilder.bb.fc'. Stop.

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # ls C config.h fkeep.F90 gpu_subtree_no_cuda.f90 solve.cu LICENCE contrib.f90 gpu inform.f90 ssids.f90 NumericSubtree.cxx contrib.h gpu_alloc.f90 ldlt_app.cxx ssidss.data README contrib_free.f90 gpu_datatypes.f90 ldlt_nopiv.cxx ssidss.f90 SymbolicSubtree.cxx cpu gpu_dense_factor.f90 ldlt_tpp.cxx subtree.f90 ThreadStats.cxx cpu_iface.f90 gpu_factor.f90 makemaster syrk.cu akeep.f90 cpu_solve.f90 gpu_interfaces.f90 profile.cxx updata_ssids anal.f90 cpu_subtree.f90 gpu_smalloc.f90 profile.hxx wrappers.cxx assemble.cu datatypes.f90 gpu_solve.f90 profile_iface.f90 cholesky.cxx dense_factor.cu gpu_subtree.f90 reorder.cu sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids #

I have the following files $GALAHAD/makefiles/:

sandbox:${WORKSPACE}/srcdir/GALAHAD/makefiles # cat binarybuilder.bb.fc

Architecture dependent makefile

(automatically generated by install_galahad)

VERSION = binarybuilder.bb.fc CUTESTVERSION =

Basic system commands

CP = cp MV = mv RM = rm -f LN = ln MKDIR = mkdir SED = sed CAT = cat GREP = grep AR = ar ARREPFLAGS = -rcuU RANLIB = ranlib DLEXT = dylib LOADALL = -Wl,-all_load LOADNONE =

Directory for binaries

PRECIS = double OBJ = $(GALAHAD)/objects/$(VERSION)/$(PRECIS) OBJS = $(GALAHAD)/objects/$(VERSION)/single OBJD = $(GALAHAD)/objects/$(VERSION)/double MOD = $(GALAHAD)/modules/$(VERSION)/$(PRECIS) CUTESTMOD = SEDS = $(GALAHAD)/seds/$(PRECIS).sed MVMODS = $(MV) -f $(OBJ)/*.mod $(MOD)/ || true

Compiler options

FORTRAN = $(FC) BASIC = -c -fno-second-underscore -fPIC MBASIC = -fno-second-underscore -fPIC MODULES = -I$(MOD) OPTIMIZATION = -O2 NOOPTIMIZATION = -O0 DEBUG = SHARED = -shared OPENMP = -fopenmp SOMP = -fopenmp MOMP = -lgomp F77 = -ffixed-form F90 = F95 = NOFMAIN = USUAL = SPECIAL = F77SUFFIX = f90 F95SUFFIX = f90 TIMER = GEN NOT64 = IS64 PLPLOTUSED = unused IEEECK =

AMPLDIR = CC = cc CCBASIC = -c -fPIC CCISO = -ansi -pedantic CCONDEF = CCDEBUG = #-DDEBUG_GALAHAD HWLOC = GETCPU = un

CXX = c++ CXXBASIC = -c -std=c++11 -fPIC CXXOPT = -O2 CXXNOOPT = -O0 CXXONDEF = CXXDEBUG =

CUDA = CUDACOMP = $(FC) CUDAARCH = CUDACODE = CUDABASIC = CUDALIBS = CUDAOPENMP = -fopenmp

PYTHONVERSION = 0.0 PYTHONINCLUDE = /usr/include/python$(PYTHONVERSION) NUMPYINCLUDE = /usr/lib/python3/dist-packages/numpy/core/include SOABI = PYSITEPACKDIR =

Special flags

MA97BUG =

Libraries

PLPLOT =

-- BLAS --

BLAS = -lblastrampoline

-- LAPACK --

LAPACK = -lblastrampoline

-- HSL --

HSL = -lgalahad_hsl

-- SPRAL --

SPRAL = -lgalahad_spral -lstdc++ -lhwloc

-- Metis --

METIS = -lgalahad_metis

-- Pardiso --

PARDISO = -lgalahad_pardiso

-- MKL Pardiso --

MKL_PARDISO = -lgalahad_mkl_pardiso

-- WSMP --

WSMP = -lgalahad_wsmp

-- PASTIX --

PASTIX = -lgalahad_pastix

-- MUMPS --

MUMPS = -lgalahad_mumps

-- UMFPACK --

UMFPACK = -lgalahad_umfpack

proper or dummy SSIDS package?

SSIDS = ssids

Shell used

BINSHELL = bash

Set directories for optional packages

include $(GALAHAD)/src/makedefs/packages

Body of makefile

include $(PWD)/makemaster

— Reply to this email directly, view it on GitHubhttps://github.com/ralna/GALAHAD/issues/18#issuecomment-1307667392, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACW4A6RD47E63L2HZ3CWVPDWHKM3VANCNFSM6AAAAAARZZJBIY. You are receiving this because you commented.Message ID: @.***>

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

amontoison commented 2 years ago 
sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f $GALAHAD/makefiles/binarybuilder.bb.fc
 Compiling guess_topology       mkdir -p /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology
sed -f /workspace/srcdir/GALAHAD/seds/have_hwloc_def.sed ../spral/config.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h
sed -f /workspace/srcdir/GALAHAD/seds/have_getcpu_undef.sed /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config.h
cp  ../spral/compat.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/
cp  ../spral/guess_topology.cxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/
cp  ../spral/guess_topology.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/
cp  ../spral/hwloc_wrapper.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/
cd /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double ; c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O2   -fopenmp \
          guess_topology.cxx \
                 || ( printf ' %-26s' "=> Disabling optimization " ; \
                    c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O0  -fopenmp \
                      guess_topology.cxx )
guess_topology.cxx:13:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
 => Disabling optimization guess_topology.cxx:13:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
make: *** [/workspace/srcdir/GALAHAD/src/ssids/makemaster:310: /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/libgalahad_spral.a(guess_topology.o)] Error 1
nimgould commented 2 years ago

Are c and c++ valid c compilers on your system? Are they aliased somehow to gcc and g++? Sorry I don't know what is available. -fopenmp is a gcc etc flag, and should pull in omp.h

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Alexis @.> Sent: Tuesday, November 8, 2022 7:03:58 PM To: ralna/GALAHAD @.> Cc: Gould, Nick (STFC,RAL,SC) @.>; Comment @.> Subject: Re: [ralna/GALAHAD] 'omp.h' file not found on Platform("aarch64", "macos") (Issue #18)

sandbox:${WORKSPACE}/srcdir/GALAHAD/src/ssids # make -f $GALAHAD/makefiles/binarybuilder.bb.fc Compiling guess_topology mkdir -p /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology sed -f /workspace/srcdir/GALAHAD/seds/have_hwloc_def.sed ../spral/config.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h sed -f /workspace/srcdir/GALAHAD/seds/have_getcpu_undef.sed /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config1.h > /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/config.h cp ../spral/compat.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/ cp ../spral/guess_topology.cxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/ cp ../spral/guess_topology.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/ cp ../spral/hwloc_wrapper.hxx /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/hw_topology/ cd /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double ; c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O2 -fopenmp \ guess_topology.cxx \ || ( printf ' %-26s' "=> Disabling optimization " ; \ c++ -I. -o guess_topology.o -c -std=c++11 -fPIC -O0 -fopenmp \ guess_topology.cxx ) guess_topology.cxx:13:10: fatal error: 'omp.h' file not found

include 

     ^~~~~~~

1 error generated. => Disabling optimization guess_topology.cxx:13:10: fatal error: 'omp.h' file not found

include 

     ^~~~~~~

1 error generated. make: *** [/workspace/srcdir/GALAHAD/src/ssids/makemaster:310: /workspace/srcdir/GALAHAD/objects/binarybuilder.bb.fc/double/libgalahad_spral.a(guess_topology.o)] Error 1

— Reply to this email directly, view it on GitHubhttps://github.com/ralna/GALAHAD/issues/18#issuecomment-1307694418, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACW4A6SMLNKS67U4FJ42YB3WHKPZ5ANCNFSM6AAAAAARZZJBIY. You are receiving this because you commented.Message ID: @.***>

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

amontoison commented 2 years ago

Yes, cc and c++ are alias for C and C++ compilers. It's gcc and g++ for Linux / Windows and Clang / Clang++ on Mac and FreeBSD.

dpo commented 2 years ago

could you try gcc/gfortran for macOS?

amontoison commented 2 years ago

Yes, I restart the compilation. I just checked and we use gcc/gfortran for SPRAL: https://github.com/JuliaPackaging/Yggdrasil/blob/master/S/SPRAL/build_tarballs.jl#L16-L19

@jfowkes Do you test the compilation of SPRAL on Mac with CI? It seems that modifications are needed for Clang / Clang++ compilers (native compilers on Mac / FreeBSD).

amontoison commented 2 years ago

The compilation of GALAHAD on Mac works with gcc/gfortran.

dpo commented 2 years ago

I would suggest that we forget about clang for now, and first stabilize everything with gcc/gfortran on every platform of interest.

nimgould commented 2 years ago

I believe that Alexis mentioned the work-around here. If c++ is from clang, one needs SOMP = '-fopenmp=libomp' in the compiler.binarybuilder.bb.fc file in archdefs, which is copied into GALAHAD's makefiles/binarybuilder.bb.fc on installation. Give that a try ... but I agree with Dominique that we should concentrate on gcc for the time being.

jfowkes commented 2 years ago

@amontoison SPRAL is only tested on Linux GCC in the CI but I have successfully compiled it with Homebrew GCC on Mac.

I believe it would take quite a bit of nontrivial work to make SPRAL work with Clang given how complex it is.

nimgould commented 2 years ago

I tried flang/clang, and the flag

SOMP = -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include

did the trick here ... but as you see it needs to know where the system header is. The alternative

-I/usr/lib/gcc/x86_64-linux-gnu/11/include/omp.h

also worked, and is marginally more portable.

Unfortunately, a following ssids compile

clang++ -I. -o NumericSubtree.o -c -std=c++11 -fPIC -O -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include NumericSubtree.cxx

produces a wave of errors, typically ./ssids/cpu/kernels/assemble.hxx:212:42: error: variable 'add_a_blk_sz' must have explicitly specified data sharing attributes add_a_block(iblk, std::min(iblk+add_a_blk_sz,snode.num_a), node, aval, scaling);

As I don't know c++, I have no idea what this means, and I can't find any compiler flag to disable it (if that is even possible). This does not happen with icx which is also built on top of LLVM, so I suspect it is just clang being too fussy.

jfowkes commented 2 years ago

Looking into the clang++ error, this is an issue with how Clang interprets the OpenMP standard with regards to const variables and the default(none) directive: Clang is of the opinion that const add_a_blk_sz should have its data sharing attribute explicitly specified (e.g. shared, private, etc, see https://www.openmp.org/spec-html/5.0/openmpsu106.html ). Technically I'd say Clang is correct here as the default(none) clause on line 209 should require that all data sharing attributes be explicitly specified as that is the whole point of default(none) (much like implicit none forces all types to be specified in Fortran).

Basically, Clang doesn't know whether add_a_blk_sz should be shared or not within the for loop on line 212 whereas GCC assumes it to be one or the other (presumably shared?).

A fix we have implemented for previous occurrences of this issue that GCC broke on is to simply remove the default(none) clause (so that such const variables simply use the default data sharing attribute) which has the benefit of being backwards compatible: https://github.com/ralna/spral/commit/6547e55399b21ad816be0c2d5fe2076cefa01fb0

Jari

From: nimgould @.> Sent: 09 November 2022 11:50 To: ralna/GALAHAD @.> Cc: Fowkes, Jaroslav (STFC,RAL,SC) @.>; Mention @.> Subject: Re: [ralna/GALAHAD] 'omp.h' file not found on Platform("aarch64", "macos") (Issue #18)

I tried flang/clang, and the flag

SOMP = -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include

did the trick here ... but as you see it needs to know where the system header is. The alternative

-I/usr/lib/gcc/x86_64-linux-gnu/11/include/omp.h

also worked, and is marginally more portable.

Unfortunately, a following ssids compile

clang++ -I. -o NumericSubtree.o -c -std=c++11 -fPIC -O -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include NumericSubtree.cxx

produces a wave of errors, typically ./ssids/cpu/kernels/assemble.hxx:212:42: error: variable 'add_a_blk_sz' must have explicitly specified data sharing attributes add_a_block(iblk, std::min(iblk+add_a_blk_sz,snode.num_a), node, aval, scaling);

As I don't know c++, I have no idea what this means, and I can't find any compiler flag to disable it (if that is even possible). This does not happen with icx which is also built on top of LLVM, so I suspect it is just clang being too fussy.

— Reply to this email directly, view it on GitHubhttps://github.com/ralna/GALAHAD/issues/18#issuecomment-1308633613, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKT2EEXIMBB3J77OSCQLJLWHOFXDANCNFSM6AAAAAARZZJBIY. You are receiving this because you were mentioned.Message ID: @.***>

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.

nimgould commented 2 years ago

Very interesting. In the block 

  #pragma omp taskgroup
  for(int iblk=0; iblk<snode.num_a; iblk+=add_a_blk_sz) {
     #pragma omp task default(none) \
        firstprivate(iblk) \
        shared(snode, node, aval, scaling, ldl)
     add_a_block(iblk, std::min(iblk+add_a_blk_sz,snode.num_a), node, aval, scaling);
  }

what precisely would you remove? I'm willing to try, but need hand holding

jfowkes commented 2 years ago

If my understanding is correct then removing the default(none) after #pragma omp task should fix it.

nimgould commented 2 years ago

I'll try tomorrow once I "get back" from Edinburgh

amontoison commented 2 years ago

Ok, so the issue is in spral and not in GALAHAD. We could move the issue to the GitHub SPRAL repository directly. I agree with Dominique, we should compile GALAHAD with gcc/gfortran on all relevant platforms.

@jfowkes I did two CI scripts for a C++ code a few months ago, you can easily adapt them for SPRAL: https://github.com/bbopt/nomad/blob/master/.github/workflows/ci.yml https://github.com/bbopt/nomad/blob/master/.cirrus.yml

jfowkes commented 2 years ago

Thanks @amontoison, I've created a SPRAL pull request to test with Clang/Flang: https://github.com/ralna/spral/pull/97 Flang doesn't seem to be working though, any suggestions? We should discuss there.

nimgould commented 2 years ago

Good call, Jari. removing the three default (none)s fixed the compiler errors, and the sls exhaustive test now runs ... and ssids gives the right answer. I should add that this is with flang/clang as compiled from https://github.com/flang-compiler/llvm.git . Next I need to make sure that these changes don't ruin ssids under other compilers. Progress on an otherwise hopeless day!

nimgould commented 2 years ago

Oh, I should add that I needed to set the galahad variables OPENMP = -fopenmp -lstdc++ -lhwloc SOMP = -fopenmp=libomp -I/usr/lib/llvm-14/lib/clang/14.0.0/include and the other is of course not portable (but presumably will be when flang is finally released as part of clang

nimgould commented 2 years ago

OK, the updates also work for gcc and icc. icx fails, but I think it did before. Jari, are you planning to make these changes in spral as well?

jfowkes commented 2 years ago

@nimgould yes I will make these changes to SPRAL, could you point me to the three default(none) statements you removed?

nimgould commented 2 years ago

Yes, in assemble.hxx (as per $GALAHAD/src/ssids/cpu/kernel/), lines 209, 281 and 389

jfowkes commented 2 years ago

@nimgould done, fixed in SPRAL.