Closed szhorvat closed 10 months ago
It seems that even if I force an OpenMP-capable compiler, -fopenmp
is not passed on the linking line, and I get linker errors for CHOLDMOD:
ld: Undefined symbols:
___kmpc_end_serialized_parallel, referenced from:
_rs_cholmod_super_numeric_worker in cholmod_l_super_numeric.c.o
_rs_cholmod_super_numeric_worker in cholmod_l_super_numeric.c.o
...
Can you check if you have a stale copy of SuiteSparse installed in /opt/local/lib/?
The line numbers are wrong in this message:
/opt/local/lib/cmake/SuiteSparse_config/SuiteSparse_configConfig.cmake:68 (find_dependency)
You might have an older beta copy there.
You should see this:
and note that line 68 isn't a find_package
, but just an if statement.
I did add a variable SUITESPARSE_USE_STRICT
, which defaults to OFF
, that may impact this. If ON
, then if OpenMP is requested then it throws an error if not found. The default of OFF
means "look for OpenMP and use it if available.".
OpenMP is not a hard requirement for any package in SuiteSparse, unless you set SUITESPARSE_USE_STRICT
to ON
, but keep the SUITESPARSE_USE_OPENMP
at its default value of ON
.
IIUC, you are building each library stand-alone. Is that correct?
If it is, I believe the issue you are seeing is resolved for SuiteSparse 7.4.0.
Could you check if you still see this issue with a recent beta?
Please, ignore the second part. Misread the original comment.
and note that line 68 isn't a
find_package
, but just an if statement.
The generated Config.cmake file looks different from the input. The actual line number might differ depending on the version of CMake that was used to generate the file. But for me, line 68 is the one with find_dependency ( OpenMP )
, too.
@szhorvat: IIUC, OpenMP is a new dependency of SuiteSparse_config in version 7.4.0. Almost, all libraries in SuiteSparse depend on SuiteSparse_config. So, most of them inherit that dependency now.
I'm not sure why you are seeing "Could NOT find SuiteSparse_config (missing: SuiteSparse_config_DIR)" though. But if I understood you correctly, that seems to be benign. Is that correct? (We'd still need to look into it. But it might not warrant blocking the release if it is benign.)
I'm not sure why you are seeing "Could NOT find SuiteSparse_config (missing: SuiteSparse_config_DIR)" though. But if I understood you correctly, that seems to be benign. Is that correct? (We'd still need to look into it. But it might not warrant blocking the release if it is benign.)
Ah. I see now: All libraries prefer to link to other SuiteSparse libraries that are currently being built in the same source tree. That is needed to support the root Makefile. To do that, the build rules first check if they can find a (not-yet installed) version of other SuiteSparse libraries in the common source tree before they look for libraries that are installed. Since you are building each library as a stand-alone project (which is fine), that first check fails. But it succeeds when looking for an installed version of the library.
So, all of the behavior that you are describing is expected with version 7.4.0. 👍
Can you check if you have a stale copy of SuiteSparse installed in /opt/local/lib/?
I am using v7.4.0.beta8
.
/opt/local/lib/cmake/SuiteSparse_config/SuiteSparse_configConfig.cmake:68 (find_dependency)
But this file is generated, so perhaps it ended up being generated differently than on your machine?
You should see this:
The file you are quoting is SuiteSparse/SuiteSparse_config/CMakeLists.txt
. The error message references SuiteSparse_configConfig.cmake
which is generated from https://github.com/DrTimothyAldenDavis/SuiteSparse/blob/v7.4.0.beta8/SuiteSparse_config/Config/SuiteSparse_configConfig.cmake.in
OpenMP is not a hard requirement for any package in SuiteSparse, unless you set
SUITESPARSE_USE_STRICT
toON
, but keep theSUITESPARSE_USE_OPENMP
at its default value ofON
.
Both of these were left at their default value.
IIUC, you are building each library stand-alone. Is that correct?
Yes, that is correct.
IIUC, OpenMP is a new dependency of SuiteSparse_config in version 7.4.0. Almost, all libraries in SuiteSparse depend on SuiteSparse_config. So, most of them inherit that dependency now.
SuiteSparse_config
is being compiled without OpenMP, and the same for AMD. Here's the output from configuring SuiteSparse_config
:
-- ------------------------------------------------------------------------
-- SuiteSparse CMAKE report for: SuiteSparseConfig
-- ------------------------------------------------------------------------
-- inside common SuiteSparse root: OFF
-- install in SuiteSparse/lib and SuiteSparse/include: OFF
-- build type: MacPorts
-- BUILD_SHARED_LIBS: ON
-- BUILD_STATIC_LIBS: ON
-- use OpenMP: no
-- C compiler: /usr/bin/clang
-- C flags: -pipe -O3 -DNDEBUG -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk
-- C++ compiler: /usr/bin/clang++
-- C++ flags:
-- C Flags release: -O3 -DNDEBUG
-- C++ Flags release:
-- Fortran compiler: none
-- compile definitions: BLAS_Apple
-- BLAS integer: int32_t
I'm not sure why you are seeing "Could NOT find SuiteSparse_config (missing: SuiteSparse_config_DIR)" though.
So, all of the behavior that you are describing is expected with version 7.4.0. 👍
Thanks for the explanation!
OpenMP is not a hard requirement for any package in SuiteSparse, unless you set
SUITESPARSE_USE_STRICT
toON
, but keep theSUITESPARSE_USE_OPENMP
at its default value ofON
.
Once again, I originally left these at their defaults, and compiling AMD failed due to a lack of OpenMP support.
If I add -DSUITESPARSE_USE_OPENMP=OFF
to SuiteSparse_config (not to AMD!) then AMD does compile correctly.
Is this expected?
Currently, in MacPorts, only GraphBLAS and UMFPACK are compiled with OpenMP support. Is this still going to be possible if SUITESPARSE_USE_OPENMP=OFF is set for SuiteSparse_config?
If I add
-DSUITESPARSE_USE_OPENMP=OFF
to SuiteSparse_config (not to AMD!) then AMD does compile correctly.Is this expected?
Yes, that is expected. If you build SuiteSparse_config without OpenMP, you don't end up with that transient dependency for all libraries that link to SuiteSparse_config.
Currently, in MacPorts, only GraphBLAS and UMFPACK are compiled with OpenMP support. Is this still going to be possible if SUITESPARSE_USE_OPENMP=OFF is set for SuiteSparse_config?
That's probably a question for @DrTimothyAldenDavis to answer.
Yes, that is expected. If you build SuiteSparse_config without OpenMP, you don't end up with that transient dependency for all libraries that link to SuiteSparse_config.
But it WAS built without OpenMP in the first place. The compiler I used simply does not support OpenMP, and the configuration output clearly indicated that it was not built with OpenMP.
To be very clear, what I did here was not to build SuiteSparse_config
without OpenMP, but simply to set SUITESPARSE_USE_OPENMP=OFF
in a scenario where OpenMP was already unavailable.
Thanks for clarifying.
I opened #628 that should fix this issue.
Currently, in MacPorts, only GraphBLAS and UMFPACK are compiled with OpenMP support. Is this still going to be possible if SUITESPARSE_USE_OPENMP=OFF is set for SuiteSparse_config?
Afaict, GraphBLAS doesn't link to SuiteSparse_config. And UMFPACK doesn't depend (directly) on OpenMP at all. (It might depend on it via CHOLMOD though.)
It's not quite acting as I expect it to.
If I do:
cd SuiteSparse/build
ccmake ..
then configure (c), it does its default initialization. I then set this in the ccmake gui:
SUITESPARSE_USE_OPENMP OFF
and hit (c) again. But the variables for the packages that directly use OpenMP:
CHOLMOD_USE_OPENMP
GRAPHBLAS_USE_OPENMP
LAGRAPH_USE_OPENMP
PARU_USE_OPENMP
are all still ON
. I guess I'm not clear how the option ( CHOLMOD_USE_OPENMP "...description..." ${SUITESPARSE_USE_OPENMP} )
works. My SuiteSparse/build cache has:
hypersparse $ pwd
/home/faculty/d/davis/dev2/SuiteSparse/build
hypersparse $ ack USE_OPENMP
CMakeCache.txt
108:// (Default: SUITESPARSE_USE_OPENMP)
109:CHOLMOD_USE_OPENMP:BOOL=ON
683:// (Default: SUITESPARSE_USE_OPENMP)
684:GRAPHBLAS_USE_OPENMP:BOOL=ON
703:// (Default: SUITESPARSE_USE_OPENMP)
704:LAGRAPH_USE_OPENMP:BOOL=ON
780:// (Default: SUITESPARSE_USE_OPENMP)
781:PARU_USE_OPENMP:BOOL=ON
859:SUITESPARSE_USE_OPENMP:BOOL=OFF
1637://MODIFIED property for variable: SUITESPARSE_USE_OPENMP
1638:SUITESPARSE_USE_OPENMP-MODIFIED:INTERNAL=ON
my ccmake log output has
OpenMP disabled
which is a status message from SuiteSparse_config/CMakeLists.txt. Then when I build all of SuiteSparse (after "g" generate and then "make"), it builds with OpenMP.
My understanding of how cmake variables work is limited, particularly when it involves the cache.
Is SUITESPARSE_USE_OPENMP gets turned off, it's suppose to turn off OpenMP for all of SuiteSparse, ignoring the settings for each individual package.
This is after merging https://github.com/DrTimothyAldenDavis/SuiteSparse/pull/628 .
If I then turn off these cached variables, all to OFF:
CHOLMOD_USE_OPENMP
GRAPHBLAS_USE_OPENMP
LAGRAPH_USE_OPENMP
PARU_USE_OPENMP
I have this in my cache:
hypersparse $ pwd
/home/faculty/d/davis/dev2/SuiteSparse/build
hypersparse $ ack USE_OPENMP
CMakeCache.txt
108:// (Default: SUITESPARSE_USE_OPENMP)
109:CHOLMOD_USE_OPENMP:BOOL=OFF
683:// (Default: SUITESPARSE_USE_OPENMP)
684:GRAPHBLAS_USE_OPENMP:BOOL=OFF
703:// (Default: SUITESPARSE_USE_OPENMP)
704:LAGRAPH_USE_OPENMP:BOOL=OFF
780:// (Default: SUITESPARSE_USE_OPENMP)
781:PARU_USE_OPENMP:BOOL=OFF
859:SUITESPARSE_USE_OPENMP:BOOL=OFF
1114://MODIFIED property for variable: CHOLMOD_USE_OPENMP
1115:CHOLMOD_USE_OPENMP-MODIFIED:INTERNAL=ON
1536://MODIFIED property for variable: GRAPHBLAS_USE_OPENMP
1537:GRAPHBLAS_USE_OPENMP-MODIFIED:INTERNAL=ON
1558://MODIFIED property for variable: LAGRAPH_USE_OPENMP
1559:LAGRAPH_USE_OPENMP-MODIFIED:INTERNAL=ON
1641://MODIFIED property for variable: PARU_USE_OPENMP
1642:PARU_USE_OPENMP-MODIFIED:INTERNAL=ON
1651://MODIFIED property for variable: SUITESPARSE_USE_OPENMP
1652:SUITESPARSE_USE_OPENMP-MODIFIED:INTERNAL=ON
but I still get all those packages linked with OpenMP, as well as KLU, UMFPACK, and SPQR which get it from CHOLMOD or perhaps from the BLAS (I'm using an OpenMP-based BLAS). GraphBLAS doesn't call the BLAS, though, so it's only getting OpenMP directly, not via the BLAS.
Oh ... wait, GraphBLAS hasn't finished building yet. Perhaps it's not being built with OpenMP.
OK, GraphBLAS finished building. I still have libgomp being linked with into CHOLMOD, KLU, UMFPACK, ParU, and SPQR, but not GraphBLAS, with all of
SUITESPARSE_USE_OPENMP
CHOLMOD_USE_OPENMP
GRAPHBLAS_USE_OPENMP
LAGRAPH_USE_OPENMP
PARU_USE_OPENMP
set of OFF, but this would be expected since I'm still using the Intel MKL, which uses libgomp. GraphBLAS and LAGraph are no longer using OpenMP. ParU is no longer using OpenMP directly (because it trigged a compiler error I had to fix just now, with this fix: https://github.com/DrTimothyAldenDavis/SuiteSparse/commit/95b3bdc2bc51e2347b131fddaeac59bf098c1577 ).
Now if I set BLA_VENDOR
to Generic
, then I can build all of SuiteSparse without the BLAS.
So I can build all of SuiteSparse without OpenMP, but I'm still puzzled by how these lines below work. Shouldn't CHOLMOD_USE_OPENMP get turned off if SUITESPARSE_USE_OPENMP get turned off?
hypersparse $ ack "option \( " | grep OPENMP
CHOLMOD/CMakeLists.txt:82:option ( CHOLMOD_USE_OPENMP "ON: Use OpenMP in CHOLMOD if available. OFF: Do not use OpenMP. (Default: SUITESPARSE_USE_OPENMP)" ${SUITESPARSE_USE_OPENMP} )
GraphBLAS/CMakeLists.txt:98:option ( GRAPHBLAS_USE_OPENMP "ON: Use OpenMP in GraphBLAS if available. OFF: Do not use OpenMP. (Default: SUITESPARSE_USE_OPENMP)" ${SUITESPARSE_USE_OPENMP} )
GraphBLAS/GraphBLAS/CMakeLists.txt:59:option ( GRAPHBLAS_USE_OPENMP "ON: Use OpenMP in GraphBLAS if available. OFF: Do not use OpenMP. (Default: SUITESPARSE_USE_OPENMP)" ${SUITESPARSE_USE_OPENMP} )
GraphBLAS/cmake_modules/SuiteSparsePolicy.cmake:124:option ( SUITESPARSE_USE_OPENMP "ON (default): Use OpenMP if available. OFF: Do not use OpenMP" ON )
LAGraph/CMakeLists.txt:158:option ( LAGRAPH_USE_OPENMP "ON: Use OpenMP in LAGraph if available. OFF: Do not use OpenMP. (Default: SUITESPARSE_USE_OPENMP)" ${SUITESPARSE_USE_OPENMP} )
LAGraph/cmake_modules/SuiteSparsePolicy.cmake:124:option ( SUITESPARSE_USE_OPENMP "ON (default): Use OpenMP if available. OFF: Do not use OpenMP" ON )
ParU/CMakeLists.txt:57:option ( PARU_USE_OPENMP "ON: Use OpenMP in ParU if available. OFF: Do not use OpenMP. (Default: SUITESPARSE_USE_OPENMP)" ${SUITESPARSE_USE_OPENMP} )
SuiteSparse_config/cmake_modules/SuiteSparsePolicy.cmake:124:option ( SUITESPARSE_USE_OPENMP "ON (default): Use OpenMP if available. OFF: Do not use OpenMP" ON )
Currently, in MacPorts, only GraphBLAS and UMFPACK are compiled with OpenMP support. Is this still going to be possible if SUITESPARSE_USE_OPENMP=OFF is set for SuiteSparse_config?
That's probably a question for @DrTimothyAldenDavis to answer.
Well ... I guess it could, but that wasn't my intent. My intent was:
(1) SUITESPARSE_USE_OPENMP is OFF: I meant this as "nobody anywhere uses OpenMP in all of SuiteSparse", but my implementation of this is broken and OpenMP is still sneaking in (ignoring the BLAS).
(2) SUITESPARSE_USE_OPENMP is ON: SuiteSparse_config uses OpenMP (see the call to omp_get_wtime here):
And each package can optionally using OpenMP, depending on its Package_USE_OPENMP flag.
So if, say, CHOLMOD_USE_OPENMP
is OFF but GRAPHBLAS_USE_OPENMP
is ON, then CHOLMOD does not directly use OpenMP (but it might still do so through the BLAS), and GraphBLAS does use OpenMP. AMD would not use OpenMP since it doesn't call the SuiteSparse_time, _tic, or _toc methods directly (so it doesn't call omp_get_wtime).
That's my intent. It's why I put in the variables CHOLMOD_USE_OPENMP
, GRAPHBLAS_USE_OPENMP
, etc.
So to get what you want (OpenMP used by GraphBLAS and UMFPACK only), then set SUITESPARSE_USE_OPENMP
to ON, and all other Package_USE_OPENMP to OFF.
I don't have an UMFPACK_USE_OPENMP
since UMFPACK only uses OpenMP via the BLAS.
I'm working on revising my use of these variables to align with my intent. But I see I also must add
SUITESPARSE_CONFIG_USE_OPENMP
SUITESPARSE_CONFIG_HAS_OPENMP
because SuiteSparse_config
itself can optionally use the omp_get_wtime
function. AMD calls SuiteSparse_config
, so if you want to use GraphBLAS with OpenMP, but AMD without, then SuiteSparse_config
cannot use OpenMP.
This should be fine, once I get it to work. If a package has OpenMP, but SuiteSparse_config
does not, and if I really want to use the omp_get_wtime
function itself in that package, then I could just call it directly, without calling SuiteSparse_time
.
Once I get this to work, to enable OpenMP in just GraphBLAS, you would need to do:
SUITESPARSE_USE_OPENMP = ON
SUITESPARSE_CONFIG_USE_OPENMP = OFF
GRAPHBLAS_USE_OPENMP = ON
there won't be an UMFPACK_USE_OPENMP
option. It might call SuiteSparse_time
, but if so, I would just get the wallclock time another way.
This is my intent, anyway. Let me know if this will be OK for macports.
OK, I think I have it working now.
If I do
SUITESPARSE_USE_OPENMP = ON
SUITESPARSE_CONFIG_USE_OPENMP = OFF
GRAPHBLAS_USE_OPENMP = ON
then only GraphBLAS, LAGraph, and packages that use the BLAS (or rely on those that do) get libgomp. SuteSparse_config and AMD do not have OpenMP. If I then add
BLA_VENDOR = Generic
Then only GraphBLAS has OpenMP; libgomp appears nowhere in all of SuiteSparse, except for GraphBLAS and LAGraph (both liblagraph.so and liblagraphx.so) since LAGraph relies on GraphBLAS.
Thanks you for releasing beta9. I just tried it and I can compile without OpenMP according to your instructions.
However, I cannot compile with OpenMP. -fopenmp
is not passed at the linking stage, so linking fails with missing symbols. This happens when I compile using MacPorts's infrastructure, but not when I tried to compile it manually. I need more time to figure out why exactly it happens.
Oh ... well, that's some progress at least.
I don't know why -fopenmp isn't passed correctly. Something to do with your FindOpenMP.cmake or OpenMPConfig.cmake perhaps?
I'm able to build OK with my M1 Mac, using this OpenMP with Apple Clang:
/usr/local/lib/libomp.dylib was installed via:
https://mac.r-project.org/openmp/
which directly installs universal binaries (arm and x86) files into:
/usr/local/lib/libomp.dylib
/usr/local/include/ompt.h
/usr/local/include/omp.h
/usr/local/include/omp-tools.h
I haven't tried my Intel Mac, which uses OpenMP from homebrew if I recall (I'm away from home and only have my M1 Mac at the moment).
But I guess this doesn't help much since you can already build with OpenMP as well, just outside of the MacPorts setup.
I can also successfully build without any OpenMP at all with SUITESPARSE_USE_OPENMP
OFF
. No package has libomp (the BLAS is from Apple Accelerate which is parallel but not via OpenMP).
I then tried it with SUITESPARSE_USE_OPENMP
set to ON
, and all other Package_USE_OPENMP
set to OFF
except for GraphBLAS, and everything worked as expected.
So unless there's something specific I need to add to allow MacPorts to work, I think this issue could be closed for SuiteSparse 7.4.0. But I'll leave it open until you get MacPorts to work.
@szhorvat: Which libraries no longer build with OpenMP? Or is it all of them?
@mmuetzel All the ones that use OpenMP (haven't tried GraphBLAS yet). This is some weird MacPorts-specific issue because if I use the same CMake configure command as MacPorts does, it works fine. Maybe MacPorts sets some environment variables that interfere. Here's the full output of an installation attempt:
Apparently it's due to setting the CC_PRINT_OPTIONS
and CC_PRINT_OPTIONS_FILE
environment variables. I can reproduce the problem when I set them.
I found this comment:
https://trac.macports.org/wiki/CompilerEnvironmentVariables
Setting CC_PRINT_OPTIONS causes -v to behave differently. For example, it breaks FindOpenMP in CMake.
It looks like this is not an issue with SuiteSparse.
Update: Yes, this was a MacPorts specific issue, which I was able to work around now. It is not a problem with SuiteSparse.
Merry Christmas!
I am closing this since the original problem I reported seems to be fixed. I will open new issues for other problems.
In version 7.4.0beta8, the AMD and related packages no longer builds if the compiler does not support OpenMP. The default compiler on macOS doers not support OpenMP. This was not an issue with 7.3.1.
Is this change intentional or accidental?
This came up while working on https://github.com/macports/macports-ports/pull/21451. I can force an OpenMP-capable compiler if necessary, but I'd like to know if this is just a bug or a new hard requirement.
Additionally I see an unexpected "Could NOT find SuiteSparse_config" message, even though it does in fact find the installed SuiteSparse_config, as evidenced by the following output lines.
(Note: the above is with a compiler that does support OpenMP.)