v4.6 build "make petsc-slepc" fails with "C compiler you provided with -with-cc= cannot be found or does not work." on Ubuntu 20.04

[cirosantilli]

Just trying it out for fun, I ran the build commands from the README:

sudo apt build-dep freefem
git clone https://github.com/FreeFem/FreeFem-sources
cd FreeFem-sources
git checkout v4.6
autoreconf -i
./configure --enable-download --enable-optim --prefix="$(pwd)/FreeFem-install"
./3rdparty/getall -a
cd 3rdparty/ff-petsc
make petsc-slepc

and the last command fails with:

dirname: missing operand
Try 'dirname --help' for more information.
mkdir -p /home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//r
chown ciro /home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//r
mkdir -p /home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//c
chown ciro /home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//c
no need of sudo
test -e "/home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//r" -a -e "/home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//c"
cd petsc-3.13.0 && ./configure MAKEFLAGS='' \
        --prefix=/home/ciro/test/FreeFem-sources/FreeFem-install/ff-petsc//r \
        --with-debugging=0 COPTFLAGS='-O3 -mtune=native' CXXOPTFLAGS='-O3 -mtune=native' FOPTFLAGS='-O3 -mtune=native' --with-cxx-dialect=C++11 --with-ssl=0 --with-x=0 --with-fortran-bindings=0 --with-cc='' --with-cxx='' --with-fc='' --with-scalar-type=real --with-blaslapack-include='' --with-blaslapack-lib='-llapack -lblas' --download-scalapack --download-metis --download-ptscotch --download-mumps --download-hypre --download-parmetis --download-superlu --download-suitesparse --download-tetgen --download-slepc --download-hpddm --download-hpddm-commit=e8639ff PETSC_ARCH=fr
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:604)                                                                                                                                                                                 *******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
C compiler you provided with -with-cc= cannot be found or does not work.
*******************************************************************************

make: *** [Makefile:210: petsc-3.13.0/tag-conf-real] Error 1

This error message is mentioned at:

• https://github.com/FreeFem/FreeFem-sources/issues/72
• https://github.com/FreeFem/FreeFem-sources/issues/116

but I couldn't find easily in those threads the solution/if we share the same problem.

Edit: working procedure

Maybe you need to uninstall openmpi before starting. Then this runs the hello world examples from the repo:

sudo apt build-dep freefem
git clone https://github.com/FreeFem/FreeFem-sources
cd FreeFem-sources
# Post v4.6 with some fixes.
git checkout 3df0e2370d9752801ac744b11307b14e16743a44

# Won't apply automatically due to tab hell.
# https://superuser.com/questions/607410/how-to-copy-paste-tab-characters-via-the-clipboard-into-terminal-session-on-gnom
git apply <<'EOS'
diff --git a/3rdparty/ff-petsc/Makefile b/3rdparty/ff-petsc/Makefile
index dc62ab06..13cd3253 100644
--- a/3rdparty/ff-petsc/Makefile
+++ b/3rdparty/ff-petsc/Makefile
@@ -204,7 +204,7 @@ $(SRCDIR)/tag-make-real:$(SRCDIR)/tag-conf-real
 $(SRCDIR)/tag-install-real :$(SRCDIR)/tag-make-real
     cd $(SRCDIR) && $(MAKE) PETSC_DIR=$(PETSC_DIR) PETSC_ARCH=fr install
     -test -x "`type -p otool`" && make changer
-    cd $(SRCDIR) && $(MAKE) PETSC_DIR=$(PETSC_DIR) PETSC_ARCH=fr check
+    #cd $(SRCDIR) && $(MAKE) PETSC_DIR=$(PETSC_DIR) PETSC_ARCH=fr check
     test -e $(DIR_INSTALL_REAL)/include/petsc.h
     test -e $(DIR_INSTALL_REAL)/lib/petsc/conf/petscvariables
     touch $@
@@ -293,7 +293,6 @@ $(SRCDIR)/tag-tar:$(PACKAGE)
     -tar xzf $(PACKAGE)
     patch -p1 < petsc-hpddm.patch
 ifeq ($(WIN32DLLTARGET),)
-    patch -p1 < petsc-metis.patch
 endif
     touch $@
 $(PACKAGE):
EOS

autoreconf -i
./configure --enable-download --enable-optim --prefix="$(pwd)/../FreeFem-install"
./3rdparty/getall -a
cd 3rdparty/ff-petsc
make petsc-slepc
cd -
./reconfigure
make -j`nproc`
make install
cd ../FreeFem-install
PATH="${PATH}:$(pwd)/bin" ./bin/FreeFem++ ../FreeFem-sources/examples/tutorial/

[prj-]

You do not have MPI installed on your machine (or at least FreeFEM can't detect it). Two options: 1) install MPI using your favorite package manager, then reconfigure, and if this still fails, send us your FreeFEM config.log; 2) switch to the develop branch, which now lets PETSc install MPICH for you in case FreeFEM can't detect MPI. Then PETSc-installed MPICH will be used by FreeFem++-mpi. This will be available in the next FreeFEM release (4.6.1 or 4.7).

Please let us know if this fixes your issue.

[cirosantilli]

Hmmm, I think I already had the package libopenmpi-dev installed. Now I have it for sure, and also installed sudo apt installlibmpich-dev` and I restarted the process, but got the same error.

The problem seems to come from:

-with-cc=''

and then if I do:

make petsc-slepc MPICC=/usr/bin/gcc MPICXX=/usr/bin/g++ MPIFC=/usr/bin/gfortran

it progresses much further and blows up at:

C++ error! MPI_Finalize() could not be located!

From logs, it failed to compile:

Executing: /usr/bin/g++ -c -o /tmp/petsc-tlf4tu2b/config.libraries/conftest.o -I/tmp/petsc-tlf4tu2b/config.setCompilers -I/tmp/petsc-tlf4tu2b/config.compilers -I/tmp/petsc-tlf4tu2b/config.types -I/tmp/petsc-tlf4tu2b/config.functions -I/tmp/petsc-tlf4tu2b/config.utilities.missing -I/tmp/petsc-tlf4tu2b/config.libraries  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden -O3 -mtune=native  -fPIC    /tmp/petsc-tlf4tu2b/config.libraries/conftest.cc 
Possible ERROR while running compiler: exit code 1
stderr:
/tmp/petsc-tlf4tu2b/config.libraries/conftest.cc:4:10: fatal error: mpi.h: No such file or directory
    4 | #include <mpi.h>
      |          ^~~~~~~
compilation terminated.
Source:
#include "confdefs.h"
#include "conffix.h"
/* Override any gcc2 internal prototype to avoid an error. */
#include <mpi.h>
static void _check_MPI_Finalize() { int ierr;
ierr = MPI_Finalize();; }

int main() {
_check_MPI_Finalize();;
  return 0;
}

because mpi.h is not found, because it needs pkg-config --cflags to provide a missing

-I/usr/lib/x86_64-linux-gnu/openmpi/include

I am able to compile that test program with:

gcc -pthread -I/usr/lib/x86_64-linux-gnu/openmpi/include -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi main.c /usr/lib/x86_64-linux-gnu/libmpi.so.40

The pkg-config --libs mpi is broken BTW, the package forgot to provide a symlink from libmpi.so.40 to libmpi.so.

The mpich version pkg-config does work directly however.

I'll try develop next.

[prj-]

This is not a PETSc problem. This is a FreeFEM problem. Please send your FreeFEM config.log. If you had OpenMPI properly installed, then it should have been passed to PETSc, and you shouldn't have empty wrappers passed.

Also, this shouldn't be make petsc-slepc MPICC=/usr/bin/gcc MPICXX=/usr/bin/g++ MPIFC=/usr/bin/gfortran but make petsc-slepc MPICC=mpicc MPICXX=mpic++ MPIFC=mpif90. You are using sequential compiler, you have to pass MPI wrappers. But again, it should be fixed in FreeFEM ./configure instead.

Also, please do not have both MPICH and OpenMPI installed, this is a terrible idea, unless you exactly know what you are doing.

[cirosantilli]

config.log: https://gist.github.com/cirosantilli/601c5b1b58827516f0e66c600d552890

3rdparty/ff-petsc/config.log

## ---------------------- ##
## Running config.status. ##
## ---------------------- ##

This file was extended by FreeFEM config.status 4.6, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  CONFIG_FILES    = 
  CONFIG_HEADERS  = 
  CONFIG_LINKS    = 
  CONFIG_COMMANDS = 
  $ ../../config.status --file=Makefile.inc:Makefile-PETSc.inc

on ciro-p51

config.status:1218: creating Makefile.inc

[prj-]

~This is not FreeFEM config.log... this is 3rdparty/ff-petsc/config.log. Look in your config.log, you'll see that MPI is not detected, and please post it here as well.~

Thanks, I'll look into that.

[cirosantilli]

I also posted config.log at https://gist.github.com/cirosantilli/601c5b1b58827516f0e66c600d552890 it's not that one?

[prj-]

Yup, there is something messed up with your MPI wrappers.

configure:12788: eval /usr/bin/mpic++ -o conftest -g  -DNDEBUG -O3 -mmmx -mavx -std=c++11 -DBAMG_LONG_LONG  -DNCHECKPTR   conftest.cpp -lrt   >&5
/usr/bin/ld: cannot find -lmpi

And later on.

configure:23475:   --  with MPI:     no

You are not the only one having difficulties with OpenMPI/MPICH from Ubuntu 20.04, cf. this post from the forum. I'd suggest removing all MPI packages, and letting PETSc install MPICH itself.

[frederichecht]

Dear all,

I think it is crazy to set

MPICC=/usr/bin/gcc MPICXX=/usr/bin/g++ MPIFC=/usr/bin/gfortran

the MPICC, MPICXX and MPIFC parable are the wrapper path of MPI compiler.

remark, to no the moi use by freefem after configure

you can do a grep in new Makefile

peu-cp2:ff-petsc hecht$ grep MPI Makefile| grep = COMPILE_OPENBLAS = DOWNLOADCOMPILE = compile-pkg install-other LOAD_COMPILE = load_compile MPICC = /usr/local/ff++/mpich-3.3.1/bin/mpicc MPICXX = /usr/local/ff++/mpich-3.3.1/bin/mpic++ MPIF77 = /usr/local/ff++/mpich-3.3.1/bin/mpif90 MPIFC = /usr/local/ff++/mpich-3.3.1/bin/mpif90 MPIPROG = FreeFem++-mpi MPIRUN = /usr/local/ff++/mpich-3.3.1/bin/mpiexec MPISCRIPT = ff-mpirun MPI_INCLUDE = -I/usr/local/ff++/mpich-3.3.1/include MPI_INC_DIR = /usr/local/ff++/mpich-3.3.1/include MPI_LIB = -Wl,-flat_namespace -Wl,-commons,use_dylibs -L/usr/local/ff++/mpich-3.3.1/lib -lmpicxx -lmpi -lpmpi MPI_LIBC = MPI_LIBFC = -Wl,-flat_namespace -L/usr/local/ff++/mpich-3.3.1/lib -lmpifort -lmpi -lpmpi MPI_LIB_DIRS = /usr/local/ff++/mpich-3.3.1/lib MPI_RUN_OPTION = SKIP_TESTS_MPI = no

Le 9 juin 2020 à 16:27, prj- notifications@github.com a écrit :

This is not a PETSc problem. This is a FreeFEM problem. Please send your FreeFEM config.log. If you had OpenMPI properly installed, then it should have been passed to PETSc, and you shouldn't have empty wrapper passed.

Also, this shouldn't be make petsc-slepc MPICC=/usr/bin/gcc MPICXX=/usr/bin/g++ MPIFC=/usr/bin/gfortran but make petsc-slepc MPICC=mpicc MPICXX=mpic++ MPIFC=mpif90. You are using sequential compiler, you have to pass MPI wrappers. But again, it should be fixed in FreeFEM ./configure instead.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/FreeFem/FreeFem-sources/issues/141#issuecomment-641333719, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLPNDCQEAPFSZDOFUFMU2LRVZBGVANCNFSM4NZG2EUA.

[cirosantilli]

OK, Ubuntu mpic++ is completely broken :-)

I now tried on develop at 9e8f0fa978aee608ab3cc5e9bf6c345610cd37e2

It progressed much further, but it failed later on with:

*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Error configuring METIS with cmake                        
*******************************************************************************

Full config.log https://gist.github.com/cirosantilli/601c5b1b58827516f0e66c600d552890#file-config2-log

3rdparty/ff-petsc/petsc-3.13.2/configure.log: https://gist.github.com/cirosantilli/601c5b1b58827516f0e66c600d552890#file-ff-petsc-config-log

/home/ciro/test/FreeFem-install/ff-petsc/r/bin/mpicc is present, maybe it does not work with ./configure --prefix?

[cirosantilli]

I wasn't very familiar how gist handles large files, github added a message to me saying I can view it in full at: https://gist.githubusercontent.com/cirosantilli/601c5b1b58827516f0e66c600d552890/raw/f66dc09cb7b5728c15e1ec427ca77892c3cbe796/ff-petsc-config.log and that appears to be the full file. Let me know if you can't see the full thing there.

cmake version 3.16.3.

[prj-]

OK, I see the issue, I'll get in touch with the PETSc developers. In the meantime, could you please comment/remove this line and start over? Thanks for your patience :)

By start over, I mean delete the folder petsc-3.13.2 and relaunch make petsc-slepc.

[cirosantilli]

With that change PETSc build appears to have worked, but then it starts running some tests to see if build was correct, and is stuck at:

C/C++ example src/snes/tutorials/ex19 run successfully with mumps

for one hour thirty. ps shows a process running all along:

./ex19 -da_refine 3 -snes_monitor_short -pc_type lu -pc_factor_mat_solver_type umfpack

Is this normal? How long does it take for you?

If it were possible to use a prebuilt PETSc from the package system, that would be awesome :-) related https://github.com/FreeFem/FreeFem-sources/issues/128 Ubuntu 20.04 is at PETSc 3.12 which is not old.

[prj-]

No, that's not normal, but you can safely skip that. If you need complex PETSc, do make petsc-3.13.2/tag-install-complex COMPLEX_CASE=1, and then in the top directoy ./reconfigure and you should be good to go.

You can of course use a prebuilt PETSc, cf. the instructions from the issue you mentioned.

It seems your previous issue comes from the fact that you have the nvcc compiler in your path, this should be fixed soon, cf. this PETSc GitLab MR.

[prj-]

I've disabled CUDA in PETSc for now, it should be much easier for you to get through the compilation process.

[cirosantilli]

Thanks for looking further into this.

I tried make clean;make petsc-3.13.2/tag-install-complex COMPLEX_CASE=1 but it still got stuck on that test.

BTW, I ctrl+c kill the run and then try to proceed with the freefem build, I got this (but might be due to failed previous step?):

SuperLu.cpp:29:10: fatal error: slu_ddefs.h: No such file or directory                                                                  
   29 | #include "slu_ddefs.h"                          
      |          ^~~~~~~~~~~~~                                      
compilation terminated.                                      
make[4]: *** [Makefile:894: SuperLu.so] Error 1 

The file /home/ciro/test/FreeFem-install/ff-petsc/r/include/slu_ddefs.h exists.

[prj-]

Could you please post your FreeFEM config.log after having ./reconfigure? I'm now realizing that before making complex PETSc, FreeFEM does a make check, which in your case fails. So you need to comment this line first. I've explicitly deactivated CUDA in https://github.com/FreeFem/FreeFem-sources/commit/1450c2f19c4d0048887fdf74c27cf35774995b30, so I hope these problems won't appear if you restart from scratch.

[cirosantilli]

This is the config.log: https://gist.githubusercontent.com/cirosantilli/601c5b1b58827516f0e66c600d552890/raw/5abcb5db3dcb98f63e8a9047e5fa529ef1779d46/reconfigure

[prj-]

You still have /usr/bin/mpic++ in your path. Mixing MPI implementations is kind of a bad idea, as highlighted before. Any chance you could completely remove your previous MPI installation?

configure:13798: eval /usr/bin/mpic++ -o conftest -g  -DNDEBUG -O3 -mmmx -mavx -std=c++11 -DBAMG_LONG_LONG  -DNCHECKPTR   conftest.cpp -lrt   >&5
conftest.cpp:31:10: fatal error: mpi.h: No such file or directory
   31 | #include <mpi.h>
      |          ^~~~~~~

[frederichecht]

Dear all, The simple idée is to add the the PATH a the beginning the directory where is mpiexec commande. like:

1) find the directory where mpiexec is installed:

peu-cp2:ff-petsc hecht$ ls /usr/local/ff++/mpich-3.3.1/bin/mpic++ /usr/local/ff++/mpich-3.3.1/bin/mpic++

2) add dir /usr/local/ff++/mpich-3.3.1/bin to your path.

peu-cp2:ff-petsc hecht$ export PATH="/usr/local/ff++/mpich-3.3.1/bin/:$PATH"

3) check which mpiexec is used:
peu-cp2:ff-petsc hecht$ which mpiexec /usr/local/ff++/mpich-3.3.1/bin//mpiexec

Best Regards,

Frédéric Hecht.

---

Laboratoire Jacques-Louis Lions, UPMC Sorbonne Université BC187, 4 Place Jussieu, 75252 PARIS cedex 05, France Campus Jussieu, Barre 15-25, 3 etage Bureau 307 Projet Alpines , Inria de Paris, 2 rue Simone Iff Voie DQ12 75012 Paris tel: +33 1 44274411, mob: +33 6 62198986, fax: +33 1 44277200 frederic.hecht@sorbonne-universite.fr https://www.ljll.math.upmc.fr/hecht software: FreeFem++ web site: http://www.freefem.org/ff++

Le 10 juin 2020 à 09:46, cirosantilli notifications@github.com a écrit :

This is the config.log: https://gist.githubusercontent.com/cirosantilli/601c5b1b58827516f0e66c600d552890/raw/5abcb5db3dcb98f63e8a9047e5fa529ef1779d46/reconfigure https://gist.githubusercontent.com/cirosantilli/601c5b1b58827516f0e66c600d552890/raw/5abcb5db3dcb98f63e8a9047e5fa529ef1779d46/reconfigure — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/FreeFem/FreeFem-sources/issues/141#issuecomment-641802178, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLPNDHVHWMQTLRU7YNHHY3RV424HANCNFSM4NZG2EUA.

[prj-]

You actually don't need to set the PATH variable since https://github.com/FreeFem/FreeFem-sources/commit/dffba71bb1e1b20262c1e586efa05c2db0c26f19. I was previously only getting the MPIRUN variable from PETSc, and your MPICXX variable was pointing to the faulty /usr/bin/mpic++ wrapper. See the config.log:

configure:13442: checking for MPIRUN
configure:13501: result: /home/ciro/test/FreeFem-install/ff-petsc/r/bin/mpiexec <-- good
[...]
configure:13775: checking for MPICXX
configure:13798: eval /usr/bin/mpic++ -o conftest -g  -DNDEBUG -O3 -mmmx -mavx -std=c++11 -DBAMG_LONG_LONG  -DNCHECKPTR   conftest.cpp -lrt   >&5 <-- bad

So, now, you don't need to uninstall your system MPI. You can git pull the develop branch, autoreconf -i && ./reconfigure. If this still fails, please do upload a newly generated config.log.

Thanks for all the troubleshooting!

[frederichecht]

Yes, but it is more simple. to set the path variable and used the good mpi.

Le 10 juin 2020 à 10:20, prj- notifications@github.com a écrit :

You actually don't need to set the PATH variable since dffba71 https://github.com/FreeFem/FreeFem-sources/commit/dffba71bb1e1b20262c1e586efa05c2db0c26f19. I was previously only getting the MPIRUN variable from PETSc, and your MPICXX variable was pointing to the faulty /usr/bin/mpic++ wrapper. See the config.log:

configure:13442: checking for MPIRUN configure:13501: result: /home/ciro/test/FreeFem-install/ff-petsc/r/bin/mpiexec <-- good [...] configure:13775: checking for MPICXX configure:13798: eval /usr/bin/mpic++ -o conftest -g -DNDEBUG -O3 -mmmx -mavx -std=c++11 -DBAMG_LONG_LONG -DNCHECKPTR conftest.cpp -lrt >&5 <-- bad So, now, you don't need to uninstall your system MPI. You can git pull the develop branch, autoreconf -i && ./reconfigure. If this still fails, please do upload a newly generated config.log.

Thanks for all the troubleshooting!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/FreeFem/FreeFem-sources/issues/141#issuecomment-641827615, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLPNDDZSGUM6XNUGD3WLDDRV4633ANCNFSM4NZG2EUA.

[cirosantilli]

I nuked Ubuntu openmpi, pulled FreeFem to 3df0e2370d9752801ac744b11307b14e16743a44 applied:

diff --git a/3rdparty/ff-petsc/Makefile b/3rdparty/ff-petsc/Makefile
index dc62ab06..13cd3253 100644
--- a/3rdparty/ff-petsc/Makefile
+++ b/3rdparty/ff-petsc/Makefile
@@ -204,7 +204,7 @@ $(SRCDIR)/tag-make-real:$(SRCDIR)/tag-conf-real
 $(SRCDIR)/tag-install-real :$(SRCDIR)/tag-make-real
    cd $(SRCDIR) && $(MAKE) PETSC_DIR=$(PETSC_DIR) PETSC_ARCH=fr install
    -test -x "`type -p otool`" && make changer
-   cd $(SRCDIR) && $(MAKE) PETSC_DIR=$(PETSC_DIR) PETSC_ARCH=fr check
+   #cd $(SRCDIR) && $(MAKE) PETSC_DIR=$(PETSC_DIR) PETSC_ARCH=fr check
    test -e $(DIR_INSTALL_REAL)/include/petsc.h
    test -e $(DIR_INSTALL_REAL)/lib/petsc/conf/petscvariables
    touch $@
@@ -293,7 +293,6 @@ $(SRCDIR)/tag-tar:$(PACKAGE)
    -tar xzf $(PACKAGE)
    patch -p1 < petsc-hpddm.patch
 ifeq ($(WIN32DLLTARGET),)
-   patch -p1 < petsc-metis.patch
 endif
    touch $@
 $(PACKAGE):

and now it worked!!!

cd ../FreeFem-install
PATH="${PATH}:$(pwd)/bin" ./bin/FreeFem++ tmp.pde

Thanks so much for your help, feel free to close if you feel that all the problems found have been addressed.

If possible, I'd highly recommend adding instructions to build with Ubuntu packaged PETSc to README :-)

[prj-]

That's good news :) With the commit you fetched, I actually don't believe there is a need to remove the petsc-metis.patch command, and I'm guessing the SuiteSparse test will not fail anymore as well. Deactivating CUDA is the proper way to go (for now), I think, and that's what I did by adding the --with-cudac=0 flag. But please do let me know if this is not the case. Anyway, I'll close the issue for now, do not hesitate re-opening it if you face new problems.