Parmetis build - Githubissues

michel2323 commented 5 years ago

Removed the code in PIPS that had METIS dependencies.
If PIPS is built using MUMPS, ParMETIS is a requirement.
ParMETIS is detected and linked using cmake
Sample Makefile.inc for MUMPS
Build script wgetmetis.sh pulls parmetis (which includes metis)

cnpetra commented 5 years ago

@jalving : does this fix your problems with solving PIPS with MUMPS ?

cnpetra commented 5 years ago

@michel2323 : I cannot push to your branch so I'll leave comments "inline"

cnpetra commented 5 years ago

PIPS-IPM does not build -> StochResourcePlanner.C: fatal error: defs.h: No such file or directory

jalving commented 5 years ago

@cnpetra I am getting a new error, but I think it has to do with how I built MUMPS on my machine.
Here is what I get when I get to the MUMPS stuff:

[ 88%] Linking CXX executable parmodelLink
/opt/PIPS_parmetis/ThirdPartyLibs/MUMPS/lib/libmumps_common.a(mumps_metis.o): In function `mumps_parmetis_':
mumps_metis.c:(.text+0x45): undefined reference to `MPI_Comm_f2c'
/opt/PIPS_parmetis/ThirdPartyLibs/MUMPS/lib/libmumps_common.a(mumps_metis64.o): In function `mumps_parmetis_64_':
mumps_metis64.c:(.text+0xc): undefined reference to `MPI_Comm_f2c'

followed by a bunch of undefined reference statements (below). I was getting these while trying to build MUMPS at first too.

/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libmpi_mpifh.so: undefined reference to `mpi_fortran_in_place__'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libmpi_mpifh.so: undefined reference to `MPI_Win_f2c'

For MUMPS, I ended up adding the lines:

INCPAR = -I/usr/lib/openmpi/include
LIBPAR = $(SCALAP) $(LAPACK) -L/usr/lib/openmpi/lib -lmpi -lmpi_mpifh

to my Makefile.inc for MUMPS. I might just need to make these openmpi libraries visible from the library path while building PIPS-NLP?

I'm building PIPS-NLP with mpich. Would that cause any issues here?

cnpetra commented 5 years ago

@cnpetra I am getting a new error, but I think it has to do with how I built MUMPS on my machine. Here is what I get when I get to the MUMPS stuff:
[ 88%] Linking CXX executable parmodelLink
/opt/PIPS_parmetis/ThirdPartyLibs/MUMPS/lib/libmumps_common.a(mumps_metis.o): In function `mumps_parmetis_':
mumps_metis.c:(.text+0x45): undefined reference to `MPI_Comm_f2c'
/opt/PIPS_parmetis/ThirdPartyLibs/MUMPS/lib/libmumps_common.a(mumps_metis64.o): In function `mumps_parmetis_64_':
mumps_metis64.c:(.text+0xc): undefined reference to `MPI_Comm_f2c'
followed by a bunch of undefined reference statements (below). I was getting these while trying to build MUMPS at first too.
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libmpi_mpifh.so: undefined reference to `mpi_fortran_in_place__'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/libmpi_mpifh.so: undefined reference to `MPI_Win_f2c'
For MUMPS, I ended up adding the lines:
INCPAR = -I/usr/lib/openmpi/include
LIBPAR = $(SCALAP) $(LAPACK) -L/usr/lib/openmpi/lib -lmpi -lmpi_mpifh
to my Makefile.inc for MUMPS. I might just need to make these openmpi libraries visible from the library path while building PIPS-NLP?

I'm building PIPS-NLP with mpich. Would that cause any issues here?

isn't because you built MUMPS with OpenMPI and you're building PIPS with mpich?

BTW, if you're using mpich, then you have to use -lmpifort instead of -lmpi_mpifh, at least that's how I have to do it on my redhad linux.

jalving commented 5 years ago

@cnpetra I think that is the issue. I have only ever built PIPS using mpich, but I can't get MUMPS to build with mpich. Does SCALAPACK also need to be built with mpich?

I still have:

SCALAP = -lscalapack-openmpi -lblacs-openmpi -lblacsF77init-openmpi -lblacsCinit-openmpi

I was never ever to find a library built against mpich.

cnpetra commented 5 years ago

@cnpetra I think that is the issue. I have only ever built PIPS using mpich, but I can't get MUMPS to build with mpich. Does SCALAPACK also need to be built with mpich?

I still have:

SCALAP = -lscalapack-openmpi -lblacs-openmpi -lblacsF77init-openmpi -lblacsCinit-openmpi

I was never ever to find a library built against mpich.

I guess everything has to be built with either mpich or openmpi. PIPS builds with openmpi.

michel2323 commented 5 years ago

1. PIPS-IPM does not build -> StochResourcePlanner.C: fatal error: defs.h: No such file or directory

Fixed

michel2323 commented 5 years ago

I tried to fix the -lmpi_mpifh and -lmpifort with OpenMPI vs MPICH issue by using find_package( MPI ), but it would just return the right linking even when using MPI_Fortran_LINK_FLAGS . So as is this belongs to the MUMPS flags that have to be set manually by the user.

jalving commented 5 years ago

@cnpetra Looks like SCALAPACK and BLACS also need the same openmpi version to build MUMPS (makes sense). I think my issues will be resolved when I get my environment setup correctly and stop using my system default libraries. I'll ultimately need to go build everything from scratch.

cnpetra commented 5 years ago

@jalving: yes, I am pretty sure this PR will work for you

michel2323 commented 5 years ago

@cnpetra Do not merge. There is an issue.

cnpetra commented 5 years ago

closing and merging this since it fixes the parmetis build problem and MPI "nodal" communicator issue will be address in a separate PR.

Argonne-National-Laboratory / PIPS

Parmetis build #46