abhyshr
commented
1 week ago What do you have for PETSC_INCLUDE_DIR in build/CMakeCache.txt? Can you send this file? @cameronrutherford @jaelynlitz
Open fbonhoff opened 1 week ago
abhyshr
commented
1 week ago What do you have for PETSC_INCLUDE_DIR in build/CMakeCache.txt? Can you send this file? @cameronrutherford @jaelynlitz
fbonhoff
commented
1 week ago Of course!
I attached the file and also copied out the petsc relevant parts:
PETSC_CFLAGS:INTERNAL=-I/home/bof/libraries/include/petsc-3.16.0/install/include PETSC_CFLAGS_I:INTERNAL= PETSC_CFLAGS_OTHER:INTERNAL= PETSC_FOUND:INTERNAL=1 PETSC_INCLUDEDIR:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install/include PETSC_INCLUDE_DIRS:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install/include PETSC_LDFLAGS:INTERNAL=-L/home/bof/libraries/include/petsc-3.16.0/install/lib;-lpetsc PETSC_LDFLAGS_OTHER:INTERNAL= PETSC_LIBDIR:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install/lib PETSC_LIBRARIES:INTERNAL=petsc PETSC_LIBRARY_DIRS:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install/lib PETSC_LIBS:INTERNAL= PETSC_LIBS_L:INTERNAL= PETSC_LIBS_OTHER:INTERNAL= PETSC_LIBS_PATHS:INTERNAL= PETSC_MODULE_NAME:INTERNAL=PETSc PETSC_PETSc_INCLUDEDIR:INTERNAL= PETSC_PETSc_LIBDIR:INTERNAL= PETSC_PETSc_PREFIX:INTERNAL= PETSC_PETSc_VERSION:INTERNAL= PETSC_PREFIX:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install PETSC_STATIC_CFLAGS:INTERNAL=-I/home/bof/libraries/include/petsc-3.16.0/install/include PETSC_STATIC_CFLAGS_I:INTERNAL= PETSC_STATIC_CFLAGS_OTHER:INTERNAL= PETSC_STATIC_INCLUDE_DIRS:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install/include PETSC_STATIC_LDFLAGS:INTERNAL=-L/home/bof/libraries/include/petsc-3.16.0/install/lib;-L/usr/lib/gcc/x86_64-linux-gnu/11;-lpetsc;-lflapack;-lfblas;-lm;-lX11;-lstdc++;-ldl;-lmpifort;-lmpi;-lgfortran;-lm;-lgfortran;-lm;-lgcc_s;-lquadmath;-lstdc++;-ldl PETSC_STATIC_LDFLAGS_OTHER:INTERNAL= PETSC_STATIC_LIBDIR:INTERNAL= PETSC_STATIC_LIBRARIES:INTERNAL=petsc;flapack;fblas;m;X11;stdc++;dl;mpifort;mpi;gfortran;m;gfortran;m;gcc_s;quadmath;stdc++;dl PETSC_STATIC_LIBRARY_DIRS:INTERNAL=/home/bof/libraries/include/petsc-3.16.0/install/lib;/usr/lib/gcc/x86_64-linux-gnu/11 PETSC_STATIC_LIBS:INTERNAL= PETSC_STATIC_LIBS_L:INTERNAL= PETSC_STATIC_LIBS_OTHER:INTERNAL= PETSC_STATIC_LIBS_PATHS:INTERNAL= PETSC_VERSION:INTERNAL=3.16.0
cameronrutherford
commented
1 week ago I am a little confused about what's going wrong here. If possible, I would suggest in updating PETSc to a newer version, and running with VERBOSE=1 make to get more detailed logs.
Since you aren't building with Ipopt, and you are running on a linux box, have you considered using our ghcr package to get started? You can try pulling this version of the ExaGO package for example.
Based on our .devcontainer/Dockerfile, something like this in a Dockerfile should get you started:
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-jammy as builder
# Install OS packages needed to build the software
RUN apt-get -yqq update && apt-get -yqq upgrade \
&& apt-get -yqq install autoconf \
&& rm -rf /var/lib/apt/lists/*
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo spack: \
&& echo ' specs:' \
&& echo ' - exago@1.6.0~ipopt+python+mpi' \
&& echo ' ^libffi@3.4.2' \
&& echo ' ^petsc~fortran~hdf5~hypre+metis' \
&& echo ' concretizer:' \
&& echo ' unify: true' \
&& echo ' reuse: true' \
&& echo ' mirrors:' \
&& echo ' spack: https://binaries.spack.io/develop' \
&& echo ' packages:' \
&& echo ' all:' \
&& echo ' providers:' \
&& echo ' mpi:' \
&& echo ' - openmpi' \
&& echo ' zlib-api:' \
&& echo ' - zlib' \
&& echo ' zlib-ng:' \
&& echo ' buildable: false' \
&& echo ' mpich:' \
&& echo ' buildable: false' \
&& echo ' config:' \
&& echo ' install_tree: /opt/software' \
&& echo ' view: /opt/views/view') > /opt/spack-environment/spack.yaml
# Find external packages
RUN cd /opt/spack-environment && spack env activate . && spack external find --all --exclude python
# Do this separate of install to cache keys...
RUN cd /opt/spack-environment && spack env activate . && spack mirror add develop https://binaries.spack.io/develop && spack buildcache keys --install --trust && spack concretize -f && ( spack mirror create -a || true )
# Install buggy linux-uuid package
RUN cd /opt/spack-environment && spack env activate . && spack install --no-checksum --fail-fast util-linux-uuid
# Install PETSc
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y
# Strip all the binaries
RUN find -L /opt/views/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . > activate.sh
RUN { \
echo '#!/bin/sh' \
&& echo '.' /opt/spack-environment/activate.sh \
&& echo 'exec "$@"'; \
} > /entrypoint.sh \
&& chmod a+x /entrypoint.sh \
&& ln -s /opt/views/view /opt/view
RUN apt-get -yqq update && apt-get -yqq upgrade \
&& apt-get -yqq install gfortran \
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT [ "/entrypoint.sh" ]
CMD [ "/bin/bash" ]
# Make sure root user gets spack packages
RUN echo "source /entrypoint.sh" >> /root/.bashrcHi Cameron, haven't considered but I will try. I am specifically using that version of PETsc as it was mentioned in the dependencies and I couldn't figure out where the errors came from using the latest version.
I have not really used docker before, but probably something that I should do anyway so I will see how far I get with that, thanks!
fbonhoff
commented
1 week ago Running the above as a Dockefile fails while installing PETsc and/or Blas:
I saved the script in Dockerfile, and with docker installed I ran:
docker build -t spack-environment .
=> ERROR [ 7/13] RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc 203.7s
------
> [ 7/13] RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc:
2.762 [+] /usr (external diffutils-3.8-enmvrffztilwxiikkrb6ecvy5hsjb56b)
2.762 [+] /usr (external glibc-2.35-kpqgx3w6iph5fz3t3bwf66r3a2zlblv4)
2.762 [+] /usr (external glibc-2.27-yw5erx4enja73o74rurllfukltkfjtzt)
2.762 [+] /usr (external curl-7.81.0-2t2wsy7vky7ld6wf2lvvn7osdjtwz732)
2.762 [+] /usr (external perl-5.34.0-dgirojtbssabs6o6udwbfodykrithqnq)
2.762 [+] /usr (external autoconf-2.71-qjwbrrz66fjbtmyyvz7uoal6lk5kjaen)
2.762 [+] /usr (external automake-1.16.5-6k34gurkwqdf2xhql665tpfmvai2irfe)
2.762 [+] /usr (external findutils-4.8.0-spqruwhfeqf4lnkpoz37k26vybh6eaob)
2.762 [+] /usr (external m4-1.4.18-6g7hqnfvtkqwgf5fkdnanehxc6orv3ea)
2.762 [+] /usr (external xz-5.2.5-kemc5sdpotneyvzffbjent5dlxjvyttd)
2.762 [+] /usr (external openssl-3.0.2-btqjat2hi4z2idnnjcdx6l65wp7i5kes)
2.762 [+] /usr (external bzip2-1.0.8-nftzrmlqhx7fpux4ljkko7jckoa7oedc)
2.762 [+] /usr (external tar-1.34-c326xrdcga5uxzcaxnwzbrk5r3td46sq)
2.762 [+] /opt/software/linux-ubuntu22.04-x86_64/gcc-11.4.0/gcc-runtime-11.4.0-v2ijgukwrehpr6hdzre65vab6npz2nxl
2.762 [+] /opt/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-runtime-7.5.0-wfzklsvdan4cn25cyn5qqxm2wwywcet2
2.762 [+] /opt/software/linux-ubuntu22.04-x86_64/gcc-11.4.0/util-linux-uuid-2.38.1-v4h7c57cxrnrl6nnns7ua67vc6e7yyyh
2.762 [+] /opt/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/gmake-4.4.1-6wourj6ffjsxgb3xjb2pl4dij446y5w3
2.762 ==> Installing openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427 [18/48]
2.762 ==> No binary for openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427 found: installing from source
3.891 ==> Using cached archive: /opt/spack/var/spack/cache/_source-cache/archive/4e/4e6e4f5cb14c209262e33e6816d70221a2fe49eb69eaf0a06f065598ac602c68.tar.gz
3.891 ==> No patches needed for openblas
3.895 ==> openblas: Executing phase: 'edit'
3.906 ==> openblas: Executing phase: 'build'
201.4 ==> Error: ProcessError: Command exited with status 2:
201.4 'make' '-j16' '-s' 'CC=/opt/spack/lib/spack/env/gcc/gcc' 'FC=/opt/spack/lib/spack/env/gcc/gfortran' 'MAKE_NB_JOBS=0' 'ARCH=x86_64' 'DYNAMIC_ARCH=1' 'DYNAMIC_OLDER=1' 'TARGET=GENERIC' 'DYNAMIC_ARCH=1' 'USE_LOCKING=1' 'USE_OPENMP=0' 'USE_THREAD=0' 'RANLIB=ranlib' 'all'
201.5
201.5 2 errors found in build log:
201.5 2853 Backtrace for this error:
201.5 2854 #0 0x7f7eafef7960 in ???
201.5 2855 #1 0x7f7eafef6ac5 in ???
201.5 2856 #2 0x7f7eafcea51f in ???
201.5 2857 #3 0x560588809208 in sgemm_oncopy_OPTERON_SSE3
201.5 2858 Illegal instruction (core dumped)
201.5 >> 2859 make[1]: *** [Makefile:167: level3] Error 132
201.5 2860 make[1]: *** Waiting for unfinished jobs....
201.5 >> 2861 make: *** [Makefile:160: tests] Error 2
201.5
201.5 See build log for details:
201.5 /tmp/root/spack-stage/spack-stage-openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427/spack-build-out.txt
201.5
201.5 ==> Warning: Skipping build of petsc-3.21.1-zduwixjogwl547jluo7qhnh22246zxsy since openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427 failed
201.6 ==> Updating view at /opt/views/view
201.6 ==> Error: Terminating after first install failure: ProcessError: Command exited with status 2:
201.6 'make' '-j16' '-s' 'CC=/opt/spack/lib/spack/env/gcc/gcc' 'FC=/opt/spack/lib/spack/env/gcc/gfortran' 'MAKE_NB_JOBS=0' 'ARCH=x86_64' 'DYNAMIC_ARCH=1' 'DYNAMIC_OLDER=1' 'TARGET=GENERIC' 'DYNAMIC_ARCH=1' 'USE_LOCKING=1' 'USE_OPENMP=0' 'USE_THREAD=0' 'RANLIB=ranlib' 'all'
------
Dockerfile:47
--------------------
45 |
46 | # Install PETSc
47 | >>> RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc
48 |
49 | # Install the software, remove unnecessary deps
--------------------
ERROR: failed to solve: process "docker-shell cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc" did not complete successfully: exit code: 1The linked docker image works though.. Is there also an available docker image including GPU support ? And if so, how can I find the available images?
Thank you already though to get me started!
cameronrutherford
commented
1 day ago The linked docker image works though.. Is there also an available docker image including GPU support ? And if so, how can I find the available images?
Thank you already though to get me started!
We don't currently have a docker image that we build / publish with GPU support unfortunately. We really need CoinHSL in order to get most of ExaGO's complex applications functioning, so this for a while has blocked us supporting builds for SCOPFLOW / SOPFLOW and GPU builds out of the box.
I suppose we could go a ghcr image for CUDA / ROCm as well, but again they wouldn't have SOPFLOW / SCOPFLOW supported...
cameronrutherford
commented
1 day ago Running the above as a Dockefile fails while installing PETsc and/or Blas: I saved the script in Dockerfile, and with docker installed I ran:
docker build -t spack-environment .=> ERROR [ 7/13] RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc 203.7s ------ > [ 7/13] RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc: 2.762 [+] /usr (external diffutils-3.8-enmvrffztilwxiikkrb6ecvy5hsjb56b) 2.762 [+] /usr (external glibc-2.35-kpqgx3w6iph5fz3t3bwf66r3a2zlblv4) 2.762 [+] /usr (external glibc-2.27-yw5erx4enja73o74rurllfukltkfjtzt) 2.762 [+] /usr (external curl-7.81.0-2t2wsy7vky7ld6wf2lvvn7osdjtwz732) 2.762 [+] /usr (external perl-5.34.0-dgirojtbssabs6o6udwbfodykrithqnq) 2.762 [+] /usr (external autoconf-2.71-qjwbrrz66fjbtmyyvz7uoal6lk5kjaen) 2.762 [+] /usr (external automake-1.16.5-6k34gurkwqdf2xhql665tpfmvai2irfe) 2.762 [+] /usr (external findutils-4.8.0-spqruwhfeqf4lnkpoz37k26vybh6eaob) 2.762 [+] /usr (external m4-1.4.18-6g7hqnfvtkqwgf5fkdnanehxc6orv3ea) 2.762 [+] /usr (external xz-5.2.5-kemc5sdpotneyvzffbjent5dlxjvyttd) 2.762 [+] /usr (external openssl-3.0.2-btqjat2hi4z2idnnjcdx6l65wp7i5kes) 2.762 [+] /usr (external bzip2-1.0.8-nftzrmlqhx7fpux4ljkko7jckoa7oedc) 2.762 [+] /usr (external tar-1.34-c326xrdcga5uxzcaxnwzbrk5r3td46sq) 2.762 [+] /opt/software/linux-ubuntu22.04-x86_64/gcc-11.4.0/gcc-runtime-11.4.0-v2ijgukwrehpr6hdzre65vab6npz2nxl 2.762 [+] /opt/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-runtime-7.5.0-wfzklsvdan4cn25cyn5qqxm2wwywcet2 2.762 [+] /opt/software/linux-ubuntu22.04-x86_64/gcc-11.4.0/util-linux-uuid-2.38.1-v4h7c57cxrnrl6nnns7ua67vc6e7yyyh 2.762 [+] /opt/software/linux-ubuntu18.04-x86_64/gcc-7.5.0/gmake-4.4.1-6wourj6ffjsxgb3xjb2pl4dij446y5w3 2.762 ==> Installing openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427 [18/48] 2.762 ==> No binary for openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427 found: installing from source 3.891 ==> Using cached archive: /opt/spack/var/spack/cache/_source-cache/archive/4e/4e6e4f5cb14c209262e33e6816d70221a2fe49eb69eaf0a06f065598ac602c68.tar.gz 3.891 ==> No patches needed for openblas 3.895 ==> openblas: Executing phase: 'edit' 3.906 ==> openblas: Executing phase: 'build' 201.4 ==> Error: ProcessError: Command exited with status 2: 201.4 'make' '-j16' '-s' 'CC=/opt/spack/lib/spack/env/gcc/gcc' 'FC=/opt/spack/lib/spack/env/gcc/gfortran' 'MAKE_NB_JOBS=0' 'ARCH=x86_64' 'DYNAMIC_ARCH=1' 'DYNAMIC_OLDER=1' 'TARGET=GENERIC' 'DYNAMIC_ARCH=1' 'USE_LOCKING=1' 'USE_OPENMP=0' 'USE_THREAD=0' 'RANLIB=ranlib' 'all' 201.5 201.5 2 errors found in build log: 201.5 2853 Backtrace for this error: 201.5 2854 #0 0x7f7eafef7960 in ??? 201.5 2855 #1 0x7f7eafef6ac5 in ??? 201.5 2856 #2 0x7f7eafcea51f in ??? 201.5 2857 #3 0x560588809208 in sgemm_oncopy_OPTERON_SSE3 201.5 2858 Illegal instruction (core dumped) 201.5 >> 2859 make[1]: *** [Makefile:167: level3] Error 132 201.5 2860 make[1]: *** Waiting for unfinished jobs.... 201.5 >> 2861 make: *** [Makefile:160: tests] Error 2 201.5 201.5 See build log for details: 201.5 /tmp/root/spack-stage/spack-stage-openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427/spack-build-out.txt 201.5 201.5 ==> Warning: Skipping build of petsc-3.21.1-zduwixjogwl547jluo7qhnh22246zxsy since openblas-0.3.26-ddf7dkrpyb6vxm3dlg3ztxerismsb427 failed 201.6 ==> Updating view at /opt/views/view 201.6 ==> Error: Terminating after first install failure: ProcessError: Command exited with status 2: 201.6 'make' '-j16' '-s' 'CC=/opt/spack/lib/spack/env/gcc/gcc' 'FC=/opt/spack/lib/spack/env/gcc/gfortran' 'MAKE_NB_JOBS=0' 'ARCH=x86_64' 'DYNAMIC_ARCH=1' 'DYNAMIC_OLDER=1' 'TARGET=GENERIC' 'DYNAMIC_ARCH=1' 'USE_LOCKING=1' 'USE_OPENMP=0' 'USE_THREAD=0' 'RANLIB=ranlib' 'all' ------ Dockerfile:47 -------------------- 45 | 46 | # Install PETSc 47 | >>> RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc 48 | 49 | # Install the software, remove unnecessary deps -------------------- ERROR: failed to solve: process "docker-shell cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc" did not complete successfully: exit code: 1
That's quite the ugly error... If you add a copy of the line RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast petsc with opeblas instead of petsc, and then specify the openblas version in your spack: spec in the dockerfile with ^openblas@version, you should be able to iterate through a few openblas versions (listed here) to see if it's just the latest version that's buggy.
If you can't get anywhere with that, it might be worth reporting a bug with the openblas folk...
Issue type
Relates to
Summary
I have followed the Manual installation as carefully as possieble, installed all dependencies ( PETSC 3.16.0, CMake 3.22.1, OpenMPI 4.1.2 and blas) and the build call runs, but the make -j call fails.
I installed petsc in a non-standard directory and and it passed the make test all call. The Environment variables $PETSC_DIR and $PETSC_ARCH are set specific to that installation. The pther dependencies are installed in standard locations.
I also tried installation using spacks but it failed also. I have no experience with spacks just gave it a quick try. I tried different petsc versions and installations but never managed to fully run make install.
Exact commands to reproduce, if applicable The cmake call:
cmake .. -DCMAKE_INSTALL_PREFIX=$INSTALLDIR -DPETSC_DIR=$PETSC_DIR -DPETSC_ARCH="petsc-3-16" -DCMAKE_BUILD_TYPE=Debug
followed by: make install or make -j
which fails.
I am running everything on a virtual linux machine (Ubuntu).
Relevant logs and/or screenshots, if applicable
The successful cmake call specifically finding petsc:
And the error message for make -j:
make install throws a similar message regarding petsc.h
p.s. I am fairly unexperienced, if I can approve on describing the issue please let me know.