Installation problem in OpenFOAM-v2006

[ys-r]

Hello, I have a problem with the installation of WENOEXT with OpenFOAM-v2006.

Operating system/version: Cray XC50 SUSE Linux Enterprise Server 12 SP3 (x86_64)

Attached file is detailed installation log and Allwmake, because I rewrote the Allwmake in order to detect cmake binary and try to fix the error.

inst_log.txt Allwmake.txt

The following is an example of an error message.

 Linking CXX executable writeWENOStats
/usr/bin/ld: cannot find -ldynamicMesh
/usr/bin/ld: cannot find -lfiniteVolume
/usr/bin/ld: cannot find -lmeshTools
/usr/bin/ld: cannot find -lspecie
/usr/bin/ld: cannot find -lsurfMesh
/usr/bin/ld: cannot find -lfileFormats
/usr/bin/ld: cannot find -lOpenFOAM
collect2: error: ld returned 1 exit status
utilities/writeWENOStats/CMakeFiles/writeWENOStats.dir/build.make:97: recipe for target 'utilities/writeWENOStats/writeWENOStats' failed

Thanks.

[JanGaertner]

Hello,

If I understand correctly you add the cmake-3.21 path so that you use cmake 3.21?

export PATH="/home/ys-r/OpenFOAM_ESI/OpenFOAM-v2006/ThirdParty/build/linux64Icc/cmake-3.21.2/bin/:$PATH"
export LD_LIBRARY_PATH="$FOAM_LIBBIN:$LD_LIBRARY_PATH"

I am not sure if it is due to the newer version of cmake or the system your compiling on, but it appears that dynamic linking is not supported. This seems quite weird to me as OpenFOAM also relies on dynamic linking.

CMake Warning (dev) at libWENOEXT/CMakeLists.txt:4 (add_library):
  ADD_LIBRARY called with SHARED option but the target platform does not
  support dynamic linking.  Building a STATIC library instead.  This may lead
  to problems.

I would advise contacting your support team of the HPC Cray system and ask why dynamic linking is not working?

[ys-r]

Thanks for your comments.

Because the version of cmake bundled with the CRAY system is cmake-3.5, I installed the latest cmake in OpenFOAM ThirdParty directory.

Compiling the OpenFOAM required some configuration work, but it went well in CRAY system. I have a question about how to use cmake. How do I specify the compiler command in cmake?

Thanks.

[JanGaertner]

You can specify the compiler with:

cd build/
cmake ../ -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CC_COMPILER=gcc
make -j 
make install 

you could also use the cmake --build . command instead of make but this older version of cmake does not support the parallel flag, so make -j would be faster.

[JanGaertner]

One addition: You have to delete the old build directory before you can change the compiler. So first rm -r build and then the listed commands

[JanGaertner]

one more addition. As you are probably running a large scale simulation you can also use the development branch. It has the newest features and speed improvements.

I am also currently working on improving the MPI calls in WENoCoeff. If you have experience in MPI programming I invite you to have a look at it.

[Mayank2011]

Hello, I have a problem with the installation of WENOEXT with OpenFOAM-v2006.

Operating system/version: Cray XC50 SUSE Linux Enterprise Server 12 SP3 (x86_64)

Attached file is detailed installation log and Allwmake, because I rewrote the Allwmake in order to detect cmake binary and try to fix the error.

inst_log.txt Allwmake.txt

The following is an example of an error message.

 Linking CXX executable writeWENOStats
/usr/bin/ld: cannot find -ldynamicMesh
/usr/bin/ld: cannot find -lfiniteVolume
/usr/bin/ld: cannot find -lmeshTools
/usr/bin/ld: cannot find -lspecie
/usr/bin/ld: cannot find -lsurfMesh
/usr/bin/ld: cannot find -lfileFormats
/usr/bin/ld: cannot find -lOpenFOAM
collect2: error: ld returned 1 exit status
utilities/writeWENOStats/CMakeFiles/writeWENOStats.dir/build.make:97: recipe for target 'utilities/writeWENOStats/writeWENOStats' failed

Thanks.

Hi, I too faced the same error when compiling it with the MPICH-3.3. I solved this error as mentioned below.

1. Try the following changes in the Allwmake.

---

!/bin/bash

mkdir -p build cd build cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx ../ cmake --build . cmake --build . --target install

---

2. And add the -ldl flag in the CMAKE_SHARED_LINKER_FLAGS present in the CMakeLists.txt file.

---

set(CMAKE_SHARED_LINKER_FLAGS "-fPIC -Xlinker --add-needed -Xlinker --no-as-needed -ldl")

---

3. Run ./Allwmake

I hope this solves the problem. Try it and let me know if you still face the same error..

PS: Your OpenFOAM should also be installed using MPICH-3.3 not with intelMPI or openMPI.

[JanGaertner]

I hope this resolved your problems. I will close the issue now

[ys-r]

Thanks for a lot of comments, and sorry for the late reply.

Due to my OpenFOAM is installed using Intel icc and icpc with CRAY-MPICH, so I can't installed WENO-OF with mpicc or gcc. CMakeLists-gnu.txt log.make.gnu.txt Allwmake-gnu.txt

Also I tried icc and icpc with CRAY-MPICH, but icpc doesn't work well for WENO-OF. Allwmake-icc.txt log.make.icc.txt CMakeLists-icc.txt

I'll have to try compiling OpenFOAM with gcc and CRAY-MPICH, and try installing WENO with gcc and CRAY-MPICH. Of course, being able to compile WENO-OF with the Intel compiler is the best thing for me.

Thanks.

[JanGaertner]

From your log it seems to fail to generate the utilities and test cases. You do not need these for the WENO scheme and in the development branch you can also deactivate the tests by specifying the RELEASE build.

Something you could try to do is to comment out the

add_subdirectory(utilities)
add_subdirectory(tests)

lines at the end of the CMake file

[ys-r]

Thanks to your instructions, the installation seems to have worked with gcc. But in my OpenFOAM application, WENOEXT.so doesn't work well.

--> FOAM Warning :
    From void *Foam::dlLibraryTable::openLibrary(const Foam::fileName &, bool)
    in file db/dynamicLibrary/dlLibraryTable/dlLibraryTable.C at line 65
    Could not load "libWENOEXT.so"
/home/ys-r/OpenFOAM_ESI/OpenFOAM-v2006/ys-r-v2006/platforms/linux64IccDPInt32Opt/lib/libWENOEXT.so: undefined symbol: _ZN4Foam9UOPstream11writeQuotedERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb

[JanGaertner]

Did you add the

libs("libWENOEXT.so")

to your system/controlDict?

[Mayank2011]

Thanks to your instructions, the installation seems to have worked with GCC. But in my OpenFOAM application, WENOEXT.so doesn't work well.

--> FOAM Warning :
    From void *Foam::dlLibraryTable::openLibrary(const Foam::fileName &, bool)
    in file db/dynamicLibrary/dlLibraryTable/dlLibraryTable.C at line 65
    Could not load "libWENOEXT.so"
/home/ys-r/OpenFOAM_ESI/OpenFOAM-v2006/ys-r-v2006/platforms/linux64IccDPInt32Opt/lib/libWENOEXT.so: undefined symbol: _ZN4Foam9UOPstream11writeQuotedERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb

This seems to me the basic linking problem between the cxx compiler and the libWENOEXT.so. Can you specify the OF-configuration like which GCC, and mpirun you are using? Also, recheck once if WENO is installed using the correct (or the same) GCC and mpirun versions by which the OpenFOAM is running.

[JanGaertner]

Mayank2011 is right. It must be linking issues as it does not find the correct Pstream library which is depending on the mpi version you use.

[ys-r]

I rebuilt OpenFOAM with GCC because I couldn't build WENOEXT with ICC. Finally, WENOEXT worked fine with OpenFOAM tutorials (rhoPimpleFoam/pitzDaily). However, in my calculation, WENOEXT did not work. My calculation is using polyhedral mesh. Does WENOEXT need the mesh condition?

Attached file is a calculation log. log.rhoSimpleFoam.txt

The following is part of the log file.

Thank you.

--> FOAM Warning :
    From Foam::triFace Foam::tetIndices::faceTriIs(const Foam::polyMesh&, bool) const
    in file meshes/polyMesh/polyMeshTetDecomposition/tetIndicesI.H at line 84
    No base point for face 4445016, 8(3870965 3871111 3871110 3871113 3871114 3871115 3870870 3870869), produces a valid tet decomposition.
--> FOAM Warning : Suppressing any further warnings.
[49]
[49]
[49] --> FOAM FATAL ERROR:
[49] Singular matrix
[49]
[49]     From void Foam::LUDecompose(Foam::scalarSquareMatrix&, Foam::labelList&, Foam::label&)
[49]     in file matrices/scalarMatrices/scalarMatrices.C at line 71.
[49]
FOAM parallel run exiting
[49]

[JanGaertner]

For WENO to work it needs to be able to decompose the mesh into tets which it then can efficiently solve by Gauss quadrature. This should be possible for all meshes with a good mesh quality. If you perform checkMesh -allGeometry -allTopology what is the result? Are mesh warnings printed?

[ys-r]

I have checked and there seems to be a problem with mesh quality. Are there any of the following items that are particularly important for running the WENO scheme?

Checking topology...
    Boundary definition OK.
    Cell to face addressing OK.
    Point usage OK.
  <<Found 1 neighbouring cells with multiple inbetween faces.
    Upper triangular ordering OK.
  <<Writing 2 unordered faces to set upperTriangularFace
    Face vertices OK.
    Topological cell zip-up check OK.
    Face-face connectivity OK.
    Number of regions: 1 (OK).

Checking geometry...
    Overall domain bounding box (-5 -3.11859565305e-07 -9.27400906004e-16) (5 1.25 1.5)
    Mesh has 3 geometric (non-empty/wedge) directions (1 1 1)
    Mesh has 3 solution (non-empty) directions (1 1 1)
    Boundary openness (-3.13510459816e-16 -2.18445292697e-13 1.09322154681e-13) OK.
 ***High aspect ratio cells found, Max aspect ratio: 13305.9168247, number of cells 655354
  <<Writing 655354 cells with high aspect ratio to set highAspectRatioCells
    Minimum face area = 2.1758922119e-11. Maximum face area = 0.0111389374227.  Face area magnitudes OK.
    Min volume = 2.6061541155e-14. Max volume = 0.00113646009553.  Total volume = 18.2259211969.  Cell volumes OK.
    Mesh non-orthogonality Max: 89.6857416707 average: 5.03331895209
   *Number of severely non-orthogonal (> 70 degrees) faces: 186378.
    Non-orthogonality check OK.
  <<Writing 186378 non-orthogonal faces to set nonOrthoFaces
    Face pyramids OK.
 ***Max skewness = 4.4123158636, 1237 highly skew faces detected which may impair the quality of the results
  <<Writing 1237 skew faces to set skewFaces
    Coupled point location match (average 0) OK.
 ***Error in face tets: 421496 faces with low quality or negative volume decomposition tets.
  <<Writing 421027 faces with low quality or negative volume decomposition tets to set lowQualityTetFaces
   *Edges too small, min/max edge length = 3.75120522824e-07 0.1103593512, number too small: 40836155
  <<Writing 40836155 points on short edges to set shortEdges
   *There are 21853 faces with concave angles between consecutive edges. Max concave angle = 89.9065665011 degrees.
  <<Writing 21853 faces with concave angles to set concaveFaces
    Face flatness (1 = flat, 0 = butterfly) : min = 0.0196816586109  average = 0.999583986991
   *There are 51 faces with ratio between projected and actual area < 0.8
    Minimum ratio (minimum flatness, maximum warpage) = 0.0196816586109
  <<Writing 51 warped faces to set warpedFaces
    Cell determinant (wellposedness) : minimum: 0 average: 9.58245461131
 ***Cells with small determinant (< 0.001) found, number of cells: 39401614
  <<Writing 39401614 under-determined cells to set underdeterminedCells
 ***Concave cells (using face planes) found, number of cells: 1439265
  <<Writing 1439265 concave cells to set concaveCells
    Face interpolation weight : minimum: 0.00047573492558 average: 0.488763485024
 ***Faces with small interpolation weight (< 0.05) found, number of faces: 1305382
  <<Writing 1305382 faces with low interpolation weights to set lowWeightFaces
    Face volume ratio : minimum: 0.000145470382721 average: 0.940210589246
 ***Faces with small volume ratio (< 0.01) found, number of faces: 525522
  <<Writing 525522 faces with low volume ratio cells to set lowVolRatioFaces

Failed 7 mesh checks.

[JanGaertner]

Following error is problematic but not necessarily a fatal one for WENO:

Cell determinant (wellposedness) : minimum: 0 average: 9.58245461131

***Cells with small determinant (< 0.001) found, number of cells: 39401614

But this error is critical and should also concern you for any other simulation: ***Error in face tets: 421496 faces with low quality or negative volume decomposition tets.

Even if you do not want to use WENO I would advise fixing these errors until you get a "Mesh ok" from checkMesh

[ys-r]

It is going to take some time to solve this problem. I will focus on fixing the grid. I would like to close this issue since WENO worked on a simple grid.

Thank you very much for your kindly comments.

[woshidaxia123]

Hello, I also meet some problems when compiling the source code: when I run ./Allwmake, the wrong info is:

[  6%] Building CXX object libWENOEXT/CMakeFiles/WENOEXT.dir/WENOBase/WENOBase.C.o
/home/dyfluid/WENOEXT/libWENOEXT/WENOBase/WENOBase.C: In member function ‘void Foam::WENOBase::distributeStencils(Foam::labelListList&)’:
/home/dyfluid/WENOEXT/libWENOEXT/WENOBase/WENOBase.C:373:39: error: ‘nonBlocking’ is not a member of ‘Foam::Pstream’

  373 |         PstreamBuffers pBufs(Pstream::nonBlocking);

my OpenFOAM version is v2006 source code version is 3.1 gcc version is 9.3.0

since I don't have much experience in cmake, I'm curious that whether it's a compiling problem or just some small mistakes in the source code.

Thank you!

[JanGaertner]

I believe it comes from not setting the OpenFOAM version rules correctly. It should set the define makro "FOAM_PSTREAM_COMMSTYPE_IS_ENUMCLASS" to true because newer OpenFOAM versions put the commsType into a enum class.

Did you use the master branch version? You can also try the development or the workingBlazeVersion branch. However, they are development branches in which I still try to fix and implement improvements. So I do not guarantee it works, however, for me it does work for OpenFOAM v2012 on multiple systems with different hardware architectures, including high performance systems.

One work around is to add to the add_definitions line in the top level CMakeLists.txt the option: -DFOAM_PSTREAM_COMMSTYPE_IS_ENUMCLASS

and then to delete the build directory and recompile with Allwmake

[woshidaxia123]

Sorry for long time no replying. I finally find my problem. It seems that the version change relies on a python file, which uses command python to trigger. However, the python I installed is python version 3.8, and the default command is python3.To deal with that problem, I sudo cp python3 python in my usr/bin, and it finally works. Thank you very much!