OpenFOAM (internal code compilation) broken?

[DetlevCM]

Dear Developers,

looking at EasyBuild in the context of work, it seems to me that OpenFOAM is broken - at least v22xx (forgot which one I tried) and v2312. There is a tutorial that is called "turbulentFlatPlate" which can be run as a test case. When I build my own OpenFOAM it runs just fine. However when I use the EasyBuild script, besides taking an eternity, its does not work...

OpenFOAM was installed using a pretty standard: eb OpenFOAM-v2312-foss-2023a.eb -r

The module environment used is lmod. To run OpenFOAM it is then module load OpenFOAM/v2312-foss-2023a followed by source $FOAM_BASH. Navigate to the directory and use the supplied ./Allclean and ./Allrun scripts.

When I try to use ./Allrun I get the following error:

# Create the setup: kOmegaSST yPlus-10

# Run the setup: kOmegaSST yPlus-10

No path to C++ compiler (OMPI_CXX=g++) ... cannot compile
Running decomposePar on /software/tmp/OpenFOAM-test/Test-OpenFOAM_inputs-v2306-eb/turbulentFlatPlate/results/kOmegaSST/10
Running simpleFoam (8 processes) on /software/tmp/OpenFOAM-test/Test-OpenFOAM_inputs-v2306-eb/turbulentFlatPlate/results/kOmegaSST/10 
Running reconstructPar on /software/tmp/OpenFOAM-test/Test-OpenFOAM_inputs-v2306-eb/turbulentFlatPlate/results/kOmegaSST/10
Not such file: /Cx
Not such file: /wallShearStress

But if I run for example g++ -v I get the following:

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/software/eb/GCCcore/12.3.0/libexec/gcc/x86_64-pc-linux-gnu/12.3.0/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
Target: x86_64-pc-linux-gnu
Configured with: ../configure --enable-languages=c,c++,fortran --without-cuda-driver --enable-offload-targets=nvptx-none --enable-lto --enable-checking=release --disable-multilib --enable-shared=yes --enable-static=yes --enable-threads=posix --enable-plugins --enable-gold --enable-ld=default --prefix=/software/eb/GCCcore/12.3.0 --with-local-prefix=/software/eb/GCCcore/12.3.0 --enable-bootstrap --with-isl=/software/tmp/eb-build/GCCcore/12.3.0/system-system/gcc-12.3.0/stage2_stuff --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.3.0 (GCC) 

(And I can repeat that for gcc, mpicc, etc. - I don't see an obvious issue.)

It also isn't all OpenFOAM cases: "periodicCubeWater" works just fine - it does not compile anything.

But as it stands, I have a test case from the tutorials ("turbulentFlatPlate") that works just fine when I compile OpenFOAM myself and that does not work in OpenFOAM coming from EasyBuild.

[boegel]

@DetlevCM What you're seeing may be because environment variables like $WM_CC are not defined, see also https://github.com/easybuilders/easybuild-easyconfigs/issues/5183 .

Can you check if that's indeed the case for you?

[DetlevCM]

@DetlevCM What you're seeing may be because environment variables like $WM_CC are not defined, see also easybuilders/easybuild-easyconfigs#5183 .

Can you check if that's indeed the case for you?

I've tried exporting CC=gcc, WM_CC=gcc and even OMPI_CC=gcc and then sourcing the OpenFOAM bash file, still the same error. (OMPI_CC seems to be cleared when sourcing the bash file, or I missed something).

So my first evaluation is that this is not it...

Though I can confirm those variables are not defined if I do not export them.

[ocaisa]

I can't seem to reproduce this problem (with the easyconfig from EasyBuild 4.9.1), it runs without issue for me:

[ocaisa@login1 ~]$ module load OpenFOAM/v2312-foss-2023a
[ocaisa@login1 ~]$ git clone https://develop.openfoam.com/Development/openfoam.git
Cloning into 'openfoam'...
remote: Enumerating objects: 777347, done.
remote: Counting objects: 100% (1897/1897), done.
remote: Compressing objects: 100% (748/748), done.
remote: Total 777347 (delta 1220), reused 1651 (delta 1142), pack-reused 775450
Receiving objects: 100% (777347/777347), 248.33 MiB | 10.96 MiB/s, done.
Resolving deltas: 100% (571340/571340), done.
Updating files: 100% (25334/25334), done.
[ocaisa@login1 ~]$ source $FOAM_BASH 
[ocaisa@login1 ~]$ cd openfoam/tutorials/incompressible/simpleFoam/turbulentFlatPlate/
[ocaisa@login1 turbulentFlatPlate]$ ./Allrun 

# Create the setup: kOmegaSST yPlus-10

# Run the setup: kOmegaSST yPlus-10

Restore 0/ from 0.orig/
Running blockMesh on /home/ocaisa/openfoam/tutorials/incompressible/simpleFoam/turbulentFlatPlate/results/kOmegaSST/10
Running renumberMesh on /home/ocaisa/openfoam/tutorials/incompressible/simpleFoam/turbulentFlatPlate/results/kOmegaSST/10
Running checkMesh on /home/ocaisa/openfoam/tutorials/incompressible/simpleFoam/turbulentFlatPlate/results/kOmegaSST/10
Running decomposePar on /home/ocaisa/openfoam/tutorials/incompressible/simpleFoam/turbulentFlatPlate/results/kOmegaSST/10
Running simpleFoam (8 processes) on /home/ocaisa/openfoam/tutorials/incompressible/simpleFoam/turbulentFlatPlate/results/kOmegaSST/10

[DetlevCM]

@ocaisa Hmm, that is curious. What could be the difference then?

I'm also on EasyBuuld 4.9.1 The underlying operating system is OpenSUSE 15.5.

[ocaisa]

First step would be to figure out where canCompile is going wrong:

[ocaisa@login1 10]$ module load OpenFOAM/v2312-foss-2023a
[ocaisa@login1 10]$ source $FOAM_BASH 
[ocaisa@login1 10]$ . ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions
[ocaisa@login1 10]$ canCompile 
[ocaisa@login1 10]$ type canCompile 
canCompile is a function
canCompile () 
{ 
    if ! command -v make > /dev/null; then
        echo "No system 'make' command found ... cannot compile" 1>&2;
        return 1;
    fi;
    if ! command -v wmake > /dev/null; then
        echo "No openfoam 'wmake' command found ... cannot compile" 1>&2;
        return 1;
    fi;
    local cxx_compiler;
    cxx_compiler="$(wmake -show-cxx 2>/dev/null)";
    if [ -z "$cxx_compiler" ]; then
        echo "No wmake rule for C++ compiler ... cannot compile" 1>&2;
        return 1;
    else
        if ! command -v "$cxx_compiler" > /dev/null; then
            echo "No path to C++ compiler ($cxx_compiler) ... cannot compile" 1>&2;
            return 1;
        fi;
    fi;
    return 0
}

This lead me to look at

[ocaisa@login1 10]$ wmake -show-cxx
g++

In your case though you seem to seem to end up with an environment definition OMPI_CXX=g++ rather than g++

[DetlevCM]

Indeed, I get OMPI_CXX=g++ instead of g++

/software/tmp/OpenFOAM-test/Test-OpenFOAM_inputs-v2306-eb/turbulentFlatPlate> wmake -show-cxx
OMPI_CXX=g++

[ocaisa]

I'm have zero experience with OpenFOAM, I'm not sure why this might happen, could be something to do with the underlying OS?

It is perhaps a good candidate for a sanity check command (source $FOAM_BASH; command -v $(wmake -show-cxx)) with OpenFOAM. Unfortunately there is no test suite for OpenFOAM, otherwise we would run it before confirming the installation.

[ocaisa]

I traced it back as far as

make --no-print-directory -f "$WM_DIR"/makefiles/info cxx

but I don't see anything fishy there. Could it be that you have have a bad definition in your environment? What does env | grep OMPI return?

[ocaisa]

Ok, we looked into it a little more and mine works by accident because the installation comes from before a fix was made in https://github.com/easybuilders/easybuild-easyblocks/pull/3073#issuecomment-1896113419. However, this fix means that OpenFOAM picks up on the environment variable rather than the actual compiler due to how it queries this Makefile.

So, you're original observation is correct, and we need to think about how to patch this.

[DetlevCM]

Thank you for the update - and the detective work.

For the env | grep OMPI I get the following:

EBVERSIONGOMPI=2023a
EBROOTGOMPI=/software/eb/gompi/2023a
EBDEVELGOMPI=/software/eb/gompi/2023a/easybuild/gompi-2023a-easybuild-devel
WM_COMPILER_LIB_ARCH=64
WM_COMPILER=Gcc
WM_COMPILE_OPTION=Opt
WM_COMPILER_TYPE=system

As to OpenFOAM tests: well.... - Technically there is a foamSystemcheck but it is very superficial (if you have a compiler/openMPI at ll). There are some test cases derived from the tutorial and a quick google will pick up on some validation cases, but they will test for very specific features - whether it covers all possible failure modes is unknown to me. The ESI OpenFOAM has this in the repo: https://develop.openfoam.com/Development/openfoam/-/blob/master/tutorials/Alltest

It would really need someone with substantial expertise in OpenFOAM to pick up on those bugs... - It is very much by accident that I picked up on this one, as I was looking for a reasonably sized test case for comparisons (and a tutorial means I have a known good case).

[ocaisa]

So we had this issue fixed via a patch in the past, but then we missed a change in OpenFOAM which made it appear to us like it was no longer needed. It was then dropped before we realised the change we missed (which was fixed in https://github.com/easybuilders/easybuild-easyblocks/pull/3073). I've working on a test that should help to ensure that doesn't happen again.

[DetlevCM]

Thank you for the update.

[ocaisa]

Ok, this is fixed via https://github.com/easybuilders/easybuild-easyblocks/pull/3328 and https://github.com/easybuilders/easybuild-easyconfigs/pull/20517 .

You can rebuild the package via EasyBuild with:

eb --include-easyblocks-from-pr=3328 --from-pr=20517 OpenFOAM-v2312-foss-2023a.eb --rebuild

[DetlevCM]

Thanks. It looks like the rebuild will take quite some while... - Tomorrow I'm at a meeting, I'll report back when I had the opportunity to test it.

[DetlevCM]

Update: Rebuild finished, this case (turbulent flat plate) now runs. -> So at this specific issue is now solved.