Dunecontrol and OPM modules

[MathiasLorenz]

I'm trying to create a dune module which uses OPM functionality from opm-parser. Relevant dune models, opm-common and opm-parser are installed globally (/usr/local/lib) from source. The duneproject script is able to detect the opm-common and opm-parser modules. I'm trying to build my code with the dunecontrol script, but I can see that -Dopm-common_DIR=/usr/local while the correct path would be /usr/local/lib/dunecontrol/opm-common and similarly for opm-parser. I have attached the error messages from the dunecontrol script in errReadDeckFiles.txt.

I have tried to create the project with an opts file, where I specified CMAKE_FLAGS="-Dopm-common_DIR=/usr/local/lib/dunecontrol/opm-common" manually, however, this did not solve the problem.

[andlaus]

this looks like a bug of the latest dune version's dunecontrol because it thinks your module is called ReadDeckFiles. (I assume you named it differently.)

even if this was fixed, you mileage with dune >= 2.5 may vary considerably due to several issues of the OPM build system, so we recommend that you use Dune 2.4 for now.

[alfbr]

Both Dune and OPM supports building directly with cmake. If you are building Dune out-of-tree, you will need to point OPM modules to its location with the CMAKE_PREFIX_PATH variable. There should be no need for dunecontrol. Basically, OPM modules are downstream of Dune modules. If you want to use opm-parser as an upstream module of Dune, I lack the overview to answer.

[andlaus]

Both Dune and OPM supports building directly with cmake.

yes, but the names of the parameters which the OPM build system expects are sometimes different from DUNE's and sometimes they are not. Also, some config.h macros -- e.g. those for SuperLU -- are renamed in newer versions of dune and the build will thus break if these dependencies are to be used. since I suppose that this situation does not make anyone involved with OPM happy, I guess it is time to seriously consider moving the whole thing to dune's build system to avoid the whack-a-mole game for Dune build system changes if a new version ought to be supported in the future.

There should be no need for dunecontrol.

technically that's correct. there was never any strict need for using dunecontrol. It just makes building the modules in the correct dependency order much easier. (I vaguely remember someone complaining about the fact that this is too freaking complicated.)

[alfbr]

It just makes building the modules in the correct dependency order much easier.

I beg to differ.

[andlaus]

I beg to differ.

I use the following command to rebuild opm-modules up to and including opm-simulators from scratch:

dunecontrol --opts=optim-gcc.opts --module=opm-simulators all

please tell me how to do this more briefly and I will switch to this.

[andlaus]

and, btw. the above automatically takes care of whether the required DUNE modules are installed via system packages or checked out from source.

[alfbr]

The problem is "it works for me" does not scale. Last time I tried dunecontrol was some weeks ago, and it cost me a full day with first rate support. In any case, we should not derail this issue with a dunecontrol discussion. I am simply pointing out that the reporter can do the full build chain directly with cmake without considering what might be causing issues within the 1322 lines of shell code in dunecontrol

[andlaus]

Last time I tried dunecontrol was some weeks ago, and it cost me a full day with first rate support.

yeah, and the reason for this are the many quirks of OPM's current build system w.r.t. being compatible with dune's build system. this is not an intrinsic issue of dunecontrol.

Also: may I inquire how long it took you to do this stuff without dunecontrol?

we should not derail this issue with a dunecontrol discussion.

agreed. we should use it to discuss whether the current build system is creating more headaches than the alternative.

I am just simply pointing out that the reporter can do the full build chain directly with cmake without considering what might be causing issues within the 1322 lines of shell code in dunecontrol

if you want to argue like this, you need to include the OPM build system into the discussion (because this you need to deal with):

find opm-common/cmake/ opm-parser/cmake/ -name "*.cmake" | xargs wc -l
[...]
  9356 total

this number does not even include the files that end on .txt or everything which is not located in opm-common and opm-parser.

[blattms]

No I am not taking part in a buildsystem discussion but try to give a constructive answer to @MathiasLorenz question:

I think @berndflemisch was the only one being so brave as to try to use opm from a dune module. In general I would expect quite a few issues because of buildsystem incompatibilities. Maybe opm-parser is even the most easy target to fix.

You are using installed modules and dunecontrol tries to guess the place where opm-parser-config.cmake is located. Unfortunately, it uses DUNE conventions and looks under ${CMAKE_INSTALL_LIBDIR}/cmake/<module-name>/ while opm-parser uses ${CMAKE_INSTALL_DATAROOTDIR}/cmake/<module-name>/. It is a bit hard for upstream to guess locations used downstream.

There are two possibilities to proceed:

1. Do not use installed modules but use the build trees.
2. This might work not work depending if definition of variables passed to cmake overwrite previous ones. Try setting -Dopm-parser_DIR=/directory/with/opm-parser-config.cmake in CMAKE_PARAMS. If it does not work I am willing to patch dunecontrol such that it allows for this work.

Beware that the next problem might appear around the next corner. Thanks for testing

[blattms]

Edited my answer to make github formatting happy with the way I describe paths.

[alfbr]

Also: may I inquire how long it took you to do this stuff without dunecontrol?

Just tested for the first time in ages, and was able to compile and install different versions. Something I have never accomplished with dunecontrol. I am on Red Hat, and do not have root access. This means out-of-tree dependencies and out-of-tree install location. Basically it goes like this:

git clone https://gitlab.dune-project.org/core/dune-module.git
mkdir dune-module/build && cd dune-module/build
cmake DCMAKE_CXX_COMPILER=/opt/rh/devtoolset-3/root/usr/bin/g++ \ #too old compiler on Red Hat 6
-DCMAKE_INSTALL_PREFIX=/path/to/install/dune \ #I do not have root access
-DCMAKE_PREFIX_PATH=/path/to/SuperLU4.3 .. #Red Hat does not ship SuperLU
make -j16
make install

dune-common first, then whatever else you need from Dune. Non-standard cmake options are exposed when you run ccmake instead of cmake directly. Once you have compiled all dune modules you need, you can use the exact same approach for opm-modules, pointing to dune install folder with the CMAKE_PREFIX_PATH variable.

It seems dune from git now requires suitesparse newer than even the latest from Red Hat, so I am not able to compile the latest and greatest (unless I install that one out-of-tree too).

[MathiasLorenz]

First of all, thanks for all the comments. I think a quick intro into what I'm trying to accomplish is warranted. I have a codebase written with Dune to simulate oil reservoirs and for these I wish to be able to read in Deck files using the functionality from opm-parser. Thus all the code is organised with dunecontrol and therefore it would be much preferred if I could keep the code this way. The module mentioned in the error file (ReadDeckFiles) was meant to be a simple test program for linking opm with Dune and using the functionality in opm-parser before implementing it into the existing codebase.

I have tried the second option in @blattms answer, however, I haven't gotten much further. I made a .opts file as shown in release.opts.txt (I know the file should not be .txt but GitHub won't allow me to upload .opts files), where I simply put the located directory of opm-common-config.cmake into CMAKE_PARAMS (and I decided to make opm-common work before opm-parser) to the best of my ability. Have I understood what you meant correctly or am I missing something?

[joakim-hove]

Hello;

I am not at all familiar with DuneControl - but I am quite familar with opm-parser; and I would be very happy if people outside the innermost OPM community could use it.

[...] Thus all the code is organised with dunecontrol [....]

This is the thing I just don't understand - it seems use of dunecontrol is an all-or-nothing question? Assuming you do not intend to develop actively on opm-parser I would assume you could start by installing opm-parser, preferably even in a standard location like /usr/local and then continue developing your own module using dunecontrol - for a situation like that the build system originally used to build and install opm-parser should be completely irrelevant? I assume an application built with dunecontrol can still link to e.g. zlib or arpack without those needing to be dune modules? Or what is it I do not understand?

where I simply put the located directory of opm-common-config.cmake into CMAKE_PARAMS (and I decided to make opm-common work before opm-parser)

Since a quite massive build system overhaul in May/June I think you should be able to build opm-parser without opm-common.

[andlaus]

for a situation like that the build system originally used to build and install opm-parser should be completely irrelevant?

I agree: As far as I see it. Also, I am now pretty sure that @MathiasLorenz' issue is not primarily caused by dunecontrol, but by opm-parser's cmake build system.

[andlaus]

@MathiasLorenz: let's move this to private email because this issue seems to be specific to your setup. I don't have your email address, though.

[MathiasLorenz]

@andlaus You are welcome to contact me at s134597 (at) student.dtu.dk.

[MathiasLorenz]

After discussing the issue with @andlaus and tinkering the code I got it to work by linking opm-parser as an external library (as suggested by a lot of the above contributors). I will below present how to get it to work for anyone else wanting to do the same.

When creating the project you do not need to specify opm-parser as a dune module for the project. If you have already created your project you can remove opm-parser as a dune module be deleting it in the dune.module file located in /your-dune-project/dune.module under Depends:. If you have installed opm-parser (and as mentioned by @joakim-hove you do not need opm-common anymore) globally the header files are included with

#include <opm/parser/eclipse/Parser/Parser.hpp>

You need to link both the opm-parser and boost libraries for the build to work. This is done by altering the file /MyDuneProject/src/CMakeLists.txt. This file has to lines by default. Open the file and insert these lines below, where MyDuneProject is of course altered to your project

# Find the OPM parser library
find_library(opmparser_LIBRARY opmparser)
message(STATUS "setting opmparser_LIBRARY="${opmparser_LIBRARY}"")

# Link libraries
# Both opm-parser and boost_system must be linked.
target_link_libraries("MyDuneProject" "${opmparser_LIBRARY}")
target_link_libraries("MyDuneProject" "boost_system")

Which should be sufficient for the build to work. The project can then be build with dunecontrol all.

I choose to close the issue as I managed to solve the problem. @andlaus is working on a more permanent fix with the build system, which might come in handy for future users.

[blattms]

Let me add to this after my short holiday. The error was:

CMake Error at /usr/local/share/dune/cmake/modules/DuneMacros.cmake:255 (message):
  Could not find dune.module file for module opm-common in ,
  /lib/dunecontrol/opm-common/, /lib64/dunecontrol/opm-common/dune.module

when configuring your module. This means that the variable opm-common_PREFIX is not set or is empty.

This might also be fixed by simply setting opm-common_PREFIX in the package configuration file opm-common-config.cmake.

[blattms]

Sorry I need to correct myself. The reason is that dunecontrol does not guess that OPM installs the package configuration files below share/cmake and not lib/cmake.

[blattms]

I would really love to replicate and investigate this. Is your module available?