FindADIOS.cmake

[andrewcorrigan]

I would like to request a fully-functional FindADIOS.cmake, which detects not only libadios and adios.h, but also all the dependent libraries.

So far, I have manually performed ADIOS detection in my code's CMakeLists.txt. This works for a minimal installation of ADIOS, but fails for an installation with all the various optional dependencies included (lustre, various Cray libraries, HDF5, NETCDF, etc.) that ADIOS may or may not require. When that fails, I typically just copy and paste flags from adios_config.flags into CMAKE_EXE_LINKER_FLAGS, but that is not an idiomatic CMake way of doing things, and introduces considerable burden on users of my code who are used to the standard way of using CMake.

    find_path(ADIOS_INCLUDE_PATH adios.h)
    if(NOT ADIOS_INCLUDE_PATH)
        message(SEND_ERROR "Please set ADIOS_INCLUDE_PATH")
    endif()
    find_package(Threads)   # pthreads seems to be required by mxml
    list(APPEND libraries ${CMAKE_THREAD_LIBS_INIT})
    find_library(ADIOS_LIBRARY NAMES adios)
    if(NOT ADIOS_LIBRARY)
        message(SEND_ERROR "Please set ADIOS_LIBRARY")
    endif()
    find_library(ADIOS_MXML_LIBRARY NAMES mxml)
    if(NOT ADIOS_MXML_LIBRARY)
        message(SEND_ERROR "Please set ADIOS_MXML_LIBRARY")
    endif()
    list(APPEND libraries ${ADIOS_LIBRARY})
    list(APPEND libraries ${ADIOS_MXML_LIBRARY})
    include_directories(${ADIOS_INCLUDE_PATH})

[QuLogic]

I use autotools and not CMake, but I use adios_config. Instead of guessing dependencies, have you tried using that?

adios_config -c returns compiler flags. adios_config -l returns linker flags.

[andrewcorrigan]

That doesn't help. I'm not guessing dependencies, nor am I asking for that. adios_config is a command-line tool, which unless it integrates with CMake, is essentially the same thing as what I am already doing.

[andrewcorrigan]

The ideal thing would be akin to how VTK detection works with CMake: FindVTK.cmake looks for VTK_DIR, and when it is found, provides a variable USE_VTK_FILE, which can be included, and then provides conventional CMake variables for specifying libraries, include paths, etc.

find_package(ADIOS REQUIRED) include (${USE_ADIOS_FILE}) ... target_link_libraries(myApplication ${ADIOS_LIBRARIES})

[QuLogic]

That doesn't help. I'm not guessing dependencies, nor am I asking for that. adios_config is a command-line tool, which unless it integrates with CMake, is essentially the same thing as what I am already doing.

You misunderstand, I am not telling you that's the solution, just that it's a more robust method of doing what you're already doing. Your example has mxml, but that's not always the only thing that could be necessary to link with ADIOS.

My CMake is a little rusty, but I'm thinking something like this. Maybe you can fix it up and the ADIOS developers can include it.

find_program(ADIOS_CONFIG_EXECUTABLE NAMES adios_config)
execute_process(COMMAND ${ADIOS_CONFIG_EXECUTABLE} -i RESULT_VARIABLE ADIOS_CFLAGS)
execute_process(COMMAND ${ADIOS_CONFIG_EXECUTABLE} -l RESULT_VARIABLE ADIOS_LIBS)
execute_process(COMMAND ${ADIOS_CONFIG_EXECUTABLE} -i -f RESULT_VARIABLE ADIOS_FCFLAGS)
execute_process(COMMAND ${ADIOS_CONFIG_EXECUTABLE} -l -f RESULT_VARIABLE ADIOS_FLIBS)

[ax3l]

@andrewcorrigan We run into the same problem that we required a certain component X but adios was compiled with additional support for component Y which pulls in further dependencies.

So I just started writing a module today which is still far from being perfect.

Components support is still missing, but if you are interested in a first draft check out https://github.com/ComputationalRadiationPhysics/picongpu/pull/271

Does basically the same: requires mxml and pulls in the dataspace/staging dependencies if they are compiled in. Still missing components from the ADIOS Manual:

• lustre api
• serial hdf5 / parallel hdf5
• serial net-cdf / net-cdf4
• fortran90 (nej ;) )
• python (not rly a dependency to compile)
• mpi (used by our project anyway so far but should be pulled to)

[QuLogic]

Does basically the same: requires mxml and pulls in the dataspace/staging dependencies if they are compiled in.

Does adios_config not provide the correct paths for compiling mxml, dataspace, etc.? From my reading of configure.ac, it appears that it is supposed to do so.

[pnorbert]

Hi,

adios_config was created for apps that use automake or manual makefiles. We are going to generate the same information in a format suitable for apps that use cmake.

Norbert

On Wed, Mar 12, 2014 at 3:34 PM, Elliott Sales de Andrade < notifications@github.com> wrote:

Does basically the same: requires mxml and pulls in the dataspace/staging dependencies if they are compiled in.

Does adios_config not provide the correct paths for compiling mxml, dataspace, etc.? From my reading of configure.ac, it appears that it is supposed to do so.

Reply to this email directly or view it on GitHubhttps://github.com/ornladios/ADIOS/issues/7#issuecomment-37453675 .

[andrewcorrigan]

@ax3l I will have to take a look at that. I've been distracted with non-ADIOS related endeavors over the past few months, but will give it a shot when I get back into it. Thanks for sharing.

@pnorbert I'm looking forward to official CMake support!

[ax3l]

As @pnorbert said, adios_config provides all one needs to compile and link.

The only thing is that this is a bit cumbersome to parse in CMake scripts. Imho a CMake module could depend on the information provided by that tool.

[QuLogic]

As @pnorbert said, adios_config provides all one needs to compile and link.

That's why I'm a bit confused why the FindADIOS.cmake in your pull request checks for mxml explicilty?

[ax3l]

Because I am not using autotools - I have to link to libmxml.a myself since adios pulls in this dependency for the final executables.

But you are right, I am writing right now on a general regex that parses the provided libs from adios_config to absolute paths to avoid exactly that.

[QuLogic]

Because I am not using autotools - I have to link to libmxml.a myself since adios pulls in this dependency for the final executables.

This autotools vs. cmake thing is a red herring. You do the same thing with autotools: call adios_config, and link with the libraries it says to use.

But you are right, I am writing right now on a general regex that parses the provided libs from adios_config to absolute paths to avoid exactly that.

Why do you need to know this? Why can't you just link with everything output from adios_config?

[pnorbert]

Elliott,

Can you do this with cmake? Can you give us an example?

You seem to know something no one else does and therefore everybody tries to come up with a complicated cmake macro to find the dependencies (not just Axel).

Thanks Norbert

On Wed, Mar 12, 2014 at 5:12 PM, Elliott Sales de Andrade < notifications@github.com> wrote:

Because I am not using autotools - I have to link to libmxml.a myself since adios pulls in this dependency for the final executables.

This autotools vs. cmake thing is a red herring. You do the same thing with autotools: call adios_config, and link with the libraries it says to use.

But you are right, I am writing right now on a general regex that parses the provided libs from adios_config to absolute paths to avoid exactly that.

Why do you need to know this? Why can't you just link with everything output from adios_config?

Reply to this email directly or view it on GitHubhttps://github.com/ornladios/ADIOS/issues/7#issuecomment-37465950 .

[QuLogic]

@pnorbert: I gave an example in an earlier comment. You then add the particular *CFLAGS and *LIBS to the compiler and link settings as usual. I did not test it out, so there may have been some small syntax errors, but I don't see why it wouldn't work out fine. I was not as confident with CMake then, so I can try to write something more substantial up now.

[ax3l]

@QuLogic it is not that easy since CMake splits libraries and library names from their pathes. There is simply no such thing as "shoot this long string to your linker".

Your execute_process example is just the beginning, one needs some parsing...

[QuLogic]

@QuLogic it is not that easy since CMake splits libraries and library names from their pathes. There is simply no such thing as "shoot this long string to your linker".

Yes, you are correct, of course. Sometimes I forget cmake's bizarreness. But I'm not sure the parsing needs to be very complicated (i.e., don't think a regex is necessary.) Let me test some ideas out.

[ax3l]

I rewrote the script to parse the output from adios_config and to create cmake-like absolute links per library.

The absolute path is derived from the -L and then the default paths, prefering static over shared libs right now.

The output, e.g. for enabled DataSpace transports (I will remove the Found something lines later on):

-- ADIOS linker flags (unparsed): -L/sw/xk6/adios/1.6.0/cle4.0_gnu4.7.2/lib -ladios -L/sw/xk6/mxml/2.6/cle4.0_gnu4.5.3/lib -lmxml -L/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib -L/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib -L/opt/cray/pmi/default/lib64 -L/opt/cray/ugni/default/lib64 -lm -lmxml -ldspaces -ldscommon -ldart -ldspaces -ldscommon -ldart -lpmi -lugni
-- Found libadios in /sw/xk6/adios/1.6.0/cle4.0_gnu4.7.2/lib/
-- Found libmxml in /sw/xk6/mxml/2.6/cle4.0_gnu4.5.3/lib/
-- Found libm in /usr/lib/
-- Found libmxml in /sw/xk6/mxml/2.6/cle4.0_gnu4.5.3/lib/
-- Found libdspaces in /sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/
-- Found libdscommon in /sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/
-- Found libdart in /sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/
-- Found libdspaces in /sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/
-- Found libdscommon in /sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/
-- Found libdart in /sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/
-- Found libpmi in /opt/cray/pmi/default/lib64/
-- Found libugni in /opt/cray/ugni/default/lib64/
-- Found ADIOS: /sw/xk6/adios/1.6.0/cle4.0_gnu4.7.2/lib/libadios.a;/sw/xk6/mxml/2.6/cle4.0_gnu4.5.3/lib/libmxml.a;/usr/lib/libm.a;/sw/xk6/mxml/2.6/cle4.0_gnu4.5.3/lib/libmxml.a;/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/libdspaces.a;/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/libdscommon.a;/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/libdart.a;/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/libdspaces.a;/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/libdscommon.a;/sw/xk6/dataspaces/1.3.0/cle4.0_gnu4.7.2/lib/libdart.a;/opt/cray/pmi/default/lib64/libpmi.a;/opt/cray/ugni/default/lib64/libugni.a (found suitable version "1.6.0", minimum required is "1.6.0")

And one can use it like this:

# ...
# you can provide a hint with the environment variable ADIOS_ROOT
# set(ADIOS_USE_STATIC_LIBS ON) # if you want to force static linking
find_package(ADIOS 1.6.0)

if(ADIOS_FOUND)
    include_directories(SYSTEM ${ADIOS_INCLUDE_DIRS})
    set(LIBS ${LIBS} ${ADIOS_LIBRARIES})
endif(ADIOS_FOUND)
# ...

To support cmake components one would have to write a lookup table library name <-> component name.

[QuLogic]

@ax3l: Good job! I barely got home; if only all our problems could be solved while we were in transit... There are a couple of minor things though; I will comment on the PR there.

[ax3l]

All right, just updated to rely totally on the paths from adios_config (even for "weird" non-default include/ and lib/lib64 install dirs).

The only thing one has to set now is: either provide a hint where to find adios_config with export ADIOS_ROOT=/your/install (the standard cmake style environment variable) or simply set ADIOS' bin/ dir in your PATH.

[ax3l]

@pnorbert are you interested in a pull request for the final script?

I would change to a version so it fits your license (3-clause BSD) and push it over. We would skip the additional notice, hope you guys are fine with that since we are not UT-BATTELLE employees.

As the second author of the script: @f-schmitt-zih do you agree to fork a BSD version of our script?

[f-schmitt]

yes, I agree

[jyamu]

Hi,

Thanks for this work. We are fine with the BSD version and your names will be on it. If you don't mind, could you give us the cmake script such that we can push it to the repo and add more dependencies to it?

Thanks, Kimmy

From: Axel Huebl notifications@github.com<mailto:notifications@github.com> Reply-To: ornladios/ADIOS reply@reply.github.com<mailto:reply@reply.github.com> Date: Friday, March 14, 2014 8:29 AM To: ornladios/ADIOS ADIOS@noreply.github.com<mailto:ADIOS@noreply.github.com> Subject: Re: [ADIOS] FindADIOS.cmake (#7)

@pnorberthttps://github.com/pnorbert are you interested in a pull request for the final script?

I would change to a version so it fits your license (3-clause BSD) and push it over. We would skip the additional notice, hope you guys are fine with that since we are not UT-BATTELLE employees.

As the second author of the script: @f-schmitt-zihhttps://github.com/f-schmitt-zih do you agree to fork a BSD version of our script?

— Reply to this email directly or view it on GitHubhttps://github.com/ornladios/ADIOS/issues/7#issuecomment-37641989.

[ax3l]

Hi @jyamu,

that sounds great!

I will just use the "usual Git(Hub) workflow" and open a pull request to your master with the modified version.

You will see the public branch that I propose in the pull request, so feel free to branch from that again. That's simple and clean, else we will loose track of different versions.

Edit: I am not sure if you need to extend the script. As long as adios_config gives out the information about additional components, the script will find it.

[jyamu]

Hi,

We are going to have a ADIOS release in five weeks. Just wondering if everyone is happy with FindADIOS.cmake module now. If there is any problem please let us know.

The module is located https://github.com/ornladios/ADIOS/blob/master/FindADIOS.cmake

Thanks you, Kimmy

[ax3l]

@jyamu I just added a slight modification today that handles the tailing white space / new lines without the need of a regex.

Might be useful for you, too - shall I open a pull request for it? Edit: See #21 Example diff in picongpu.