Blocking download of dependencies

mboisson commented 1 year ago

Is your feature request related to a problem? Please describe. When installing software on a high performance computing cluster, we typically want to reuse existing dependencies. SU2 seems to download a lot of things which we already have: autotools, catch2, cgns, metis, ninja, parmetis, ...

It will very likely not compile those correctly for our system.

Describe the solution you'd like Give a flag to disable all download and fail when dependencies are not found. Give corresponding flags to point to the dependencies' folders, and use those.

Describe alternatives you've considered Reverse engineering SU2's meson/ninja build to patch it to use existing dependencies.

bigfooted commented 1 year ago

Hi, so in your case, su2 didn't compile on your system using the auto-downloaded tools, but by using the existing system-wide installation of the tools, it did compile? Or you only suspected that it will not compile?

In any case if you have a working solution with these flags in place please feel free to submit a pull request.

pcarruscag commented 1 year ago

Please run preconfigure.py --help and see if it addresses what you need

mboisson commented 1 year ago

Please run preconfigure.py --help and see if it addresses what you need

You mean the script in the legacy folder ?

It does not provide any flag to tell the configure script to use existing external dependencies.

mboisson commented 1 year ago

Hi, so in your case, su2 didn't compile on your system using the auto-downloaded tools, but by using the existing system-wide installation of the tools, it did compile? Or you only suspected that it will not compile?

In any case if you have a working solution with these flags in place please feel free to submit a pull request.

I did manage to compile it but it did not reuse any of the software we already have. I even had to modify build scripts to not use meson and ninja from the system, but rather the one that is vendored in the SU2 source code, which by the way is contrary to the documentation which states that meson from the system can be used, while the actual script states the opposite: https://github.com/su2code/SU2/blob/master/meson.build#L9C1-L11C6

That means I have no clue how Metis or Parmetis were compiled for example. Nor do I have a clue which versions of those were used.

pcarruscag commented 1 year ago

The preconfigure in the root of the repo, use the develop branch. Does your system use pkg config for modules?

mboisson commented 1 year ago

There's no using the develop branch. Develop software don't make their way in our module systems, we only install releases, with tarbals of the sources and checksums of those tarballs, as we aim for reproducibility.

We have a large number of build tools, including pkgconfig, cmake, meson, ninja, bazel, autoconf, etc. We use EasyBuild to wrap things up and generate modules.

pcarruscag commented 1 year ago

Yes but it would be kind of you to check if a feature of a future release is going in the direction of what you need. Compiled libraries would be easy to handle via pkg config. But your choices of words sound a bit unfriendly so I honestly don't care, we can also close the issue until the next release.

mboisson commented 1 year ago

Yes but it would be kind of you to check if a feature of a future release is going in the direction of what you need. Compiled libraries would be easy to handle via pkg config.

Ah, I did not understand that this is something you were already working on and were asking me to test, I thought you were suggesting to use develop to install the software on our production cluster. I should not have assumed that, it's just something that is unfortunately quite frequent in the field of scientific software (just use the head of the git repo). I did not have any clone of the repo, so did not see that script (only the one in legacy) initially.

$ ./preconfigure.py --help
usage: preconfigure.py [-h] [--with-own-meson] [--no-codi] [--no-medi]
                       [--no-opdi] [--no-mpp] [--no-coolprop] [--no-mel]
                       [--no-mlpcpp]

optional arguments:
  -h, --help        show this help message and exit
  --with-own-meson  download own copies of Meson and Ninja
  --no-codi         do not download own copy of CoDiPack
  --no-medi         do not download own copy of MeDiPack
  --no-opdi         do not download own copy of OpDiLib
  --no-mpp          do not download own copy of Mutationpp
  --no-coolprop     do not download own copy of CoolProp
  --no-mel          do not download own copy of MEL
  --no-mlpcpp       do not download copy of MLpCpp

It seems like to go in the right direction. Is there a plan to provide options to specify how to find installed copies of those ? Something like --medi=/path/to/installed/medi and such ? pkgconfig or other auto-detection mechanisms usually work, but sometimes it doesn't, and explicit workarounds sometimes help.

I also don't see other packages that end up in externals, such as metis, parmetis, autotools, catch2. I am not sure how those are used (or if they are used), but I noticed folders for them in the externals folder. Is there a plan to allow using external versions of those ?

But your choices of words sound a bit unfriendly so I honestly don't care, we can also close the issue until the next release.

My apologies, I misunderstood your request.

pcarruscag commented 1 year ago

The developer who worked on it was @frx-wintermute in #1951 A lot of those packages are header-only, not compiled libraries, if you use different versions than the ones we use in our regression tests you have no guarantees that the code will work properly, and it seems you are mostly interested in stability. If you want to make changes to the meson.build script to look for local versions of libraries like cgns, parmetis, etc. see e.g. https://github.com/su2code/SU2/discussions/2064 We also have examples like this:

  pastix_dep = dependency('pastix', required: false)

  if not pastix_dep.found()
    pastix_root = get_option('pastix_root')+'/install'
    scotch_root = get_option('scotch_root')+'/lib'
    pastix_dep  = declare_dependency(include_directories: pastix_root,
                                    link_args: [ '-L../'+pastix_root, '-L../'+scotch_root,
                                    '-lpastix', '-lscotch', '-lptscotch', '-lptscotcherr',
                                    '-lm', '-lrt', '-lpthread'])
  endif
  su2_deps += pastix_dep

where we try to use pkg-config and fall back to specifying paths. I recommend using pkg-config whenever possible. If either you or @frx-wintermute are interested in complementing the feature by adding an option to prefer system libraries (via pkg-config, no paths because 99.9% of users will not know what to do) instead of the versions shipped with SU2, please go ahead and open a PR.

frx-wintermute commented 1 year ago

[...]

If either you or @frx-wintermute are interested in complementing the feature by adding an option to prefer system libraries (via pkg-config, no paths because 99.9% of users will not know what to do) instead of the versions shipped with SU2, please go ahead and open a PR.

Hello @pcarruscag!

To be honest, my daydream is: I would love to see SU2 shipping nothing in 'externals/' (or just empty subdirectories such as 'externals/meson/' ); then those external dependencies are by default downloaded from external repositories or tar archives. This defaul behavior would be obtained by configuring with the './meson.py' script and then by building with './ninja'.

But, if a user wants to build SU2 from source and prefers to use system-wide Meson and Ninja and system-wide versions of the external dependencies, it would be possible to avoid downloading these external dependencies. By running './preconfigure.py' with appropriate options, it would be possible to decide which external dependencies to not download. At that point, system-wide Meson and Ninja could be used to build SU2 from source. And the 'meson.build' file would do the following:

for each required or enabled (at configure time) dependency \<foo>, it would check whether it is found in 'externals/\<foo>/'; if not, it would check whether it can be found as a system-wide installation (perhaps via pkgconfig); if not, it would exit with error
for each dependency which is not required nor enabled at configure time, it would avoid using it at all

I think this daydream is achievable, and I hope that other people consider it as a useful goal.

su2code / SU2

Blocking download of dependencies #2086