Clean up dependencies

[h-vetinari]

Now that the revival (#75) happened, let's clean up the dependencies a bit. There's a lot under build: that doesn't belong there, not enough under host:, and a lot under run: that's covered by so-called "run-exports" by the respective host-dependencies.

A lot of this is visible also in the link check in CI, for example

WARNING (cyclus,lib/python3.10/site-packages/eventhooks.so): Needed DSO lib/libglibmm-2.68.so.1 found in ['conda-forge/linux-64::glibmm-2.68==2.78.1=h2f54ecc_0']
WARNING (cyclus,lib/python3.10/site-packages/eventhooks.so): .. but ['conda-forge/linux-64::glibmm-2.68==2.78.1=h2f54ecc_0'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)

tells us that eventhooks.so uses symbols from glibmm, and we are thus exposed to glibmm's ABI, and must therefore make it a host-dependence (so that cyclus can be recompiled correctly if an ABI-breaking version of glibmm is released).

The diff is probably easier to review commit-by-commit.

[conda-forge-webservices[bot]]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

OK @conda-forge/cyclus, something needs to be fixed here (or upstream). Cyclus is ignoring all ways I can think of to correctly specify the path where the linker should look for libraries, and then failing with:

[  1%] Linking CXX shared library ../lib/eventhooks.so
/home/conda/feedstock_root/build_artifacts/cyclus_1715053292867/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lxml++-4.0: No such file or directory
/home/conda/feedstock_root/build_artifacts/cyclus_1715053292867/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lxml2: No such file or directory
/home/conda/feedstock_root/build_artifacts/cyclus_1715053292867/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lglibmm-2.68: No such file or directory
/home/conda/feedstock_root/build_artifacts/cyclus_1715053292867/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgobject-2.0: No such file or directory
/home/conda/feedstock_root/build_artifacts/cyclus_1715053292867/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lglib-2.0: No such file or directory
/home/conda/feedstock_root/build_artifacts/cyclus_1715053292867/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lsigc-3.0: No such file or directory
collect2: error: ld returned 1 exit status

Note, the current workaround (even if unintentional) of putting the required dependencies into the build environment is not a long-term solution and needs to be removed.

[gonuke]

Thanks for your efforts here @h-vetinari . Can you point us to documentation about the subtleties of the different environments so that we can better understand how they fit into our ecosystem?

[h-vetinari]

Can you point us to documentation about the subtleties of the different environments so that we can better understand how they fit into our ecosystem?

Hey @gonuke, I've wanted to write something up about this for a long time, as it's a thing that takes people a long time to grasp. It'll hopefully happen soon-ish in the context of recent pushes to update the docs of https://github.com/conda-forge/conda-forge.github.io (see issues there).

In the meantime, here's a comment I wrote not too long ago that goes some way towards explaining things:

I still don't have a firm grasp of what goes in build vs. host vs. run. So if it really belongs elsewhere I am happy to change it. Or if you can suggest some documentation / tutorial that explains how each ought to be used, and would be thankful for that too.

It's a thing that's hard to grasp indeed. I think the cross-compilation-case is actually the most instructive. Imagine a x64 agent, but we want to compile a library for aarch64. That means the stuff we need in the build environment (compiler, linker, etc. - everything that's actually executed during the build) needs to match the x64 architecture.

On the other hand, the host environment needs to match the target architecture (here aarch64), and any libraries we're compiling against must be in the host-environment, with the right ABI. Compiling against a static or header-only library only requires a host-dependence (everything necessary gets consumed during compilation), whereas compiling against a shared library generates a runtime requirement (a "run-export" from host: to run:), which ensures that the actual DSO that's necessary to provide the symbols at runtime will be present. That's why we need libgoogle-cloud-X-devel in host, but only run-export libgoogle-cloud-X.

While there are some exceptions (e.g., we might need libprotobuf in the build environment for cross-compilation, because we need to execute protoc, which we can only do with the CPU arch of the agent), it's generally a sign of something going wrong to include "libraries I need to compile against" in the build environment. What's likely happening is that something somewhere is using a relative lookup (or some other heuristic, e.g. relative to the compilers) that expects the libraries in $BUILD_PREFIX, but we really should point that logic to $PREFIX (==host). In CMake that's usually something like CMAKE_PREFIX_PATH=$PREFIX.

Now the above focussed on the cross-compilation part for illustration purposes, but it holds as well for the native compilation case. The build environment is where the compilers and other build tools live, which are unrelated (including any transitive dependencies they might drag along) from the libraries we're compiling against.

In short though, cyclus' build setup needs to respect when we set CMAKE_PREFIX_PATH=$PREFIX (ideally), or any of the other ways I've tried to tell the linker where to look for the right symbols.

[bennibbelink]

Thank you for this information @h-vetinari you've been very helpful! We'll probably need to resolve some of this upstream, and will update you with our progress (with more questions I'm sure 😁)