amogan/issue139 Parse CMakeLists dependencies automatically

[andrewmogan]

This PR modifies make-release-repo.py and the package.py files for both FD and ND packages. The depends_on("package") list in the spack recipes are replaced by XDEPENDSX which, similar to XVERSIONX and XHASHX, will be replaced when generating the package repository. Basically, DAQRelease.generate_daq_package will now parse each package's CMakeLists.txt and infer dependencies from find_package calls. Some exceptions are handled with brute force if/elif statements. The Alma9 nightly build succeeds when using this branch, and I can successfully create a local area based on this branch from cvmfs.

[andrewmogan]

Forgot to reference Issue #139

[jcfreeman2]

Thanks a lot. This overall looks good, but there are three things to address: 1) While it's currently the case that far-detector-specific packages and near-detector-specific packages get the benefits of the new algorithm, it appears that this is not the case for the common packages (e.g. appfwk, ipm, etc.) 2) When get_cmake_dependencies is called, regardless of the desired branch of the package in question, it's always the CMakeLists.txt file from the develop branch which gets parsed 3) If a failure to curl a CMakeLists.txt file occurs, currently this parses as "print a warning and give the package in question no dependencies". We should instead treat this as a catastrophic error and have the script exit at this point; see this article by software guru Martin Fowler on the principle behind this advice.

[jcfreeman2]

Now we're getting to fine tuning things. I did a side-by-side comparison of all the package.py files both using the traditional, "hardwired" package.pys and the package.pys generated by this script. The following need to be addressed, and once this is done we should be good to go:

• The script appears to not ignore commented lines. E.g., the line #find_package(Boost ... in appdal's CMakeLists.txt results in the addition of depends_on("boost") in the package.py file.
• It appears that devtools doesn't depend on systems, and externals doesn't depend on devtools in the package.py files which appear; this should be restored.
• hdf5libs depends on highfive, and the script correctly picks up on this. However, not unlike the situation we ran into with the folly dependency, we need to make sure that highfive is built with MPI support, i.e. depends_on("highfive +mpi") even though this fact isn't provided in CMakeLists.txt
• Similar to what we observed with find_package(felix ...., a couple more packages don't have names which are precisely referred to in CMakeLists.txt. Mapping their CMakeLists.txt names to their Spack names, they are:
  • PkgConfig -> pkgconf (from readoutlibs)
  • TBB -> intel-tbb (from cmdlib)
• In a couple of cases, there are dependencies which should exist but aren't reflected in the CMakeLists.txt file. In these situations, we should just hardwire the dependencies into the template package.py just below the XDEPENDSX token. Specifically:
  • daq-cmake should have a dependency on py-pybind11. The reasoning behind this is that even though py-pybind11 isn't used to build daq-cmake itself, packages which use daq-cmake may need to use it for Python bindings.
  • erskafka should have dependencies on cyrus-sasl and openssl. To be honest I can't recall the original motivation for this; until I figure it out we should have them in there based on the principle of Chesterton's fence

[jcfreeman2]

With the current head of the feature branch, I successfully created a test nightly for the v4.3.0 release (https://github.com/DUNE-DAQ/daq-release/actions/runs/8270039867) and ran minimal_system_quick_test.py with it both on daq.fnal.gov and np04-srv-015. Approved.