search

DUNE-DAQ / daq-buildtools

Make life for developers easier through a collection of CMake functions and scripts
0 stars 1 forks source link

Link schemas between different repositories #124

Closed mroda88 closed 2 years ago

mroda88 commented 3 years ago

appfwk has the capability to include cmdlib and rcif jsonnet files in the appfwk jsonnet, see for example https://github.com/DUNE-DAQ/appfwk/blob/099c93f71d3daa294d427de92f18f47050c6c6c5/schema/appfwk/app.jsonnet#L13

I want to do the same between dfmessages and dfmodules, specifically dfmodules needs to include jsonnet files from dfmessages. I attempted the exercise using https://github.com/DUNE-DAQ/dfmessages/tree/mroda/data_request_modes not the default branch. As you can see, this branch for dfmessages does the following

  1. defines a new schema file
  2. adds the line ``daq_codegen(*.jsonnet TEMPLATES Structs.hpp.j2 Nljs.hpp.j2 )```` in CMakeLists.txt
  3. extends dfmessagesConfig.cmake.in in order to set dfmessages_DAQSHARE All these things are taken from appfwk.

What I want to do in the dfmodules is to change its daq_codegen from daq_codegen(*.jsonnet TEMPLATES Structs.hpp.j2 Nljs.hpp.j2 ) to daq_codegen(*.jsonnet DEP_PKGS dfmessages TEMPLATES Structs.hpp.j2 Nljs.hpp.j2 ) in order to "link" against those jsonnet files. If I do this, the dfmessages/schema directory is not created in the build directory and the compilation complains that the jsonnet files cannot be created.

I think the issues are two.

  1. What is the correct way to do this operation
  2. Check how can it be that a change in dfmodules causes dfmessages building to not complete, when its dfmodules that depends on dfmessages and not viceversa:
mroda88 commented 3 years ago

I should probably add that Florian tried it as well and he got the same results. So this is not specific to my build.

brettviren commented 3 years ago

Hi @mroda88

I don't know the CMake solution but I think you are right to think of Jsonnet imports like a "link" or maybe more closely a #include. The moo flag -M is then like GCC's -L or -I flags.

I'd suggest that whatever CMake mechanism allows #include to be resolved might be extended to provide moo -M ...

jcfreeman2 commented 3 years ago

When I try recreating the problem, what I see is the following error message:

[16/238] Generating codegen/include/dfmodules/hdf5datastore/Nljs.hpp
FAILED: dfmodules/codegen/include/dfmodules/hdf5datastore/Nljs.hpp 
cd /afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build && /afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/dbt-pyvenv/bin/moo -T/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/sourcecode/dfmodules/schema -T/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmessages/schema -M/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/sourcecode/dfmodules/schema -M/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmessages/schema -g /lang:ocpp.jsonnet -A path=dunedaq.dfmodules.hdf5datastore -A ctxpath=dunedaq -A os=dfmodules/hdf5datastore.jsonnet render -o /afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmodules/codegen/include/dfmodules/hdf5datastore/Nljs.hpp omodel.jsonnet onljs.hpp.j2 && /afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/dbt-pyvenv/bin/moo -T/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/sourcecode/dfmodules/schema -T/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmessages/schema -M/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/sourcecode/dfmodules/schema -M/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmessages/schema -g /lang:ocpp.jsonnet -A path=dunedaq.dfmodules.hdf5datastore -A ctxpath=dunedaq -A os=dfmodules/hdf5datastore.jsonnet render-deps -t dfmodules/codegen/include/dfmodules/hdf5datastore/Nljs.hpp -o /afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmodules/codegen/include/dfmodules/hdf5datastore/Nljs.hpp.d omodel.jsonnet onljs.hpp.j2
Usage: moo [OPTIONS] COMMAND [ARGS]...
Try 'moo --help' for help.

Error: Invalid value for '-T' / '--tpath': Directory '/afs/cern.ch/work/j/jofreema/public/daqbuild_issue124_test/build/dfmessages/schema' does not exist.

i.e., the build system attempts generate a dfmodules header, dfmodules/hdf5datastore/Nljs.hpp, via a moo call which relies on the dfmessages' schema directory having already been copied into the build/ area.

I can restate what I just wrote more formally in terms of the targets involved. The generation of dfmodules/hdf5datastore/Nljs.hpp is associated with a target which is designed to be the first target "built" in the dfmodules build. Similarly, the copying of the dfmessages schema/ directory into build/ is associated with a target which is designed to be the first target "built" in the dfmessages build. It seems like we may need to ensure that if we call daq_codegen with one or more packages suppied via the DEP_PKGS argument we need to ensure that the first target handled in the package-in-question isn't "built" until the first targets of the dependent packages have already been "built".

brettviren commented 3 years ago

An aside to this overall issue but about Invalid value for '-T' / '--tpath', I can relax this in moo. I had no strong intention to elevate it to a fatal error (I took the click module's default interpretation). Unless there is a reason to prefer the hard error I will change moo to ignore any missing template directory and emit a warning.

dingp commented 2 years ago

@jcfreeman2 Can you check if the issue still exits in 2.11 with moo 0.5.7? We can close the issue if not.

jcfreeman2 commented 2 years ago

This was actually addressed via daq-cmake commit 85f147122bc3, used to address https://github.com/DUNE-DAQ/daq-cmake/issues/49. In a nutshell, if a package is specified as an argument to DEP_PKGS in daq_codegen AND that dependent package is also a repo in sourcecode/, daq-cmake will simply access the schema files in sourcecode/<dependent package>/schema rather than expecting that subdirectory to be in the build area.