Resolved path of dependent files not appearing in output *.d file

[jcfreeman2]

Phil's noticed that with the recent switch from 0.5.7 to 0.6.1, some of the dependencies listed in a *.d file produced by moo don't actually exist, forcing unnecessary full rebuilds even when *.jsonnet files haven't been touched. This can be seen if we build, say, the cmdlib package. After logging onto an np04 node, and doing the following:

cd <directory above your workarea>
source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt dunedaq-v3.1.1  
dbt-create -c -n last_successful daqbuild_moo_dependency_study
cd daqbuild_moo_dependency_study/sourcecode
git clone https://github.com/DUNE-DAQ/cmdlib
cd ..
dbt-workarea-env
dbt-build
less ./build/cmdlib/codegen/include/cmdlib/cmd/Structs.hpp.d

...you then see the following output (newlines added for readability):

cmdlib/codegen/include/cmdlib/cmd/Structs.hpp: 
<directory above your workarea>/daqbuild_moo_dependency_study/build/omodel.jsonnet 
<directory above your workarea>/daqbuild_moo_dependency_study/build/ostructs.hpp.j2 
<directory above your workarea>/daqbuild_moo_dependency_study/dbt-pyvenv/lib/python3.8/site-packages/moo/templates/ocpp.hpp.j2 
<directory above your workarea>/daqbuild_moo_dependency_study/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code/oschema.jsonnet

Where omodel.jsonnet and ostructs.hpp.j2 don't actually exist in their listed directories. moo resolve indicates that during the build they're taken from <directory above your workarea>/daqbuild_moo_dependency_study/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code and templates, respectively.

The upshot is that if you rerun dbt-build, even without touching anything in cmdlib a full, bottom-up build is actuated. This would be fixed if the actual resolved path (rather than what appears to be the path moo is executing in) appeared in the *.d files for omodel.jsonnet, etc.

[brettviren]

This is a guess, @jcfreeman2

Your reported #30 got fixed a while ago but post 0.5.7 and so is seen first in a release in 0.6.1. That fix made moo render-deps include the model and template file given on the command line to the list of deps.

So, what I guess is happening is we now see omodel.jsonnet and ostructs.hpp.j2 showing up in the deps list for the first time. Is it correct that these files are not listed at all when moo 0.5.7 is used?

If so, I further guess they are being located in build/ because they are being copied there by the DAQ build system (?) and then those copies are given to moo render-deps CLI during the first build, then they are being removed from build/ and then the subsequent rebuild copies them anew thus getting new mtimes. Since you say they are missing after the re-build, this story requires that re-build to also re-remove them.

Does this story fit?

If so, I wonder if the DAQ build system should be changed so as to not copy these files to build/ but rely on them being found automatically as part of moo's installation area?

[jcfreeman2]

That's certainly a plausible hypothesis, but what I'm finding is that if I directly run the moo render-deps command from the command line (render-deps being what daq-cmake calls) it remains the case that regardless of the directory you're in - <workarea>/build or otherwise - it saves the paths of omodel.jsonnet and ostructs.hpp.j2 as whatever directory you're executing the command out of. E.g., if I've already set up a workarea as described above, if I do the following:

<workarea>/dbt-pyvenv/bin/moo \
-T <workarea>/sourcecode/cmdlib/schema \
-M <workarea>/sourcecode/cmdlib/schema \
-g /lang:ocpp.jsonnet \
-A path=dunedaq.cmdlib.cmd -A ctxpath=dunedaq -A os=cmdlib/cmd.jsonnet \
render-deps -t <workarea>/build/cmdlib/codegen/include/cmdlib/cmd/Structs.hpp 
-o ./Structs.hpp.d  \
omodel.jsonnet \
ostructs.hpp.j2

then if I look at ./Structs.hpp.d it'll look something like the following (the example below, I was running moo out of my own Documents folder):

/home/jcfree/daqbuild_sep20/build/cmdlib/codegen/include/cmdlib/cmd/Structs.hpp:
/home/jcfree/Documents/omodel.jsonnet
/home/jcfree/Documents/ostructs.hpp.j2
/home/jcfree/daqbuild_sep20/dbt-pyvenv/lib/python3.8/site-packages/moo/templates/ocpp.hpp.j2
/home/jcfree/daqbuild_sep20/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code/oschema.jsonnet

...where it's important to note that at no point was there an omodel.jsonnet or an ostructs.hpp.j2 file in my Documents directory.

[jcfreeman2]

I should add that moo resolve will provide the expected path, e.g.

<workarea>/dbt-pyvenv/bin/moo resolve oschema.jsonnet

will return

/home/jcfree/daqbuild_sep20/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code/oschema.jsonnet

[brettviren]

Ah! Yes, thanks, this makes things very clear now.

moo currently assumes the user gives valid paths for the model and template files on the CLI.

It's easy enough and friendly to resolve those files as well. I'll add this now and make a new release.

[jcfreeman2]

The line in moo which got changed resulting in the 0.6.2 tag does indeed fix the problem; with the new call toresolve on the model and template files I confirmed we got the correct path of the two dependencies in the *.d file, rather than incorrect paths.

Following this confirmation, Kurt suggested we check that fixing the paths of the dependencies not alter moo's behavior elsewhere in our system. So I did the following: 1) Confirmed that the header files which get produced by moo during builds of our repos didn't have their output code changed due to the path fix, i.e. Structs.hpp, Nljs.hpp, InfoStructs.hpp and InfoNljs.hppare the same whether we use the fixed line or the previous line. 2) With the code passing the diff test, I then ran integration tests of our software components built with and without the fixed line in moo. Using the instructions in the dunedaq-v3.2.0 test spreadsheet I found that minimal_system_quick_test and fake_data_producer_test yielded the same (passing) result both with and without the line fix.