Closed jcfreeman2 closed 1 year ago
This is a guess, @jcfreeman2
Your reported #30 got fixed a while ago but post 0.5.7 and so is seen first in a release in 0.6.1. That fix made moo render-deps
include the model and template file given on the command line to the list of deps.
So, what I guess is happening is we now see omodel.jsonnet
and ostructs.hpp.j2
showing up in the deps list for the first time. Is it correct that these files are not listed at all when moo 0.5.7 is used?
If so, I further guess they are being located in build/
because they are being copied there by the DAQ build system (?) and then those copies are given to moo render-deps
CLI during the first build, then they are being removed from build/
and then the subsequent rebuild copies them anew thus getting new mtimes. Since you say they are missing after the re-build, this story requires that re-build to also re-remove them.
Does this story fit?
If so, I wonder if the DAQ build system should be changed so as to not copy these files to build/
but rely on them being found automatically as part of moo's installation area?
That's certainly a plausible hypothesis, but what I'm finding is that if I directly run the moo render-deps
command from the command line (render-deps
being what daq-cmake
calls) it remains the case that regardless of the directory you're in - <workarea>/build
or otherwise - it saves the paths of omodel.jsonnet
and ostructs.hpp.j2
as whatever directory you're executing the command out of. E.g., if I've already set up a workarea as described above, if I do the following:
<workarea>/dbt-pyvenv/bin/moo \
-T <workarea>/sourcecode/cmdlib/schema \
-M <workarea>/sourcecode/cmdlib/schema \
-g /lang:ocpp.jsonnet \
-A path=dunedaq.cmdlib.cmd -A ctxpath=dunedaq -A os=cmdlib/cmd.jsonnet \
render-deps -t <workarea>/build/cmdlib/codegen/include/cmdlib/cmd/Structs.hpp
-o ./Structs.hpp.d \
omodel.jsonnet \
ostructs.hpp.j2
then if I look at ./Structs.hpp.d
it'll look something like the following (the example below, I was running moo
out of my own Documents
folder):
/home/jcfree/daqbuild_sep20/build/cmdlib/codegen/include/cmdlib/cmd/Structs.hpp:
/home/jcfree/Documents/omodel.jsonnet
/home/jcfree/Documents/ostructs.hpp.j2
/home/jcfree/daqbuild_sep20/dbt-pyvenv/lib/python3.8/site-packages/moo/templates/ocpp.hpp.j2
/home/jcfree/daqbuild_sep20/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code/oschema.jsonnet
...where it's important to note that at no point was there an omodel.jsonnet
or an ostructs.hpp.j2
file in my Documents
directory.
I should add that moo resolve
will provide the expected path, e.g.
<workarea>/dbt-pyvenv/bin/moo resolve oschema.jsonnet
will return
/home/jcfree/daqbuild_sep20/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code/oschema.jsonnet
Ah! Yes, thanks, this makes things very clear now.
moo currently assumes the user gives valid paths for the model and template files on the CLI.
It's easy enough and friendly to resolve those files as well. I'll add this now and make a new release.
The line in moo
which got changed resulting in the 0.6.2
tag does indeed fix the problem; with the new call toresolve
on the model and template files I confirmed we got the correct path of the two dependencies in the *.d
file, rather than incorrect paths.
Following this confirmation, Kurt suggested we check that fixing the paths of the dependencies not alter moo
's behavior elsewhere in our system. So I did the following:
1) Confirmed that the header files which get produced by moo
during builds of our repos didn't have their output code changed due to the path fix, i.e. Structs.hpp
, Nljs.hpp
, InfoStructs.hpp
and InfoNljs.hpp
are the same whether we use the fixed line or the previous line.
2) With the code passing the diff
test, I then ran integration tests of our software components built with and without the fixed line in moo
. Using the instructions in the dunedaq-v3.2.0 test spreadsheet I found that minimal_system_quick_test
and fake_data_producer_test
yielded the same (passing) result both with and without the line fix.
Phil's noticed that with the recent switch from 0.5.7 to 0.6.1, some of the dependencies listed in a
*.d
file produced bymoo
don't actually exist, forcing unnecessary full rebuilds even when*.jsonnet
files haven't been touched. This can be seen if we build, say, thecmdlib
package. After logging onto an np04 node, and doing the following:...you then see the following output (newlines added for readability):
Where
omodel.jsonnet
andostructs.hpp.j2
don't actually exist in their listed directories.moo resolve
indicates that during the build they're taken from<directory above your workarea>/daqbuild_moo_dependency_study/dbt-pyvenv/lib/python3.8/site-packages/moo/jsonnet-code
andtemplates
, respectively.The upshot is that if you rerun
dbt-build
, even without touching anything incmdlib
a full, bottom-up build is actuated. This would be fixed if the actual resolved path (rather than what appears to be the pathmoo
is executing in) appeared in the*.d
files foromodel.jsonnet
, etc.