Putting install scripts into dockerfile

[shimwell]

work in progress, not ready for review

Description

As mentioned in project https://github.com/svalinn/DAGMC/projects/6 it has been proposed to move the install scripts into the dockerfile.

Motivation and Context

This simplifies the passing of environments to cmake and other commands as they are all visible in one file and no longer creating in one script to be used in another.

Changes

What kind of change does this PR introduce? (Bug fix, feature, documentation update...) refactored dockerfile to include more ARGs ENVs and install commands.

Behavior

What is the current behavior? What is the new behavior? old = dockerfile copies in sh scripts and runs them, new = dockerfile runs install commands directly

[shimwell]

When building this new dockerfile with the defaults (moab 5.4.0) I am getting a Pymoab / PYTHONPATH error which is similar to the error I am seeing on PR #816

 58%] Building CXX object test/CMakeFiles/intx_in_plane_test.dir/intx_in_plane_test.cpp.o
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/bld/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/bld/pymoab:
    running develop
    Checking .pth file support in /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages
    /usr/bin/python3 -E -c pass
    TEST FAILED: /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH

    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /root/build_dir/moab/bld/pymoab/lib/python3.6/site-packages

    and your PYTHONPATH environment variable currently contains:

        ''

    Here are some of your options for correcting the problem:

    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files

    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)

    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:

      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
  Can't roll back pymoab; was not uninstalled
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/bld/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/bld/pymoab" failed with error code 1 in /root/build_dir/moab/bld/pymoab/
pymoab/CMakeFiles/pymoab-local-install.dir/build.make:58: recipe for target 'pymoab/CMakeFiles/pymoab-local-install' failed
make[2]: *** [pymoab/CMakeFiles/pymoab-local-install] Error 1
CMakeFiles/Makefile2:1238: recipe for target 'pymoab/CMakeFiles/pymoab-local-install.dir/all' failed
make[1]: *** [pymoab/CMakeFiles/pymoab-local-install.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

[shimwell]

A moab compile error has appeared when trying to build this docker file locally with moab 5.4.0.

Building CXX object test/CMakeFiles/intx_in_plane_test.dir/intx_in_plane_test.cpp.o
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/root/build_dir/moab/build/pymoab/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --prefix=/root/build_dir/moab/build/pymoab:
    running develop
    Checking .pth file support in /root/build_dir/moab/build/pymoab/lib/python3.6/site-packages
    /usr/bin/python3 -E -c pass
    TEST FAILED: /root/build_dir/moab/build/pymoab/lib/python3.6/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH

    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /root/build_dir/moab/build/pymoab/lib/python3.6/site-packages

    and your PYTHONPATH environment variable currently contains:

        ''

    Here are some of your options for correcting the problem:

    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files

    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)

    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:

      https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
  Can't roll back pymoab; was not uninstalled

[gonuke]

For some reason, these latest commits didn't seem to trigger rebuild of the images on your fork???

[shimwell]

Sorry about that, I've just manually trigger it https://github.com/shimwell/DAGMC/actions/runs/2939772739

[shimwell]

I just noticed that the last commit here passed the CI. Is it worth catching up on this PR at the next DAGMC meeting

[shimwell]

I think that removing either of the ENV would cause the ENV not to be found in that build stage. So I opted for adding the hdf5 install path as a global ARG instead as that can be a single string that is accessed by both build stages.

[gonuke]

I think that removing either of the ENV would cause the ENV not to be found in that build stage. So I opted for adding the hdf5 install path as a global ARG instead as that can be a single string that is accessed by both build stages.

Hmmmm... I think I need to do some independent testing of this on a minimum example. My understanding was somewhat reversed. A different, but important, question is which is used to perform substitution in a docker command. Is the ENV available for substitution in a RUN command that is in the dockerfile, or does it need to be an ARG anyway???

[shimwell]

I think it is the ARG values that can be set with a docker run build-arg command.

This thread has a few minimal examples in that might be handy https://github.com/moby/moby/issues/37345

[gonuke]

This thread has a few minimal examples in that might be handy moby/moby#37345

This example is a little different because both stages start from the fresh base image - the second stage doesn't build on the first stage.

Try this:

from ubuntu:20.04 as stage1

ENV foo=test1
ARG bar=test2
RUN echo ${foo}
RUN echo ${bar}
ENV foo=test3

from stage1 as stage2
RUN echo ${foo}
RUN echo ${bar}

[shimwell]

Thanks for that example. I see the ENV got to the stage2 and the ARG didn't. Sorry for my misunderstanding

[shimwell]

Super. I synced the fork which appears to have triggered some ci and manually trigger some docker file building CI over here https://github.com/shimwell/DAGMC/actions/runs/3823442736

Note to self. I noticed a few of our status badges on the main readme are red

[shimwell]

manual triggered CI failed while trying to compile MOAB

error: bad install directory or PYTHONPATH

[pshriwise]

@shimwell I noted that in our latest CI run after merging #845. I have a proposed fix submitted to MOAB, so hopefully that will be resolved soon.

I'd vote we go ahead and merge this if @gonuke doesn't have further comments. Once that PR goes into MOAB, I'll make sure to re-run our CI here and confirm it fixes our issue.

[gonuke]

But merging this will cause all of our CI images to fail and leave us without any CI, I think

[shimwell]

Thanks for making this python path fix PR @pshriwise mentioned.

I guess it would take effect with the next release of MOAB, would it make sense to therefore wait for the next release of MOAB before merging this PR that changes the dockerfile

[pshriwise]

@gonuke I guess I was assuming that this would only affect the master and develop builds of MOAB, but even those failing would cause the others to be canceled, as seen in this build https://github.com/svalinn/DAGMC/actions/runs/3715840076 (the one successful job is present because I manually restarted it).

At any rate, my thought was that merging this isn't going to change the current state of our CI for a merge to our develop, unfortunately, so I figured we may as well wrap it up if there aren't any other issues.

I've opened up #847 to revisit how we handle experimental builds and continue this thread of discussion.

[gonuke]

It is possible that this won't change our CI status - but only because it will fail :). We do stage the new CI images and test them before we commit to them as the official base image for testing, so it's likely that we would not disrupt our current CI test images. However, I worry about building in some technical debt by merging this now.

[pshriwise]

Either way is good w/ me 👍🏻

[shimwell]

I have downgraded to moab 5.3.0 for consistency with other parts of the CI

I have manually trigger the docker image building https://github.com/shimwell/DAGMC/actions/runs/3847616064

[gonuke]

Looks like a new cause of failure - you might need to move/copy the contents of the "install.sh" script into to bootstrap test job further down this docker_publish.yml file

[shimwell]

ah yes, sorry there is more work to be done here. Before this PR is ready

I proposed to:

1. add a stage for DAGMC building to the dockerfile
2. rework the test_img stage of docker_publish.yml so that it builds this new stage of the Dockerfile with the various argument combinations.
3. Add new variables such as FC, static_exe, build_mw_reg_tests which I can introduce as ARGs to the Dockerfile

I have started on this with the last commit

[shimwell]

I have added stages for building and testing dagmc to the dockerfile I have also reworked the dockr_publish.yml to build these stages.

The docker build ci didn't need manually trigger this time and can been seen over here

The current error while building the dockerfile locally with default ARGS is on the last step where I run a cmake . -DDAGMC_ROOT=${dagmc_install_dir} command

Step 80/80 : RUN cd ${dagmc_build_dir}/cmake/test_config &&     cmake . -DDAGMC_ROOT=${dagmc_install_dir} &&     make all test
 ---> Running in 1f41a2bf7a80
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- /root/opt/dagmc
-- Found DAGMC in DAGMC_CMAKE_CONFIG-NOTFOUND
CMake Error at FindDAGMC.cmake:20 (include):
  include could not find load file:

    DAGMC_CMAKE_CONFIG-NOTFOUND/DAGMCConfig.cmake
Call Stack (most recent call first):
  CMakeLists.txt:11 (find_package)

[shimwell]

After adding the dagmc_test stage to the dockerfile it took me a while to get it building locally

Happy to say that it is now working locally with the extra testing and building of dagmc

Let us see if the docker build CI works over here https://github.com/shimwell/DAGMC/actions/runs/3913008543

[gonuke]

This is failing now in a strange place - make_watertight testing??

[shimwell]

That is a new fail I've not seen before. I was building the dockerfile locally with the defaults and it built successfully. I can try building that particular combination of build args locally and try to reproduce.

Won't be able to do this till late next week as I am on hotel WiFi for a while

[shimwell]

this is failing on my local CI https://github.com/shimwell/DAGMC/actions/runs/3917829965

[shimwell]

I am trying to build the specific dockerfile with the same build args that failed with this command

docker build -t dagmc --build-arg UBUNTU_VERSION=18.04 --build-arg COMPILER=clang --build-arg HDF5=1.10.4 --build-arg MOAB=master --build-arg static_exe=OFF -f CI/Dockerfile .

[shimwell]

reporting on that docker image build with the build args that match the failing test

It build successfully with and without the addition PATH=${dagmc_install_dir}/bin:$PATH before the make test

[gonuke]

I think we need to turn on some verbose reporting of tests to figure this out

[shimwell]

verbose

I'm not quite sure how to make a verbose output from the make all test test command.

Also the docker file is building locally which is nice, it passes the tests

-- Installing: /root/opt/dagmc/tests/small_overlap.h5m
Removing intermediate container db51e2b9bba7
 ---> b6de7c8119cc
Step 78/84 : FROM dagmc as dagmc_test
 ---> b6de7c8119cc
Step 79/84 : ARG dagmc_build_dir
 ---> Running in f3009d86ebaa
Removing intermediate container f3009d86ebaa
 ---> 2a9292f709b7
Step 80/84 : ARG dagmc_install_dir
 ---> Running in a4b7e3f86f8d
Removing intermediate container a4b7e3f86f8d
 ---> d741ee9bc61d
Step 81/84 : ENV VERBOSE=1
 ---> Running in 03c59aa9a7ba
Removing intermediate container 03c59aa9a7ba
 ---> da1142b01bf5
Step 82/84 : RUN cd ${dagmc_build_dir}/build &&     PATH=${dagmc_install_dir}/bin:$PATH make test
 ---> Running in 2e28c8cef8df
Running tests...
/usr/bin/ctest --force-new-ctest-process 
Test project /root/build_dir/dagmc/build
      Start  1: dagmc_unit_tests
 1/25 Test  #1: dagmc_unit_tests ........................   Passed    0.72 sec
      Start  2: dagmc_pointinvol_test
 2/25 Test  #2: dagmc_pointinvol_test ...................   Passed    0.71 sec
      Start  3: dagmc_rayfire_test
 3/25 Test  #3: dagmc_rayfire_test ......................   Passed    0.32 sec
      Start  4: dagmc_simple_test
 4/25 Test  #4: dagmc_simple_test .......................   Passed    0.38 sec
      Start  5: dagmc_graveyard_test
 5/25 Test  #5: dagmc_graveyard_test ....................   Passed   11.79 sec
      Start  6: uwuw_unit_tests
 6/25 Test  #6: uwuw_unit_tests .........................   Passed    0.09 sec
      Start  7: uwuw_unit_tests_tally
 7/25 Test  #7: uwuw_unit_tests_tally ...................   Passed    0.02 sec
      Start  8: uwuw_unit_tests_preprocessor
 8/25 Test  #8: uwuw_unit_tests_preprocessor ............   Passed    1.00 sec
      Start  9: test_KDEKernel
 9/25 Test  #9: test_KDEKernel ..........................   Passed    0.01 sec
      Start 10: test_KDEMeshTally
10/25 Test #10: test_KDEMeshTally .......................   Passed    1.31 sec
      Start 11: test_KDENeighborhood
11/25 Test #11: test_KDENeighborhood ....................   Passed    1.04 sec
      Start 12: test_PolynomialKernel
12/25 Test #12: test_PolynomialKernel ...................   Passed    0.01 sec
      Start 13: test_Quadrature
13/25 Test #13: test_Quadrature .........................   Passed    0.01 sec
      Start 14: test_CellTally
14/25 Test #14: test_CellTally ..........................   Passed    0.01 sec
      Start 15: test_TallyEvent
15/25 Test #15: test_TallyEvent .........................   Passed    0.00 sec
      Start 16: test_TallyData
16/25 Test #16: test_TallyData ..........................   Passed    0.04 sec
      Start 17: test_Tally
17/25 Test #17: test_Tally ..............................   Passed    6.67 sec
      Start 18: test_TrackLengthMeshTally
18/25 Test #18: test_TrackLengthMeshTally ...............   Passed  247.76 sec
      Start 19: make_watertight_cylinder_tests
19/25 Test #19: make_watertight_cylinder_tests ..........   Passed    1.68 sec
      Start 20: make_watertight_cone_tests
20/25 Test #20: make_watertight_cone_tests ..............   Passed  101.21 sec
      Start 21: make_watertight_no_curve_sphere_tests
21/25 Test #21: make_watertight_no_curve_sphere_tests ...   Passed   10.54 sec
      Start 22: make_watertight_sphere_n_box_test
22/25 Test #22: make_watertight_sphere_n_box_test .......   Passed   10.96 sec
      Start 23: dagmcnp_unit_tests
23/25 Test #23: dagmcnp_unit_tests ......................   Passed    0.28 sec
      Start 24: dagsolid_unit_tests
24/25 Test #24: dagsolid_unit_tests .....................   Passed    0.70 sec
      Start 25: overlap_check_test
25/25 Test #25: overlap_check_test ......................   Passed   59.56 sec

100% tests passed, 0 tests failed out of 25

Total Test time (real) = 456.85 sec
Removing intermediate container 2e28c8cef8df
 ---> 02c30c3ddf18
Step 83/84 : RUN cd ${dagmc_build_dir} &&     git clean -dxf .
 ---> Running in fc7fea2252ab
Removing build/
Removing src/pyne/pyne.cpp
Removing src/pyne/pyne.h
Removing intermediate container fc7fea2252ab
 ---> a6d3443ed6e3
Step 84/84 : RUN cd ${dagmc_build_dir}/cmake/test_config &&     cmake . -DDAGMC_ROOT=${dagmc_install_dir} &&     make all test
 ---> Running in a718dfb750a8
Re-run cmake no build system arguments
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- /root/opt/dagmc
-- Found DAGMC in /root/opt/dagmc/lib/cmake/dagmc
-- Configuring DAGMC 3.2.2
-- Configuring done
-- Generating done
-- Build files have been written to: /root/build_dir/dagmc/cmake/test_config
/usr/bin/cmake -H/root/build_dir/dagmc/cmake/test_config -B/root/build_dir/dagmc/cmake/test_config --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /root/build_dir/dagmc/cmake/test_config/CMakeFiles /root/build_dir/dagmc/cmake/test_config/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/root/build_dir/dagmc/cmake/test_config'
make -f CMakeFiles/test_DAGMCConfig.dir/build.make CMakeFiles/test_DAGMCConfig.dir/depend
make[2]: Entering directory '/root/build_dir/dagmc/cmake/test_config'
cd /root/build_dir/dagmc/cmake/test_config && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /root/build_dir/dagmc/cmake/test_config /root/build_dir/dagmc/cmake/test_config /root/build_dir/dagmc/cmake/test_config /root/build_dir/dagmc/cmake/test_config /root/build_dir/dagmc/cmake/test_config/CMakeFiles/test_DAGMCConfig.dir/DependInfo.cmake --color=
Dependee "/root/build_dir/dagmc/cmake/test_config/CMakeFiles/test_DAGMCConfig.dir/DependInfo.cmake" is newer than depender "/root/build_dir/dagmc/cmake/test_config/CMakeFiles/test_DAGMCConfig.dir/depend.internal".
Dependee "/root/build_dir/dagmc/cmake/test_config/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/root/build_dir/dagmc/cmake/test_config/CMakeFiles/test_DAGMCConfig.dir/depend.internal".
Scanning dependencies of target test_DAGMCConfig
make[2]: Leaving directory '/root/build_dir/dagmc/cmake/test_config'
make -f CMakeFiles/test_DAGMCConfig.dir/build.make CMakeFiles/test_DAGMCConfig.dir/build
make[2]: Entering directory '/root/build_dir/dagmc/cmake/test_config'
[ 50%] Building CXX object CMakeFiles/test_DAGMCConfig.dir/test_DAGMCConfig.cpp.o
/usr/bin/c++   -I/root/opt/dagmc/include -I/root/opt/moab/include -I/usr/include/eigen3  -std=gnu++1z -o CMakeFiles/test_DAGMCConfig.dir/test_DAGMCConfig.cpp.o -c /root/build_dir/dagmc/cmake/test_config/test_DAGMCConfig.cpp
[100%] Linking CXX executable test_DAGMCConfig
/usr/bin/cmake -E cmake_link_script CMakeFiles/test_DAGMCConfig.dir/link.txt --verbose=1
/usr/bin/c++    -rdynamic CMakeFiles/test_DAGMCConfig.dir/test_DAGMCConfig.cpp.o  -o test_DAGMCConfig  -L/root/opt/dagmc/lib  -L/root/opt/moab/lib -Wl,-rpath,/root/opt/dagmc/lib:/root/opt/moab/lib -lMOAB -ldagmc -luwuw -lpyne_dagmc -ldagtally -lmakeWatertight -ldagsolid 
make[2]: Leaving directory '/root/build_dir/dagmc/cmake/test_config'
[100%] Built target test_DAGMCConfig
make[1]: Leaving directory '/root/build_dir/dagmc/cmake/test_config'
/usr/bin/cmake -E cmake_progress_start /root/build_dir/dagmc/cmake/test_config/CMakeFiles 0
Running tests...
/usr/bin/ctest --force-new-ctest-process 
Test project /root/build_dir/dagmc/cmake/test_config
    Start 1: DAGMC-Configuration-Test
1/1 Test #1: DAGMC-Configuration-Test .........   Passed    0.02 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.02 sec
Removing intermediate container a718dfb750a8
 ---> b67409f26766
[Warning] One or more build-args [COMPILER HDF5 MOAB] were not consumed
Successfully built b67409f26766
Successfully tagged dagmc:latest
(openmc_dev) jshimwell@FL000093:~/DAGMC-my_fork$ 

[gonuke]

Maybe this? https://stackoverflow.com/questions/5709914/using-cmake-how-do-i-get-verbose-output-from-ctest

[shimwell]

Attempting to rerun the dockerfile building over here on my fork CI

[gonuke]

As @bam241 suspected, this has something to do with the tests not finding the H5M files needed for regression testing.

[gonuke]

I think this may be an ARG/ENV thing. Your dagmc_test stage only refers dagmc_build_dir as ARG but they are defined in the previous stage only as ENV. I know we discussed this before, but this seems like a fragile construct.

I also noticed that the dagmc build is repeating all the previous stages. This is because your docker build command is not using the already built container as a cache. Instead, you are using it as the container on which to run docker. Therefore, docker does not know that you have already completed the previous commands in the Dockerfile. I'm sure it's possible to get this right, but I think we need to think about this all more carefully.

[shimwell]

Thanks Paul, I have had a go at fixing those two points.

The docker image now uses ENV for dagmc_build_dir and and ENV for dagmc_install_dir Also the CI now has a stage that targets dagmc and another stage that targets dagmc_test I have renamed some tags of the resulting dockerfiles so that we don't overwritten previous ones.

The latest docker build CI can be seen here

[shimwell]

I just spotted an error with the tag names I had so I've corrected that and trying again

latest build action is here

[shimwell]

CI failed to find image, I suspect this is due to it not being pushed, latest action can be found here

[shimwell]

:pray: work this time. Latest action link is here

[shimwell]

Got to the stage where it fails dagmc tests on my CI fork actions

Anyone got any ideas on this output error, looks like it can't find some h5m files (iter_imprinted.h5m and bllite30matls.h5m) which I also can't find on the repo

#17 266.9 23/26 Test #23: make_watertight_regression_tests ........***Failed    0.01 sec
#17 266.9 [==========] Running 3 tests from 3 test cases.
#17 266.9 [----------] Global test environment set-up.
#17 266.9 [----------] 1 test from ITERMakeWatertightRegressionTest
#17 266.9 [ RUN      ] ITERMakeWatertightRegressionTest.ITERRegressionTest
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: iter_imprinted.h5m: No such file or directory!
#17 266.9 MOAB ERROR: serial_load_file() line 518 in Core.cpp
#17 266.9 MOAB ERROR: load_file() line 467 in Core.cpp
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:27: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 7
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:45: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 6
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:51: Failure
#17 266.9 Expected: (0) != (num_meshsets), actual: 0 vs 0
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: file set not found!
#17 266.9 MOAB ERROR: get_sealing_mesh_tags() line 1976 in Gen.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not get the mesh tags!
#17 266.9 MOAB ERROR: make_mesh_watertight() line 1803 in MakeWatertight.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not make the mesh watertight!
#17 266.9 MOAB ERROR: seal_and_check() line 355 in test_classes.cpp
#17 266.9 [  FAILED  ] ITERMakeWatertightRegressionTest.ITERRegressionTest (1 ms)
#17 266.9 [----------] 1 test from ITERMakeWatertightRegressionTest (1 ms total)
#17 266.9 
#17 266.9 [----------] 1 test from FNSFMakeWatertightRegressionTest
#17 266.9 [ RUN      ] FNSFMakeWatertightRegressionTest.FNSFRegressionTest
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: model1_360_1.h5m: No such file or directory!
#17 266.9 MOAB ERROR: serial_load_file() line 518 in Core.cpp
#17 266.9 MOAB ERROR: load_file() line 467 in Core.cpp
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:27: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 7
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:45: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 6
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:51: Failure
#17 266.9 Expected: (0) != (num_meshsets), actual: 0 vs 0
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: file set not found!
#17 266.9 MOAB ERROR: get_sealing_mesh_tags() line 1976 in Gen.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not get the mesh tags!
#17 266.9 MOAB ERROR: make_mesh_watertight() line 1803 in MakeWatertight.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not make the mesh watertight!
#17 266.9 MOAB ERROR: seal_and_check() line 355 in test_classes.cpp
#17 266.9 [  FAILED  ] FNSFMakeWatertightRegressionTest.FNSFRegressionTest (0 ms)
#17 266.9 [----------] 1 test from FNSFMakeWatertightRegressionTest (0 ms total)
#17 266.9 
#17 266.9 [----------] 1 test from BLiteMakeWatertightRegressionTest
#17 266.9 [ RUN      ] BLiteMakeWatertightRegressionTest.BLiteRegressionTest
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: bllite30matls.h5m: No such file or directory!
#17 266.9 MOAB ERROR: serial_load_file() line 518 in Core.cpp
#17 266.9 MOAB ERROR: load_file() line 467 in Core.cpp
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:27: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 7
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:45: Failure
#17 266.9 Expected equality of these values:
#17 266.9   result
#17 266.9     Which is: 6
#17 266.9   moab::MB_SUCCESS
#17 266.9     Which is: 0
#17 266.9 /root/build_dir/dagmc/src/make_watertight/tests/test_classes.cpp:51: Failure
#17 266.9 Expected: (0) != (num_meshsets), actual: 0 vs 0
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: file set not found!
#17 266.9 MOAB ERROR: get_sealing_mesh_tags() line 1976 in Gen.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not get the mesh tags!
#17 266.9 MOAB ERROR: make_mesh_watertight() line 1803 in MakeWatertight.cpp
#17 266.9 MOAB ERROR: --------------------- Error Message ------------------------------------
#17 266.9 MOAB ERROR: could not make the mesh watertight!
#17 266.9 MOAB ERROR: seal_and_check() line 355 in test_classes.cpp
#17 266.9 [  FAILED  ] BLiteMakeWatertightRegressionTest.BLiteRegressionTest (1 ms)
#17 266.9 [----------] 1 test from BLiteMakeWatertightRegressionTest (1 ms total)
#17 266.9 
#17 266.9 [----------] Global test environment tear-down
#17 266.9 [==========] 3 tests from 3 test cases ran. (2 ms total)
#17 266.9 [  PASSED  ] 0 tests.
#17 266.9 [  FAILED  ] 3 tests, listed below:
#17 266.9 [  FAILED  ] ITERMakeWatertightRegressionTest.ITERRegressionTest
#17 266.9 [  FAILED  ] FNSFMakeWatertightRegressionTest.FNSFRegressionTest
#17 266.9 [  FAILED  ] BLiteMakeWatertightRegressionTest.BLiteRegressionTest
#17 266.9 
#17 266.9  3 FAILED TESTS
#17 266.9 

[bam241]

maybe they are created by the tests.

But were the tests are running and where the tests used to be run are not the same location, so the files are not created where the tests are looking for them ?

[bam241]

or do the test have permission to write new file in those location ?

[gonuke]

I think I found it! There is a whole logic in the current system about whether it is a pull request or not (then a merge) that determines whether or not to run this regression test. It should only run on merge. Simultaneously, if it is not a pull request, there are lines to wget the missing files!!

[shimwell]

Shall we simplify that logic so that it always downloads the model and runs the test

I can see this wget command in the install.sh script

  wget ${MW_REG_TEST_MODELS_URL} -O mw_reg_test_files.tar.gz -o wget.out

I did a quick grep for the MW_REG_TEST_MODELS_URL string but I can't see where it is set.

[makeclean]

${docker-env} file?

[makeclean]

All these bash scripts should have set -u in them for safety for undefined variables and '-e' if we want to exit on errors.

[gonuke]

This PR is aiming to get rid of the scripts!

[gonuke]

Shall we simplify that logic so that it always downloads the model and runs the test

I can see this wget command in the install.sh script

  wget ${MW_REG_TEST_MODELS_URL} -O mw_reg_test_files.tar.gz -o wget.out

I did a quick grep for the MW_REG_TEST_MODELS_URL string but I can't see where it is set.

Hmmmm.... it appears to have been lost in CI transitions...

[shimwell]

Latest attempt to get the dockerfile building is running on my forks actions over here

[shimwell]

Action on my fork is passing 🎉

[shimwell]

If this one gets merged can I make a follow up one that removes the old scripts or is that a step too far