Can't run any example

[RemDelaporteMathurin]

Steps to reproduce:

1. Pull the latest docker image docker pull ghcr.io/fusion-energy/fusion-neutronics-workflow
2. Run a new container docker run -it -v $PWD:/local_dir ghcr.io/fusion-energy/fusion-neutronics-workflow
3. Navigate to folder cd example_01_single_volume_cell_tally
4. Run step 1 python3 1_create_cad_and_convert_to_dagmc.py Produces:

   Saved file as  steel.stl

5. Run neutronics step python3 2_run_neutronics_simulation.py Produces:

   Using the DOUBLE-DOWN interface to Embree.
   Loading file dagmc.h5m
   Initializing the GeomQueryTool...
   Using faceting tolerance: 1.81323e-317
   Building acceleration data structures...
   Illegal instruction

Same is true for all the examples. I simply cannot run anything 😢

I've investigated it a bit and it seems to be related to the call

odw.Geometry(h5m_filename='dagmc.h5m')

[shimwell]

I think I might need to change the way a compile is done as I think this is the same issue https://openmc.discourse.group/t/dagmc-geometry-open-mc-aborted-unexpectedly/1369/25?u=pshriwise

[shimwell]

I'm now not sure what is happening. I've rebuilt the image with the new embree compile for compatibility but getting the same error when trying to attach artifacts to the release. Wondering if you have seen this before @pshriwise

Failing with strange faceting tolerance despite new Embree compile https://github.com/fusion-energy/fusion_neutronics_workflow/runs/4465224057?check_suite_focus=true

[RemDelaporteMathurin]

I wonder if it's an issue with openmc-dagmc-wrapper rather than fusion-neutronics-workflow

[RemDelaporteMathurin]

So I tried reproducing the issue with odw.

1. Create a container from openmc-dagmc-wrapper:dependencies image
2. Install dependencies
3. Create dagmc.h5m file from example-01 (in fusion-neutronics-workflow)
4. Copy-paste the file in the container created in 1.
5. Run the code below

import openmc
import openmc_dagmc_wrapper as odw
import openmc_plasma_source as ops

my_h5m_filename = "dagmc.h5m"

material_tag_to_material_dict = {
    "mat1": "copper"
}

materials = odw.Materials(
    h5m_filename=my_h5m_filename,
    correspondence_dict=material_tag_to_material_dict,
)

tally = odw.CellTally(
    tally_type="heating",
    target="mat1",
    materials=materials
)

tallies = openmc.Tallies([tally])

geometry = odw.Geometry(h5m_filename=my_h5m_filename)

settings = odw.FusionSettings()
settings.batches = 1
settings.particles = 100
# assigns a ring source of DT energy neutrons to the source using the
# openmc_plasma_source package
settings.source = ops.FusionPointSource(fuel="DT", coordinate=(1000, 1000, 1000))

my_model = openmc.Model(
    materials=materials, geometry=geometry, settings=settings, tallies=tallies
)
my_model.run()

This produces the following error:

                                %%%%%%%%%%%%%%%
                           %%%%%%%%%%%%%%%%%%%%%%%%
                        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
                      %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
                    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
                   %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
                                    %%%%%%%%%%%%%%%%%%%%%%%%
                                     %%%%%%%%%%%%%%%%%%%%%%%%
                 ###############      %%%%%%%%%%%%%%%%%%%%%%%%
                ##################     %%%%%%%%%%%%%%%%%%%%%%%
                ###################     %%%%%%%%%%%%%%%%%%%%%%%
                ####################     %%%%%%%%%%%%%%%%%%%%%%
                #####################     %%%%%%%%%%%%%%%%%%%%%
                ######################     %%%%%%%%%%%%%%%%%%%%
                #######################     %%%%%%%%%%%%%%%%%%
                 #######################     %%%%%%%%%%%%%%%%%
                 ######################     %%%%%%%%%%%%%%%%%
                  ####################     %%%%%%%%%%%%%%%%%
                    #################     %%%%%%%%%%%%%%%%%
                     ###############     %%%%%%%%%%%%%%%%
                       ############     %%%%%%%%%%%%%%%
                          ########     %%%%%%%%%%%%%%
                                      %%%%%%%%%%%

                 | The OpenMC Monte Carlo Code
       Copyright | 2011-2021 MIT, UChicago Argonne LLC, and contributors
         License | https://docs.openmc.org/en/latest/license.html
         Version | 0.13.0-dev
        Git SHA1 | 0157dc219ff8dca814859b3140c6cef1e78cdee1
       Date/Time | 2021-12-09 10:49:29
   MPI Processes | 1
  OpenMP Threads | 12

 Reading settings XML file...
 Reading cross sections XML file...
 Reading materials XML file...
 Reading geometry XML file...
Using the DOUBLE-DOWN interface to Embree.
Loading file dagmc.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.01
Building acceleration data structures...
Traceback (most recent call last):
  File "test.py", line 50, in <module>
    my_model.run()
  File "/opt/openmc/openmc/model/model.py", line 547, in run
    openmc.run(particles, threads, geometry_debug, restart_file,
  File "/opt/openmc/openmc/executor.py", line 276, in run
    _run(args, output, cwd)
  File "/opt/openmc/openmc/executor.py", line 116, in _run
    raise RuntimeError(error_msg)
RuntimeError: OpenMC aborted unexpectedly.

Which is yet another error message.

Could the problem come from the cad_to_h5m package @shimwell? These are the files I used files.zip

[shimwell]

I've just run your example locally (without docker) and it works. I shall keep investigating

[shimwell]

I've just run the example with the ghcr.io/fusion-energy/fusion-neutronics-workflow:latest docker image with a volume mount and it works as well.

So I guess that implies problems with the docker file and the CPU. Do you have an AMD CPU

[RemDelaporteMathurin]

Intel(R) Xeon(R) E-2186G CPU @ 3.80GHz, 3792 MHz

[shimwell]

I can rebuild the docker file and see if changing the Embree compile flags back helps

[shimwell]

We could also try rebuilding without embree and double down as a backup option

[shimwell]

I've made a couple of new docker images with and without AVX enabled

Would you be able to try these two and let me know if either work for you

https://github.com/fusion-energy/fusion_neutronics_workflow/pkgs/container/fusion-neutronics-workflow-no-avx

https://github.com/fusion-energy/fusion_neutronics_workflow/pkgs/container/fusion-neutronics-workflow-avx

[shimwell]

One more docker image, I think this one has no embree usage in dagmc https://github.com/fusion-energy/fusion_neutronics_workflow/pkgs/container/fusion-neutronics-workflow-no-dd

[pshriwise]

Wondering if you have seen this before @pshriwise

Can't say I've seen an error like this, no. To isolate the problem can we maybe try some of the DAGMC based tools to make sure the problem isn't in OpenMC? (build_obb, overlap_check, etc.)

[RemDelaporteMathurin]

One more docker image, I think this one has no embree usage in dagmc https://github.com/fusion-energy/fusion_neutronics_workflow/pkgs/container/fusion-neutronics-workflow-no-dd

This image fixed the bug for examples 1, 2, 4. Example 5 is not working but it's not related to this.

I guess this means there's an issue with embree?

Thanks for taking the time to investigate this @shimwell

[shimwell]

Super, thanks for testing, all the feedback is super useful. I shall write up some details on the docker images and tidy things up a bit. I think the issue can be narrowed down in the future and avoided for now by providing a range of docker images for different CPUs