ImportError: libimf.so on IDAaaS

[samtygier-stfc]

Summary

Crash on starting Mantid Imaging on IDAaaS

Steps To Reproduce

Launch Mantid Imaging Unstable on IDAaaS

Expected Behaviour

Current Behaviour

Crash, see below

Context

Mantid Imaging 2.5.0a4.post8

Failure Logs

Starting mantidimaging unstable
Traceback (most recent call last):
  File "/opt/mambaforge/envs/mantidimaging_unstable/bin/mantidimaging", line 6, in <module>
    from mantidimaging.main import main
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/main.py", line 10, in <module>
    import mantidimaging.core.parallel.manager as pm
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/__init__.py", line 5, in <module>
    from . import operations, io, parallel, utility  # noqa: F401
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/io/__init__.py", line 5, in <module>
    from . import (  # noqa: F401
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/io/saver.py", line 17, in <module>
    from ..operations.rescale import RescaleFilter
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/operations/rescale/__init__.py", line 5, in <module>
    from .rescale import RescaleFilter  # noqa:F401
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/operations/rescale/rescale.py", line 11, in <module>
    from mantidimaging.gui.utility.qt_helpers import Type
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/gui/__init__.py", line 5, in <module>
    from .gui import execute, setup_application  # noqa: F401  # noqa:F821
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/gui/gui.py", line 13, in <module>
    from mantidimaging.gui.windows.main import MainWindowView
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/gui/windows/main/__init__.py", line 6, in <module>
    from .view import MainWindowView  # noqa: F401
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/gui/windows/main/view.py", line 22, in <module>
    from mantidimaging.core.utility.command_line_arguments import CommandLineArguments
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/utility/command_line_arguments.py", line 11, in <module>
    filter_names = [package.filter_name for package in load_filter_packages()]
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/operations/loader.py", line 49, in load_filter_packages
    module = MODULES_OPERATIONS[name].load_module(name)
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/operations/circular_mask/__init__.py", line 5, in <module>
    from .circular_mask import CircularMaskFilter  # noqa:F401
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/mantidimaging/core/operations/circular_mask/circular_mask.py", line 8, in <module>
    import tomopy
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/tomopy/__init__.py", line 68, in <module>
    from tomopy.misc.corr import *
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/tomopy/misc/corr.py", line 57, in <module>
    import tomopy.util.mproc as mproc
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/tomopy/util/mproc.py", line 62, in <module>
    import numexpr as ne
  File "/opt/mambaforge/envs/mantidimaging_unstable/lib/python3.9/site-packages/numexpr/__init__.py", line 24, in <module>
    from numexpr.interpreter import MAX_THREADS, use_vml, __BLOCK_SIZE1__
ImportError: libimf.so: cannot open shared object file: No such file or directory
OpenMPI module has been loaded. To unload type the following into the terminal: module unload mpi/openmpi-x86_64

Screenshot(s)

[samtygier-stfc]

I can reproduce locally if I install with mamba env create -f https://raw.githubusercontent.com/mantidproject/mantidimaging/main/environment.yml. And run python -m mantidimaging from that environment. I can fix by switching numexpr from 2.8.1p y39ha90e767_1 intel to 2.8.3 mkl_py39h6189487_1 conda-forge/linux-64

So a quick work around is mamba activate mantidimaging-nightly && mamba install numexpr=2.8.3

[samtygier-stfc]

python setup.py create_dev_env installs the 2.8.3 version from conda-forge. This uses the mamba env create underneath, the difference is that it explicitly puts numexpr=2.8.* in the environment file rather than getting it from the mantidimaging package dependencies.

[samtygier-stfc]

Options:

1. Specify channel for package. Conda/mamba allow using conda-forge::numexpr as a specifier. For us numexpr is specified in meta.yaml, which accepts this, but it is not propagated into the package metadata, so at the user install step they would still get the bad package.
   2. Specify channel for package in environment. This should work, but means that the dependency needs to be written in 2 places.
   3. Use channel_priority: strict conda setting. There does not seem to be a way for use to put this in the env file or on the command line. This would work, but means asking users to set a non-default configuration, so endless troubleshooting.
   4. Set numexpr version to 2.8.3, should work for now, but would break if intel update their package.

[samtygier-stfc]

IDAaaS has updated and is working now.