DLL Load failed while import _ITKCommonPython error

[PranjalSahu]

Description

Getting error while reading mesh using ITK.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 a = itk.meshread("C:\\Users\\pranj\\Downloads\\testpoly.vtk", itk.D)
      2 print(a)

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\extras.py:1080, in meshread(filename, pixel_type, fallback_only)
   1078     except (KeyError, itk.TemplateTypeError):
   1079         pass
-> 1080 TemplateReaderType = itk.MeshFileReader
   1081 io_filename = f"{filename}"
   1082 increase_dimension = False

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\lazy.py:137, in LazyITKModule.__getattribute__(self, attr)
    135 module = self.__belong_lazy_attributes[attr]
    136 namespace = {}
--> 137 base.itk_load_swig_module(module, namespace)
    138 self.loaded_lazy_modules.add(module)
    139 for k, v in namespace.items():

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:102, in itk_load_swig_module(name, namespace)
    100     deps = l_data.get_module_dependencies()
    101     for dep in deps:
--> 102         itk_load_swig_module(dep, namespace)
    104 if itkConfig.ImportCallback:
    105     itkConfig.ImportCallback(name, 0)

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:102, in itk_load_swig_module(name, namespace)
    100     deps = l_data.get_module_dependencies()
    101     for dep in deps:
--> 102         itk_load_swig_module(dep, namespace)
    104 if itkConfig.ImportCallback:
    105     itkConfig.ImportCallback(name, 0)

    [... skipping similar frames: itk_load_swig_module at line 102 (2 times)]

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:102, in itk_load_swig_module(name, namespace)
    100     deps = l_data.get_module_dependencies()
    101     for dep in deps:
--> 102         itk_load_swig_module(dep, namespace)
    104 if itkConfig.ImportCallback:
    105     itkConfig.ImportCallback(name, 0)

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:110, in itk_load_swig_module(name, namespace)
    107 # SWIG-generated modules have 'Python' appended. Only load the SWIG module
    108 # if we haven't already.
    109 loader = LibraryLoader()
--> 110 l_module = loader.load(swig_module_name)
    112 # OK, now the modules on which this one depends are loaded and
    113 # template_feature-instantiated, and the SWIG module for this one is also loaded.
    114 # We're going to put the things we load and create in two places: the
   (...)
    120 # stomp on an existing 'swig' namespace, nor do we want to share 'swig'
    121 # namespaces between this_module and namespace.
    123 if namespace is None:

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:259, in LibraryLoader.load(self, name)
    257     # since version 3.4: Use importlib.util.find_spec() instead.
    258     l_spec = importlib.util.find_spec(name)
--> 259     l_spec.loader.exec_module(l_module)  # pytype: disable=attribute-error
    260     return l_module
    261 finally:

File <frozen importlib._bootstrap_external>:843, in exec_module(self, module)

File <frozen importlib._bootstrap>:219, in _call_with_frames_removed(f, *args, **kwds)

File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\..\ITKCommonPython.py:13, in <module>
     11 # Import the low-level C/C++ module
     12 if __package__ or "." in __name__:
---> 13     from . import _ITKCommonPython
     14 else:
     15     import _ITKCommonPython

ImportError: DLL load failed while importing _ITKCommonPython: The specified module could not be found.

This could potentially create problems when working with Coiled clusters.

Steps to Reproduce

Expected behavior

Actual behavior

Reproducibility

100%

Versions

5.3rc4, 5.3rc3, 5.3rc2.

Environment

In a Windows 10 Home VM where the host machine is Ubuntu 20. When ITK is installed in a conda environment.

Additional Information

5.3rc1 and 5.2.1post1 work fine.

[tbirdso]

@PranjalSahu Could you please try running the small script below and report back on whether it succeeds? This will help with understanding whether the ITKCommon module is present and can be loaded at all.

import itk
itk.Image  # implicitly load ITKCommon module

[PranjalSahu]

Coiled error message

ImportError                               Traceback (most recent call last)
<ipython-input-46-b727c8b9b1bb> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', '\nl = compute(numParam)')

16 frames
/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-> 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119 

<decorator-gen-53> in time(self, line, cell, local_ns)

/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/usr/local/lib/python3.7/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
   1191         else:
   1192             st = clock2()
-> 1193             exec(code, glob, local_ns)
   1194             end = clock2()
   1195             out = None

<timed exec> in <module>()

/usr/local/lib/python3.7/dist-packages/dask/base.py in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    571         postcomputes.append(x.__dask_postcompute__())
    572 
--> 573     results = schedule(dsk, keys, **kwargs)
    574     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    575 

/usr/local/lib/python3.7/dist-packages/distributed/client.py in get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2992                     should_rejoin = False
   2993             try:
-> 2994                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2995             finally:
   2996                 for f in futures.values():

/usr/local/lib/python3.7/dist-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   2150                 direct=direct,
   2151                 local_worker=local_worker,
-> 2152                 asynchronous=asynchronous,
   2153             )
   2154 

/usr/local/lib/python3.7/dist-packages/distributed/utils.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    308         else:
    309             return sync(
--> 310                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    311             )
    312 

/usr/local/lib/python3.7/dist-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    374     if error:
    375         typ, exc, tb = error
--> 376         raise exc.with_traceback(tb)
    377     else:
    378         return result

/usr/local/lib/python3.7/dist-packages/distributed/utils.py in f()
    347                 future = asyncio.wait_for(future, callback_timeout)
    348             future = asyncio.ensure_future(future)
--> 349             result = yield future
    350         except Exception:
    351             error = sys.exc_info()

/usr/local/lib/python3.7/dist-packages/tornado/gen.py in run(self)
   1131 
   1132                     try:
-> 1133                         value = future.result()
   1134                     except Exception:
   1135                         self.had_exception = True

/usr/local/lib/python3.7/dist-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   2007                             exc = CancelledError(key)
   2008                         else:
-> 2009                             raise exception.with_traceback(traceback)
   2010                         raise exc
   2011                     if errors == "skip":

/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/utils.py in offload()

/opt/conda/envs/coiled/lib/python3.7/concurrent/futures/thread.py in run()

/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/utils.py in <lambda>()

/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/worker.py in _deserialize()

/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/protocol/pickle.py in loads()

/opt/conda/envs/coiled/lib/python3.7/site-packages/itk/itkCompositeTransformPython.py in <module>()

ImportError: PyCapsule_Import could not import module "_ITKCommonPython"

[thewtex]

A few conditions that may be problematic here:

/opt/conda/envs/coiled/lib/python3.7/site-packages

is being used with

/usr/local/lib/python3.7/dist-packages/

-- can the same environment be used?

----> 1 get_ipython().run_cell_magic('time'

Can we avoid running in ipython, at least for the test, so there are not any fancy dask/ipython issues?

[PranjalSahu]

I will check this.

[PranjalSahu]

I have got it to work on the coiled cluster by passing the dict as a function argument instead of the mesh object. So we can build the pipeline with this.

@delayed
def perform_sum1(mesh_dict):
  import itk
  mesh = itk.mesh_from_dict(mesh_dict)
  return mesh['dimension']

I will also check how to get it to work when directly passing objects.

[tbirdso]

@PranjalSahu Does this mean you were able to import ITK modules successfully? It looks like mesh_from_dict is loading ITKMesh:

def mesh_from_dict(mesh_dict: Dict) -> "itkt.Mesh":
    """Deserialize an dictionary representing an itk.Mesh object."""
    import itk

    MeshType = mesh_type_from_wasm_type(mesh_dict["meshType"]) # I assume this returns a type like itk.Mesh[itk.F,3]
    mesh = MeshType.New()  # Mesh instantiation requires ITKMesh DLL to be loaded
...

You could verify that modules are loading by finding the ITK extras.py being used by your environment and inserting itk.auto_progress(2) to get printouts from module loading.

[tbirdso]

@PranjalSahu Another script to try:

>> import itk
>> itk.auto_progress(2)
>> itk.Image[itk.D,3].New()

EDIT: above script fails on Pranjal's VM; ITKPyBase imports successfully and then ITKCommon is not found

[PranjalSahu]

Adiditional information

Edition Windows 10 Home
Version 21H2
Installed on    ‎3/‎19/‎2022
OS build    19044.1645
Experience  Windows Feature Experience Pack 120.2212.4170.0

[tbirdso]

For reference Pranjal is using a Python 3.8.12 virtual environment on his Win10 Home virtual machine.

EDIT: Note that there are two machines under discussion here, an individual Windows 10 VM and generalized Dask clusters (running a Debian Linux image) managed with the Coiled Python package. Both are showing the same error string where the ITKCommon library cannot be successfully loaded.

Coiled / Dask Cluster: AWS Debian Linux instance spawned via Coiled, Python 3.7

Local: Windows 10 Home, Python 3.8.12

[tbirdso]

During our discussion @PranjalSahu found an apparent workaround where, on one runner, first executing an operation with a mesh dictionary and then with an itk.Mesh object is successful and does not show a loading error. It looks like there is not an easy way to SSH directly into the runner for validating packages and testing functionality with printouts. @PranjalSahu to investigate reproducing the error on a separate EC2 instance launched from the same Docker image. This issue takes on lessened priority as work is no longer blocked.

[PranjalSahu]

Additional Information:

The method works when we first perform computation using itk.dict_from_mesh. Next time even directly passing mesh object as function argument works.

@delayed
def perform_sum5(img_temp):
  import itk
  import numpy as np
  # here img_temp is a dict object which we convert to mesh inside the method
  mesh = itk.mesh_from_dict(img_temp)
  return np.sum(mesh['points'])

@delayed
def perform_sum6(img_temp):
  import itk
  import numpy as np
  # here img_temp is a mesh object
  return np.sum(img_temp['points'])

Here calling perform_sum6 in a fresh was worker fails. But first calling perform_sum5 and then perform_sum6 works.

[thewtex]

It looks like there is not an easy way to SSH directly into the runner for validating packages and testing functionality with printouts.

@jrbourbeau @mrocklin any tips to SSH into a Coiled Dask work for debugging?

[thewtex]

xref https://github.com/InsightSoftwareConsortium/ITKPythonPackage/issues/194

[GenevieveBuckley]

@PranjalSahu Could you please try running the small script below and report back on whether it succeeds? This will help with understanding whether the ITKCommon module is present and can be loaded at all.

import itk
itk.Image  # implicitly load ITKCommon module

If it's helpful to know, I can run this without error in my jupyter notebook (and I think I might be running into the same problem, in a different context, discussion here https://github.com/dask/dask-blog/issues/138)

[thewtex]

ImportError: PyCapsule_Import could not import module "_ITKCommonPython"

This is addressed in: https://github.com/InsightSoftwareConsortium/ITK/pull/3494

However, we likely still need to import itk in the Dask function due to issues with pickling of builtin-functions :-(.

ImportError: DLL load failed while importing _ITKCommonPython: The specified module could not be found.

I think this (Anaconda + Windows) is likely a different issue. It may be that there are different Windows runtime DLLs that cannot be found.

I tried mamba / conda-forge environments:

• 3.7: Worked
• 3.8: Failed
• 3.9: Worked

• 3.10: Worked

  A hint from @jcfr, there was a recent fix in CMake: https://github.com/Kitware/CMake/commit/787ab7ff20ca74b57cfd3c18b0e92be1844ba244. I will try using a newer CMake for the builds.

[thewtex]

Status update: itk-5.3rc4.post2 incorporates #3494, and dask support is stable! :tada: (caveat: add import itk in delayed dask functions).

The Anaconda Windows issue is separate, we could address with conda-forge native package, perhaps via clang module optimized packages (I am investigating this).

[MStarmans91]

The issue still seems to persist under Anaconda Windows, is someone intending to fix this?

[thewtex]

Resolved in itk-5.3.0 per https://github.com/InsightSoftwareConsortium/ITKPythonPackage/issues/194