mitsuba-renderer / mitsuba2

Mitsuba 2: A Retargetable Forward and Inverse Renderer
Other
2.05k stars 267 forks source link

Cannot import Python module on Windows #223

Closed eliemichel closed 2 years ago

eliemichel commented 4 years ago

question/bug

Summary

When calling import mitsuba in Python, I get the good old "The specified module could not be found" error, yet according to dependencies all dll paths are resolved correctly.

System configuration

I've built the current master successfuly on Windows 10, using CUDA 10.2 and Python 3.8.3, default variants plus gpu_autodiff_rgb. Build in Release mode with MSVC16 (Visual Studio 2019). The mitsuba.exe binary in build/dist runs correctly. The only unexpected thing is tha INSTALL target that did not copy the content of build/dist.

Steps to reproduce

Here is a self-contained attempt to load the mitsuba module with correct PATH:

import sys
sys.path.append(r"C:\src\mitsuba2\build-msvc16\dist\python")
import os
os.environ['PATH'] += r";C:\src\mitsuba2\build-msvc16\dist"
os.environ['PATH'] += r";C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin"
import mitsuba

which gives this output:

[...] (Traceback)
ImportError: DLL load failed while importing core_ext: The specified module could not be found.
[...] (Traceback)
ImportError: The 'mitsuba' native modules could not be imported. You're likely trying to use Mitsuba within a Python binary (e:\python38\python.exe) that is different from the one for which the native module was compiled (E:\Python38\python.exe).

(so, yes, I am using the very same Python exe for building and for running)

Is the core_ext pyd module doing anything fancy with DLLs that could be the cause of such an error besides usual missing path entries? (Once again, I checked using Dependencies, with cuda bin and dist in the PATH nothing seem to miss).

merlinND commented 4 years ago

Hi @eliemichel,

Do the steps in #195 help in your case?

eliemichel commented 4 years ago

Hi @merlinND No unfortnately it didn't help because python finds the mitsuba module, it is just not able to load it (because of some dynamic linking issue -- that's what the opaque "The specified module could not be found." means on Windows).

FTR, the log of steps from #195:

C:\src\mitsuba2>setpath.bat

C:\src\src\mitsuba2>python
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mitsuba
Traceback (most recent call last):
  File "C:\src\mitsuba2\build-msvc16\dist\python\mitsuba\__init__.py", line 12, in <module>
    _import('mitsuba.core_ext')
  File "E:\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing core_ext: The specified module could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\src\mitsuba2\build-msvc16\dist\python\mitsuba\__init__.py", line 33, in <module>
    raise exc
ImportError: The 'mitsuba' native modules could not be imported.
mehrab2603 commented 4 years ago

Hi @eliemichel, Does changing this line in setpath.bat to

set PATH=%MITSUBA_DIR%\dist;%MITSUBA_DIR%\build\dist;%PATH%

help? I was having similar errors and doing that fixed it for me. Apparently, I had a matlab entry in PATH that was preventing the Mitsuba path to be detected properly for some reason. Also, make sure to use the Windows command prompt. Using Windows Powershell will not work.

eliemichel commented 4 years ago

Didn't help neither :/ And I reset my PATH to have only python, cuda and mitsuba in it, still not working. I am using cmd indeed. I really don't know how to debug this, it's frustrating.

mjben commented 4 years ago

Same problem on linux mint / anaconda3

mjben commented 4 years ago

fixed it by specifying /build/ext_build/tbb in environment variable LD_LIBRARY_PATH so that the good libtbb.so is dynamically loaded instead of an older one in my /opt/intel/.... path. It seems to be a problem in cmake configuration.

scalingsolution commented 4 years ago

Im having the same issue attemting to import the Python module on Windows:

C:\Users\nuss_mx\Desktop\Render\mitsuba2>setpath.bat

Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mitsuba
Traceback (most recent call last):
  File "C:\....\mitsuba2\dist\python\mitsuba\__init__.py", line 12, in <module>
    _import('mitsuba.core_ext')
  File "C:\....\AppData\Local\Programs\Python\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing core_ext: The specified module could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\....\mitsuba2\dist\python\mitsuba\__init__.py", line 33, in <module>
    raise exc
ImportError: The 'mitsuba' native modules could not be imported.

Setting the LD_LIBRARY_PATH variable seems to be Linux specific.

Has anyone found a solution yet?

Speierers commented 4 years ago

Is this using Anaconda? And if yes, are you running this from the Anaconda terminal?

scalingsolution commented 4 years ago

Hi @Speierers ,

thanks for the fast reply. I was trying to avoid using Anaconda, as I had some trouble with it in the past. So just python and pip install.

Speierers commented 4 years ago

Ok. Could you try to build in a build directory maybe?

scalingsolution commented 4 years ago

I tried the compilation from a build directory but I still get the same error.

During the build process I noticed that some processes failed. Could that cause the issue? I can run Mitsuba without python though it's just the bindings that don't seem to work.

cmake -G "Visual Studio 16 2019" -A x64 ..

-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.17763.
-- The C compiler identification is MSVC 19.27.29111.0
-- The CXX compiler identification is MSVC 19.27.29111.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: C:/Users/nuss_mx/AppData/Local/Programs/Python/Python38/python.exe (found version "3.8.5")
-- Found PythonLibs: C:/Users/nuss_mx/AppData/Local/Programs/Python/Python38/libs/Python38.lib
-- Building the following variants of Mitsuba:
--  * scalar_rgb
--  * scalar_spectral
-- Mitsuba: building the Python plugin.
-- Enoki: building the Python plugin.
-- pybind11 v2.4.dev4
-- Performing Test HAS_MSVC_GL_LTCG
-- Performing Test HAS_MSVC_GL_LTCG - Success
-- LTO enabled
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - not found
-- Looking for unistd.h
-- Looking for unistd.h - not found
-- Found ZLIB: C:/Users/nuss_mx/Desktop/Render/mitsuba2/build/ext_build/zlib/$<CONFIGURATION>/zlib.lib
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Performing Test HAVE_GCC_INLINE_ASM_AVX
-- Performing Test HAVE_GCC_INLINE_ASM_AVX - Failed
-- Performing Test HAVE_SYSCONF_NPROCESSORS_ONLN
-- Performing Test HAVE_SYSCONF_NPROCESSORS_ONLN - Failed
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/ml64.exe
-- Performing Test SUPPORTS_VOLATILE_FLAG
-- Performing Test SUPPORTS_VOLATILE_FLAG - Success
-- Performing Test __CxxFlag__arch_AVX
-- Performing Test __CxxFlag__arch_AVX - Success
-- Performing Test __CxxFlag__arch_AVX2
-- Performing Test __CxxFlag__arch_AVX2 - Success
-- Performing Test __CxxFlag__std_c__latest
-- Performing Test __CxxFlag__std_c__latest - Success
-- Performing Test __CxxFlag__std_c__14
-- Performing Test __CxxFlag__std_c__14 - Success
-- Mitsuba: using builtin implementation for CPU ray tracing.
-- Mitsuba: LTO support enabled.
-- Check if the system is big endian
-- Searching 16 bit integer
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Searching 16 bit integer - Using unsigned short
-- Check if the system is big endian - little endian
-- Mitsuba: sampling profiler disabled.
-- Found Sphinx: C:/Users/nuss_mx/AppData/Local/Programs/Python/Python38/Scripts/sphinx-build.exe
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/nuss_mx/Desktop/Render/mitsuba2/build
Speierers commented 4 years ago

Those failures are fine.

Could you maybe list the files in the dist directory? Maybe something is missing?

scalingsolution commented 4 years ago

That would be:

asmjit.dll* 
data/ 
Half.dll 
Iex.dll 
IlmImf.dll
IlmThread.dll
Imath.dll
 jpeg-mitsuba.dll
libpng16-mitsuba.dll
mitsuba.exe
mitsuba-core.dll
mitsuba-render.dll
plugins/
pugixml.dll
python/
tbb.dll
zlib1.dll
Speierers commented 4 years ago

And dist/python/mitsuba?

scalingsolution commented 4 years ago

So this is all fomr within the build directtory. dist/python/mitsuba:

__init__.py
core_ext.cp38-win_amd64.pyd
core_scalar_spectral_ext.cp38-win_amd64.pyd
render_ext.cp38-win_amd64.pyd
render_scalar_spectral_ext.cp38-win_amd64.pyd
config.py
core_scalar_rgb_ext.cp38-win_amd64.pyd
python/
 render_scalar_rgb_ext.cp38-win_amd64.pyd*
Speierers commented 4 years ago

After running the setpath.bat script, can you run mitsuba --help? I wonder if your problem is actually related to python, or if it is just a matter of setting up the right env variables.

scalingsolution commented 4 years ago

Yes mitsuba --help works, it's jjust the importing of the Python module that doesn't. It does find the correct directory for the Python module I assume (File "C:\....\mitsuba2\dist\python\mitsuba\__init__.py", line 12, in <module>), but for some reason the module is not found.

Speierers commented 4 years ago

And this is on Windows 10 I assume?

scalingsolution commented 4 years ago

Yes

Speierers commented 4 years ago

And which terminal are you using? Bare cmd?

scalingsolution commented 4 years ago

I tested both the Windows cmd and git bash. Both could not import.

merlinND commented 4 years ago

It does find the correct directory for the Python module I assume (File "C:....\mitsuba2\dist\python\mitsuba__init__.py", line 12, in ), but for some reason the module is not found.

Actually this is a bit surprising, you were building into a build directory, but now it finds mitsuba2/dist/python/...? I would suggest starting from a clean clone in case there are leftovers from previous attempts that are messing up the state.

Another thing to check: is everything (your compiler, CMake target, Python version) 64bit?

scalingsolution commented 4 years ago

Sorry, I just copied the old path. Running it with the build directory gives me "C:\....\mitsuba2\build\dist\python\mitsuba\__init__.py I ran it from a new clone just to check, but the same error occurs.

My setup is: Python version 3.8.5 Cmake version 3.18.1 Visual Studio Community 2019 version 16.7.1

To compile I've been running this from the build directory: cmake -G "Visual Studio 16 2019" -A x64 .. cmake --build .

merlinND commented 4 years ago

I see, thanks. And just to check (because I have no better idea), could you please make sure your Python is 64-bits?

python3 -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)'

According to this StackOverflow answer, this will allow you to find out easily.

$ python-32 -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)'
('7fffffff', False)
$ python-64 -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)'
('7fffffffffffffff', True)
scalingsolution commented 4 years ago

Thanks, I do have a 64bit version.

merlinND commented 4 years ago

You could maybe check if the DLL that fails to load depends on another DLL that's incompatible or not found? https://stackoverflow.com/questions/7378959/how-to-check-for-dll-dependency

This is also what @eliemichel had checked in the first post of this thread, would be good to verify in your case too.

scalingsolution commented 4 years ago

Yes, I've checked the dependencies as suggested, none fail. I also am able to render using the mistuba.exe.

It is just the Python bindings that cause import errors.

scalingsolution commented 4 years ago

I ended up switching to Miniconda. This solveed the Problem for me.

ALLIESXO commented 3 years ago

This is still relevant and not working. I'm having the exact same issues when following the official docs.

Speierers commented 3 years ago

@ALLIESXO could you please provide more information about your setup? (e.g. OS, python version ,...)

ALLIESXO commented 3 years ago

@Speierers Sorry for not mentioning that.

I tried the whole process using Python 3.8 on Windows 10 - Microsoft Visual Studio Community Version 16.8.3 and cmake version 3.15.4 Also I tried to do the whole process using Anaconda Python3.6.

The mitsuba.exe (System Paths were set correct) works fine but I get the same import error inside Python: Traceback (most recent call last): " File "D:...\mitsuba2\dist\python\mitsuba__init.py", line 12, in _import('mitsuba.core_ext') File "C:\ProgramData\Anaconda3\envs\mitsuba\lib\importlib\init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 658, in _load_unlocked File "", line 571, in module_from_spec File "", line 922, in create_module File "", line 219, in _call_with_frames_removed ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 1, in File "D:...\mitsuba2\dist\python\mitsuba__init__.py", line 33, in raise exc ImportError: The 'mitsuba' native modules could not be imported. "

I looked after the .pyd files which should be imported dynamically and they are in the same folder as the init.py is. Theoretically it should be able to find those modules.

Maybe the cmake files are having problems? On Ubuntu the installation process worked fine. But on Windows this problem still exists.

binorchen commented 3 years ago

I have exactly the same problem.

System: Windows 10
Python: 3.7.7
Compiler: Visual Studio 16 2019
Cmake: 3.16.9

Tested with and without Anaconda Tested both Windows Powershell and Anaconda Powershell Prompt. mitsuba or mitsuba --help works.

PS E:\mitsuba2> mitsuba
2021-01-12 16:08:21 INFO  main  [optix_api.cpp:56] Dynamic loading of the Optix library ..
Mitsuba version 2.2.1 (master[fe2a1bd7], Windows, 64bit, 16 threads, 8-wide SIMD)
Copyright 2020, Realistic Graphics Lab, EPFL
Enabled processor features: cuda avx2 avx fma f16c sse4.2 x86_64

Usage: mitsuba [options] <One or more scene XML files>

Options:

    -h, --help
        Display this help text.

    -m, --mode
        Rendering mode. Defines a combination of floating point
        and color types.

        Default: gpu_autodiff_rgb

        Available modes:
              scalar_rgb
              gpu_autodiff_rgb

    -v, --verbose
        Be more verbose. (can be specified multiple times)

    -t <count>, --threads <count>
        Render with the specified number of threads.

    -D <key>=<value>, --define <key>=<value>
        Define a constant that can referenced as "$key"
        within the scene description.

    -s <index>, --sensor <index>
        Index of the sensor to render with (following the declaration
        order in the scene file). Default value: 0.

    -u, --update
        When specified, Mitsuba will update the scene's
        XML description to the latest version.

    -a <path1>;<path2>;..
        Add one or more entries to the resource search path.

    -o <filename>, --output <filename>
        Write the output image to the file "filename".

But it failed when import mitsuba in python.

PS E:\mitsuba2> python
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mitsuba
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mitsuba'

Please help me out. Thanks a lot!

binorchen commented 3 years ago

I have exactly the same problem.

System: Windows 10
Python: 3.7.7
Compiler: Visual Studio 16 2019
Cmake: 3.16.9

Tested with and without Anaconda Tested both Windows Powershell and Anaconda Powershell Prompt. mitsuba or mitsuba --help works.

PS E:\mitsuba2> mitsuba
2021-01-12 16:08:21 INFO  main  [optix_api.cpp:56] Dynamic loading of the Optix library ..
Mitsuba version 2.2.1 (master[fe2a1bd7], Windows, 64bit, 16 threads, 8-wide SIMD)
Copyright 2020, Realistic Graphics Lab, EPFL
Enabled processor features: cuda avx2 avx fma f16c sse4.2 x86_64

Usage: mitsuba [options] <One or more scene XML files>

Options:

    -h, --help
        Display this help text.

    -m, --mode
        Rendering mode. Defines a combination of floating point
        and color types.

        Default: gpu_autodiff_rgb

        Available modes:
              scalar_rgb
              gpu_autodiff_rgb

    -v, --verbose
        Be more verbose. (can be specified multiple times)

    -t <count>, --threads <count>
        Render with the specified number of threads.

    -D <key>=<value>, --define <key>=<value>
        Define a constant that can referenced as "$key"
        within the scene description.

    -s <index>, --sensor <index>
        Index of the sensor to render with (following the declaration
        order in the scene file). Default value: 0.

    -u, --update
        When specified, Mitsuba will update the scene's
        XML description to the latest version.

    -a <path1>;<path2>;..
        Add one or more entries to the resource search path.

    -o <filename>, --output <filename>
        Write the output image to the file "filename".

But it failed when import mitsuba in python.

PS E:\mitsuba2> python
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mitsuba
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mitsuba'

Please help me out. Thanks a lot!

I have found the solution. It seems setpath.bat didn't create PYTHONPATH as expected. Just create a new System Variable named PYTHONPATH with value your_mitsuba_dir/build/dist can solve the problem.

kopetri commented 3 years ago

I have found the solution. It seems setpath.bat didn't create PYTHONPATH as expected. Just create a new System Variable named PYTHONPATH with value your_mitsuba_dir/build/dist can solve the problem.

Are you using windows powershell? If yes, setpath.bat does exactly nothing without an error. I converted setpath.bat to a corresponding powershell version, see:

# ***************************************************************
# * This script adds Mitsuba to the current path on Windows.
# * It assumes that Mitsuba is either compiled within the 
# * source tree or within a subdirectory named 'build'.
# ***************************************************************

$env:MITSUBA_DIR=Get-Location
$env:PATH=$env:PATH + ";" + $env:MITSUBA_DIR + "\dist;" + $env:MITSUBA_DIR + "\build\dist"
$env:PYTHONPATH=$env:PYTHONPATH + ";" +$env:MITSUBA_DIR + "\dist\python;" + $env:MITSUBA_DIR + "\build\dist\python"

you can check if PYTHONPATH, PATH and MITSUBA_DIR was set correctly using:

PS C:\Users\kopetri\mitsuba2> $env:PYTHONPATH
;C:\Users\kopetri\mitsuba2\dist\python;C:\Users\kopetri\mitsuba2\build\dist\python

PS C:\Users\kopetri\mitsuba2> $env:PATH
;C:\Users\kopetri\mitsuba2\dist;C:\Users\kopetri\mitsuba2\build\dist

PS C:\Users\kopetri\mitsuba2> $env:MITSUBA_DIR
;C:\Users\kopetri\mitsuba2

Starting over with fresh compilation (by deleting mitsuba/build folder and cmake configure->generate and then vs2019->build project) and then using setpath.ps1 solves the problem for me.

sndels commented 3 years ago

I also ran into the original issue of import mitsuba throwing ImportError: DLL load failed while importing core_ext: The specified module could not be found on my W10 box with standard Python.

It seems something goes sideways with Python 3.8+ as the import is quiet when Mitsuba is compiled against 3.7 but throws the error when compiled against 3.8 or 3.9. Mitsuba itself dumps the help print without issues on all builds. Bumping pybind to the most recent release did not help. I can poke around if you have some suggestions of things to check/compare.

Here are the steps I used to test each interpreter with the stuff from setpath.bat set to my global env.

Theo-Wu commented 3 years ago

Hi @eliemichel, Does changing this line in setpath.bat to

set PATH=%MITSUBA_DIR%\dist;%MITSUBA_DIR%\build\dist;%PATH%

help? I was having similar errors and doing that fixed it for me. Apparently, I had a matlab entry in PATH that was preventing the Mitsuba path to be detected properly for some reason. Also, make sure to use the Windows command prompt. Using Windows Powershell will not work.

It works for me surprisingly!

bananeeg commented 3 years ago

I had the same problem as some of those here:

Microsoft Windows [Version 10.0.19043.1083]
(c) Microsoft Corporation. All rights reserved.

C:\Users\evan.giavina\mitsuba2>setpath.bat

C:\Users\evan.giavina\mitsuba2>python

Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mitsuba
Traceback (most recent call last):
  File "C:\Users\evan.giavina\mitsuba2\dist\python\mitsuba\__init__.py", line 12, in <module>
    _import('mitsuba.core_ext')
  File "C:\Users\evan.giavina\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1173, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: DLL load failed while importing core_ext: The specified module could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\evan.giavina\mitsuba2\dist\python\mitsuba\__init__.py", line 33, in <module>
    raise exc
ImportError: The 'mitsuba' native modules could not be imported.

Since I found a solution that worked for me (or rather @Speierers, who helped me, found it) here it is: Using gitbash, I used the following commands:

git clone --recursive https://github.com/mitsuba-renderer/mitsuba2.git
cd mitsuba2
mkdir build
cd build
cmake ..
cmake --build . -j 8
cd ..
source setpath.sh
python -c "import mitsuba"

Using the versions Python 3.7.8 (check with 'python --version') CMake 3.17.1 (check with 'CMake --version') MSVC 19.29.30038.1 (check with 'cl.exe')

After a few tests, I found that cmd or gitbash both work, that I could use 'cmake --build . -j 8' or build it from within visual studio GUI, and it worked all the same. In the end, the only thing I did differently was uninstalling python and CMake and re installing python 3.7.8 and CMake 3.17.1. One or the other was probably the cause of the problem.

sxsong1207 commented 2 years ago

I met the same issue as you guys. The module mitsuba.core_ext cannot be imported.

I am running on Windows, with Anaconda env.

My solution: I have manually installed embree via conda install embree and git cloned pyembree & setup.

After removed embree and pyembree from my anaconda environment, the problem was solved.

My guess is the built in embree conflict with embree.dll installed from anaconda.

Li-Jiren commented 2 years ago

The same problem in Win 10

Python 3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mitsuba
Traceback (most recent call last):
  File "C:\...\mitsuba2\dist\python\mitsuba\__init__.py", line 12, in <module>
    _import('mitsuba.core_ext')
  File "C:\...\miniconda3\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\...\mitsuba2\dist\python\mitsuba\__init__.py", line 33, in <module>
    raise exc
ImportError: The 'mitsuba' native modules could not be imported.

I've tried using cmd and miniconda, python 3.9 and python 3.7, building in VS 2022 and with cmake. All attempts got the same error.

Li-Jiren commented 2 years ago

I also tried to find out all dependencies of core_ext.cp39-win_amd64.pyd with dumpbin and move all the dll files to ...\mitsuba2\build\dist directory, and sys.path actually contains ...\mitsuba2\build\dist , but still got the same error.

C:\...\mitsuba2\build\dist\python\mitsuba>dumpbin /dependents core_ext.cp39-win_amd64.pyd
Microsoft (R) COFF/PE Dumper Version 14.30.30709.0
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file core_ext.cp39-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    python39.dll
    mitsuba-core.dll
    asmjit-mitsuba.dll
    tbb.dll
    MSVCP140D.dll
    VCRUNTIME140D.dll
    VCRUNTIME140_1D.dll
    ucrtbased.dll
    KERNEL32.dll

  Summary

        8000 .data
       1D000 .pdata
      4E9000 .rdata
        4000 .reloc
        1000 .rsrc
      216000 .text
Speierers commented 2 years ago

Have you tried to build (clean) and import Mitsuba from a miniconda terminal? This helped me in the past IIRC.

Li-Jiren commented 2 years ago

It works. Thank you! I tried miniconda before while used setpath.bat after compiling, and it didn't work. After seeing your reply, I tried again and used setpath.ps1 and it works.

eliemichel commented 2 years ago

I finally figured what the issue was with my original report: As stated in Python 3.8 update notes, the PATH variable is no longer used on Windows to resolve dynamic linking of modules (for security reasons, see notes).

The solution is to call os.add_dll_directory() for each path that contains dependencies:

import sys
sys.path.append(r"C:\src\mitsuba2\build-msvc16\dist\python")
import os
os.add_dll_directory(r"C:\src\mitsuba2\build-msvc16\dist")
os.add_dll_directory(r"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin")
import mitsuba
merlinND commented 2 years ago

That's good to know, thank you for reporting @eliemichel !