Import Error of PyDeepCL

[mohmsslk]

Hello, when I import PyDeepCL I get the following error : On Ubuntu(virtualbox): ImportError: .../anaconda2/lib/python2.7/site-packages/PyDeepCL.so: undefined symbol: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev On Windows : ImportError: DLL load failed: The specified module could not be found. Do you know why? Thank you.

[viper7882]

Hi,

I've encountered similar error in Windows 10, 64-bit: Traceback (most recent call last): File "", line 1, in ImportError: DLL load failed: %1 is not a valid Win32 application.

I've figured out that the issue is related to PyDeepCL.py unable to load main module of PyDeepCL.pyd file. Tracing back the source of PyDeepCL.pyd file is an auto-generated PyDeepCL.cpp file using Cython. Even I've moved to latest Cython version 0.25.2, the issue still persist. I suspect there is no main module created in PyDeepCL.pyd file. When I manually hack PyDeepCL.py to load EasyCL.dll (manually copied over into egg folder), the DLL load failure error is gone.

Likely this is an issue with PyDeepCL.pyx and/or its dependencies *.pyx files.

[viper7882]

Hi Hugh Perkins,

I believe I've found out what the real issue is. I'm using Visual Studio 2015 (v14) but the Cython was compiled by setuptools was using VC9.0 for Python. I've googled and found the following comment from one of the Google Groups, https://groups.google.com/forum/#!topic/cython-users/NYjxdX2UA8I: "Setuptools is running Visual Studio 9.0, but you have for some reason inserted VS 14.0's include dir into the list. Trying to change Setuptools to use VS 14.0 is a nightmare that probably won't end, even when you rebuild Python itself and all of the packages you're using with VS 14.0 also.

VS 9.0 is required to build Python extensions for Python 2.7 to make sure it uses the same version of MSVCRT (to Linux folks, it is basically libc) for the extensions as Python itself uses. Mismatched versions of MSVCRT can lead to problems caused by each version of MSVCRT using its own malloc heap, stdio buffers, locale settings, et cetera. Usually you're stopped right away with a DLL load error."

When I attempted to use VC9.0 for Python, ran Microsoft\Visual C++ for Python\9.0\VC\bin\vcvarsx86_amd64.bat and then cmake-gui, cmake failed with devenv.com error.

With this import error present in Python, I wonder how you could run it previously? If you know how to address this issue, kindly share your solution here as the instruction to build Python Wrapper probably need updating due to lack of clarity.

Thank you in advance.

[hughperkins]

Ok, so... here's what I remember, with a quick revision from https://docs.python.org/devguide/setup.html#windows-compiling :

• python 2.7 is built with visual studio 2008. You can get visual studio 2008 from https://www.microsoft.com/en-us/download/details.aspx?id=44266 Whether that's what I used, I'm thinking through as I write
• python 3.5 is built using visual studio 2015. which is why i chose msvc2015. And per the link above.
• if I remember rightly, what I do is:
  • deepcl itself is built as a separate dll, using msvc2015. Its a c++ project, that builds using msvc2015, to a dll
  • the api potentially separates one msvc runtime from another
  • specifically, I put a fair amount of effort into making sure objects on one side of the c-interface are never freed on the other side, nor are they introspected/used on the other side: just treated as a black box
• then, the python bits are compiled with the relevant compiler:
  • python 3.5 cython bits are built using msvc 2015
  • python 2.7 cython bits are built using msvc 2008
• to figure out whether everything I've written above is true or not, or a bit, or not at all, the reference documentation really is: the jenkins build scripts, ie, https://github.com/hughperkins/DeepCL/tree/master/jenkins
  • if you're building python 2.7 on 64-bit windows, then the approprirate script is: https://github.com/hughperkins/DeepCL/blob/master/jenkins/win64-py27.bat

Lets work through waht it does, bit by bit:

https://github.com/hughperkins/DeepCL/blob/master/jenkins/win64-py27.bat

call %~dp0win-py.bat 64 27

Ok. First parameter is bitness, 64 bit or 32bit. Second parameter is python version, 2.7 or 3.5. Let's look at win-py.bat: https://github.com/hughperkins/DeepCL/blob/master/jenkins/win-py.bat

set bitness=%1
set pyversion=%2

ok, so now bitness will be 64, andpyversion` will be 27. Continuing:

call %~dp0win-cpp.bat %bitness%

This will build the c++ deepcl bit, using msvc 2015. This is the non-python bit. This is basically the whole of c++ deepcl, so this single line will take a while. Then, we have:

call \py%pyversion%-%bitness%\scripts\activate

This is going to activate the relevant python virtual environmetn. In this case it will be a 64-bit python 2.7 virtuale nevironemtn.

Then we have:

cd python
python setup.py build_ext -i

So this is going to build the python wrappers, using cython. It will use whatever compiler was used to compile the underlying virtual environemtn, which in this case is msvc 2008, and it will use that.

So, we have three 'bricks':

• deepcl c++ brick: msvc 2015
• python 3.5 wrappers: msvc 2015
• python 2.7 wrappers: msvc 2008

To what extent is this somewhat consistent with your experiences above?

[viper7882]

Hi Hugh,

I'm using Python 2.7 with both MSVC 2015 and MVCS 9.0 for Python. Similar to your setup, I've compiled both DeepCL and PyDeepCL using MSVC 2015 and generated those required DLL and PyDeepCL.pyd.

Good news is that I've successfully import PyDeepCL in Python 2.7 after running https://github.com/hughperkins/DeepCL/blob/master/jenkins/win64-py27.bat. However, I've encountered the DLL load issue again when I repeat the same import command in another command prompt. Probably there is something required in win64-py27.bat that must be executed. I'll take a closer look.

Thank you so much Hugh.

[hughperkins]

Good news is that I've successfully import PyDeepCL in Python 2.7 after running https://github.com/hughperkins/DeepCL/blob/master/jenkins/win64-py27.bat.

Cool :-)

However, I've encountered the DLL load issue again when I repeat the same import command in another command prompt. Probably there is something required in win64-py27.bat that must be executed. I'll take a closer look.

Is it the activate line? https://github.com/hughperkins/DeepCL/blob/master/jenkins/win-py.bat#L31

call dist\bin\activate.bat

[viper7882]

It is not the activate line but the environment vars. Error from my end. Many thanks still for saving my research.

[hughperkins]

Cool :)