search

JuliaPy / PyCall.jl

Package to call Python functions from the Julia language
MIT License
1.47k stars 189 forks source link

Trying to get PyCall to function with Abaqus python #463

Open glwc opened 6 years ago

glwc commented 6 years ago

Dear all,

I’m taking a stab to try to get the python distributed with SIMULIA Abaqus version “2017” (also known as v6.14-5, which reports that it comes with python 2.7.3) to work from Julia 0.6.2 with PyCall. I'm on Windows 10.

I know PyCall only recommends calling Anaconda python (which works great), but if the Abaqus scripting interface could be directly exposed to Julia this would be extremely useful for job control, post processing, etc. I hope this is the best place to post this query.

I can get PyCall setup for Abaqus python and import my own .py modules, but run into trouble trying to successfully import Abaqus .pyd modules (although these files are being found). In other words, I can get python’s Inp.module_find() to find and identify a .pyd module correctly, but Inp.module_load() runs into trouble, for example reporting

PyError (ccall(@pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, arg, C_NULL)) <type 'exceptions.ImportError'>
ImportError('DLL load failed: The specified procedure could not be found.',)
  File "C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib\odb2vtk.py", line 15, in <module>
    from odbAccess import *

when my own odb2vtk.py module does from odbAccess import *. The steps I've taken are explained in full below.

I’m wondering if there are other Abaqus locations that need to be added to the pyimport path. odbAccess.pyd is a very small file (20 kB) which is exposing a large part of the Abaqus scripting interface.
Any hints or suggestion highly appreciated, and apologies for the length of this post.

GC

The following is how I proceeded. I did this from Jupyter 5.4.0 running Julia 0.6.2

ENV["PYTHON"] = "C:\\SIMULIA\\CAE\\2017\\win_b64\\tools\\SMApy\\python2.7\\python.exe" Pkg.build("PyCall")

Then I restarted the Julia kernel.
Checking things out from the a Jupyter Julia cell looks good: 

PyCall.pyprogramname
"C:\\SIMULIA\\CAE\\2017\\win_b64\\tools\\SMApy\\python2.7\\python.exe"

PyCall.libpython
"C:\\SIMULIA\\CAE\\2017\\win_b64\\tools\\SMApy\\python2.7\\python27"

pyversion
v"2.7.3"

Then I added some locations to the pyimport path as per the PyCall docs – the last one added has the Abaqus .pyd file I want to access. 

unshift!(PyVector(pyimport("sys")["path"]), "")
pypath = unshift!(PyVector(pyimport("sys")["path"]), "C:\\SIMULIA\\CAE\\2017\\win_b64\\tools\\SMApy\\python2.7\\Lib");
pypath = unshift!(PyVector(pyimport("sys")["path"]), "C:\\SIMULIA\\CAE\\2017\\win_b64\\code\\bin");
[println("$(pn)") for pn in pypath];
C:\SIMULIA\CAE\2017\win_b64\code\bin
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib

C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\python27.zip
C:\Program Files\FreeCAD 0.16\bin
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\DLLs
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\plat-win
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\lib-tk
C:\Users\graham\AppData\Local\Julia-0.6.2\bin
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\win32
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\win32\lib
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\Pythonwin

While @pyimport fails for my module odb2vtk.py, the following code suggestion from https://github.com/dhoegh/Hawk.jl/blob/master/src/Hawk.jl#L7-13 works, at least up to the point of importing Abaqus module odbAccess.pyd: 

@pyimport imp
filename = "C:\\SIMULIA\\CAE\\2017\\win_b64\\tools\\SMApy\\python2.7\\Lib\\odb2vtk.py"
(path, name) = dirname(filename), basename(filename)
# (name, ext) = rsplit(name, '.', 2)                         # Deprecated way to do this - rsplit no longer exists
(ext, name) = reverse.(split(reverse(name),'.'; limit=2))    # Keep only first two results (in case of more .'s) 
# Using python module search 
(file, pathname, description) = imp.find_module(name, [path])
println("name: $(name)")
println("file: $(file)")
println("pathname: $(pathname)")
println("description: $(description)")
odb2vtk = imp.load_module(name, file, pathname, description)

Outputing

name: odb2vtk
file: PyObject <open file 'C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib\odb2vtk.py', mode 'U' at 0x0000000025843420>
pathname: C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib\odb2vtk.py
description: (".py", "U", 1)
PyError (ccall(@pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, arg, C_NULL)) <type 'exceptions.ImportError'>
ImportError('DLL load failed: The specified procedure could not be found.',)
  File "C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib\odb2vtk.py", line 15, in <module>
    from odbAccess import *

Stacktrace:
 [1] pyerr_check at C:\Users\graham\.julia\v0.6\PyCall\src\exception.jl:56 [inlined]
 [2] pyerr_check at C:\Users\graham\.julia\v0.6\PyCall\src\exception.jl:61 [inlined]
 [3] macro expansion at C:\Users\graham\.julia\v0.6\PyCall\src\exception.jl:81 [inlined]
 [4] #_pycall#67(::Array{Any,1}, ::Function, ::PyCall.PyObject, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:653
 [5] _pycall(::PyCall.PyObject, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:641
 [6] #pycall#71(::Array{Any,1}, ::Function, ::PyCall.PyObject, ::Type{PyCall.PyAny}, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:675
 [7] pycall(::PyCall.PyObject, ::Type{PyCall.PyAny}, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:675
 [8] #call#72(::Array{Any,1}, ::PyCall.PyObject, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:678
 [9] (::PyCall.PyObject)(::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:678
 [10] include_string(::String, ::String) at .\loading.jl:522

Then if I try to import odbAccess.pyd myself directly in a Jupyter Julia cell using the Inp interface and PyCall, I get the same error:

filename = "C:\\SIMULIA\\CAE\\2017\\win_b64\\code\\bin\\odbAccess.pyd"

@pyimport imp
(path, name) = dirname(filename), basename(filename)
# (name, ext) = rsplit(name, '.', 2)                         # Deprecated way to do this - rsplit no longer exists
(ext, name) = reverse.(split(reverse(name),'.'; limit=2))    # Keep only first two results (in case of more .'s) 
# Using python module search 
println("Using python module search to look for $(name) in $(path)")
(file, pathname, description) = imp.find_module(name, [path])
println("name: $(name)")
println("file: $(file)")
println("pathname: $(pathname)")
println("description: $(description)")

odb = imp.load_module(name, file, pathname, description)

Giving

Using python module search to look for odbAccess in C:\SIMULIA\CAE\2017\win_b64\code\bin
name: odbAccess
file: PyObject <open file 'C:\SIMULIA\CAE\2017\win_b64\code\bin\odbAccess.pyd', mode 'rb' at 0x0000000025843300>
pathname: C:\SIMULIA\CAE\2017\win_b64\code\bin\odbAccess.pyd
description: (".pyd", "rb", 3)
PyError (ccall(@pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, arg, C_NULL)) <type 'exceptions.ImportError'>
ImportError('DLL load failed: The specified procedure could not be found.',)

Stacktrace:
 [1] pyerr_check at C:\Users\graham\.julia\v0.6\PyCall\src\exception.jl:56 [inlined]
 [2] pyerr_check at C:\Users\graham\.julia\v0.6\PyCall\src\exception.jl:61 [inlined]
 [3] macro expansion at C:\Users\graham\.julia\v0.6\PyCall\src\exception.jl:81 [inlined]
 [4] #_pycall#67(::Array{Any,1}, ::Function, ::PyCall.PyObject, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:653
 [5] _pycall(::PyCall.PyObject, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:641
 [6] #pycall#71(::Array{Any,1}, ::Function, ::PyCall.PyObject, ::Type{PyCall.PyAny}, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:675
 [7] pycall(::PyCall.PyObject, ::Type{PyCall.PyAny}, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:675
 [8] #call#72(::Array{Any,1}, ::PyCall.PyObject, ::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:678
 [9] (::PyCall.PyObject)(::RevString{SubString{String}}, ::Vararg{Any,N} where N) at C:\Users\graham\.julia\v0.6\PyCall\src\PyCall.jl:678
 [10] include_string(::String, ::String) at .\loading.jl:522
stevengj commented 6 years ago

I know PyCall only recommends calling Anaconda python (which works great)

This is not true, it also works with other Python installations. The only Python I haven't been able to get to work is Canopy.

Have you looked at C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib\odb2vtk.py", line 15 from the error message? Clearly it is trying to open a DLL that is in some path that is not being found. (Normally, Windows looks in the path of the executable for DLLs, so if it is expecting the DLL to be in the path of python.exe then it is not going to work when you are running julia.exe.) Possibly you just need to update your PATH environment variable to help it find the DLL.

glwc commented 6 years ago

Thanks @stevengj, that is very encouraging about getting PyCall going with different python distributions.

To answer the question: Line 15 of C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib\odb2vtk.py is the first line of code in that file:

from odbAccess import *

The odbAccess module is sitting in C:\SIMULIA\CAE\2017\win_b64\code\bin on my machine. I've taken your suggestion and updated my PATH environment variable, adding all the places I could see that Abaqus python stores .pyd dynamically loadable shared libraries (see below for my updated PATH ). Unfortunately this did not help. As I wrote above,Inp.module_find() is correctly finding the module odbAccess.pyd. Inp.module_load(name, file, pathname, description) is also finding odbAccess.pyd correctly because if I rename that file I get a different error:

PyError (ccall(@pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, arg, C_NULL)) <type 'exceptions.ImportError'>
ImportError('No module named odbAccess',)

It seems like the error is to do with the module name I'm requesting (which is odbAccess)- hence the specific error DLL load failed: The specified procedure could not be found.

GC

My updated PATH environment [println("$(pn)") for pn in sort(split(ENV["PATH"],';'))]

C:\Anaconda3
C:\Anaconda3
C:\Anaconda3\Library\bin
C:\Anaconda3\Library\bin
C:\Anaconda3\Library\mingw-w64\bin
C:\Anaconda3\Library\usr\bin
C:\Anaconda3\Scripts
C:\Anaconda3\Scripts
C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL
C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT
C:\Program Files (x86)\Intel\iCLS Client\
C:\Program Files (x86)\Skype\Phone\
C:\Program Files\Common Files\Intel\WirelessCommon\
C:\Program Files\Common Files\Intel\WirelessCommon\
C:\Program Files\Intel\Intel(R) Management Engine Components\DAL
C:\Program Files\Intel\Intel(R) Management Engine Components\IPT
C:\Program Files\Intel\WiFi\bin\
C:\Program Files\Intel\WiFi\bin\
C:\Program Files\Intel\iCLS Client\
C:\Program Files\Microsoft MPI\Bin\
C:\Program Files\nodejs\
C:\SIMULIA\Abaqus\Commands
C:\SIMULIA\CAE\2017\win_b64\code\bin
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\DLLs
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\Lib
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\lib-tk
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\plat-win
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\Pythonwin
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\pywin32_system32
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\win32
C:\SIMULIA\CAE\2017\win_b64\tools\SMApy\python2.7\lib\site-packages\win32\lib
C:\SIMULIA\Commands
C:\Users\graham\AppData\Local\Julia-0.6.2\bin
C:\Users\graham\AppData\Local\Microsoft\WindowsApps
C:\Users\graham\AppData\Local\Microsoft\WindowsApps
C:\Users\graham\AppData\Local\atom\bin
C:\Users\graham\AppData\Roaming\npm
C:\WINDOWS
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\WINDOWS\system32
C:\WINDOWS\system32
C:\texlive\2016\bin\win32
stevengj commented 6 years ago

What DLL is it trying to open, where is that DLL on your filesystem, and what symbol is it looking up in that DLL?

glwc commented 6 years ago

odbAccess.pyd, C:\SIMULIA\CAE\2017\win_b64\code\bin, and odbAccess

stevengj commented 6 years ago

Does cglobal((:odbAccess,raw"C:\SIMULIA\CAE\2017\win_b64\code\bin\odbAccess.pyd")) work in Julia, i.e. is it able to find that symbol in that DLL?

glwc commented 6 years ago

That's negative: cglobal((:odbAccess,raw"C:\SIMULIA\CAE\2017\win_b64\code\bin\odbAccess.pyd"))

could not load library "C:\SIMULIA\CAE\2017\win_b64\code\bin\odbAccess.pyd"
The specified module could not be found.

Stacktrace:
 [1] include_string(::String, ::String) at .\loading.jl:522
glwc commented 6 years ago

@stevengj - Thanks again for the suggestions. I may have been a little optimistic about this, given Dessault Systemes very guarded approach to their software. Comments like this https://stackoverflow.com/questions/45006126/i-need-to-convert-a-python-script-developed-in-abaqus-pde-to-an-executable-file suggest that what I'm trying to do may not be possible, at least with the time and skills of a scientist and not a real hacker. I've read elsewhere that their python is modified and rebuilt from scratch. At the very least, using some python components requires accessing their kernels which in turn requires their flexlm license to be called/activated. (Edit: I have a full licensed Abaqus install on my machine of course, but it has not been activated while I run my Julia code - I'll try again with the license activated. Also the Abaqus toolkit provides their own DOS shell environment, maybe if I execute Julia from there...) At any rate, I'd hoped the "output database" structures and odbAccess group of python code was not part of this requirement.

I took a quick peek into odbAccess.pyd with a hex editor to see if the correct abaqus python symbols might be obvious, but I didn't see anything right away. I'll probably continue to play around with this a little longer, but it feels like I'm out of my depth.

Another approach to this might be to use pyjulia if it can be installed into the Abaqus python environment, and then pass the odb datastructures back over to Julia. I have no idea if/how this will work, might look into it a bit.

Graham