Open ttsesm opened 4 years ago
This wrapper attempts to mimic the Embree C API closely. So, if you want to figure out how to do something, my goal is for a user to be able to consult the Embree API first.
That said, this wrapper is in a prototype stage and hasn't been used or tested thoroughly yet.
Here's some code that I'm using currently for raytracing, somewhat out of context:
device = embree.Device()
geometry = device.make_geometry(embree.GeometryType.Triangle)
scene = device.make_scene()
vertex_buffer = geometry.set_new_buffer(
embree.BufferType.Vertex, # buf_type
0, # slot
embree.Format.Float3, # fmt
3*np.dtype('float32').itemsize, # byte_stride
V.shape[0], # item_count
)
vertex_buffer[:] = V[:]
index_buffer = geometry.set_new_buffer(
embree.BufferType.Index, # buf_type
0, # slot
embree.Format.Uint3, # fmt
3*np.dtype('uint32').itemsize, # byte_stride,
F.shape[0]
)
index_buffer[:] = F[:]
geometry.commit()
geom_id = scene.attach_geometry(geometry)
geometry.release()
scene.commit()
n0, p0 = N[i0], P[i0]
dP = P - p0
mask = np.logical_and(dP@n0 >= 0, np.sum(dP*N, axis=1) <= 0)
P_mask = P[mask]
M = sum(mask)
rayhit = embree.RayHit1M(M)
rayhit.tnear[:] = 0
rayhit.tfar[:] = np.inf
rayhit.prim_id[:] = embree.INVALID_GEOMETRY_ID
rayhit.geom_id[:] = embree.INVALID_GEOMETRY_ID
rayhit.org[:] = p0 + eps*n0
rayhit.dir[:] = P_mask - p0
context = embree.IntersectContext()
scene.intersect1M(context, rayhit)
H = np.zeros((P.shape[0],), dtype=bool)
I_mask = np.where(mask)[0]
H[I_mask] = rayhit.prim_id == I_mask
In this example, I'm shooting some rays from perturbed triangle centroids to other triangles and seeing which rays actually hit their target triangles. In this case, I've loaded an OBJ using another Python library (doesn't really matter which), and then used Embree's API to tell it about the mesh's vertices and faces.
If you aren't familiar with Embree, if you consult the Embree API, you'll find that the code above matches what is done in Embree's provided examples.
Please let me know if you run into any trouble, and feel free to post code and mesh files here.
Please note: the entire API isn't wrapped yet, but should be very easy to wrap more functions. If a function from the Embree API that you need is missing, please say so. You can check embree.pyx to see if a function has been wrapped.
@sampotter thanks for the prompt response and the example. However, just before we go to code examples I would like to clarify the installation procedure. How someone should install the wrapper?
I tried to run pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" .
from my virtual environment where C:\Program Files\Intel\Embree3
is the path to my Embree3 installation but it failed giving me some errors (I can post the terminal output if needed), thus I guess there is something I am missing.
No problem.
Can you tell me a little more about your setup? I see that you're using Windows, which should work in principle, but is less familiar to me. Are you using Anaconda?
At this point, I haven't tested this wrapper on Windows, so we'll need to figure out an appropriate workflow for getting it installed.
And yes, please post the build output.
Yes, currently I am on a windows machine. I know that usually with this kind of things in linux is more straight forward. However, in principle someone should be able to install the wrapper on both systems.
Anyways back to the point, in my project I am using Pycharm where I have set both a conda
and a virtualenv
environment, the pip
command above I've run it from my virtualenv
environment.
As you correctly figured out I am on a windows 10 machine, with Python 3.8, and I have installed Embree v3 by using the windows installer.
This is the output from the build:
>>pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" .
c:\users\ttsesm\desktop\dev\radiosity\r_venv\lib\site-packages\pip\_internal\commands\install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
Processing c:\users\ttsesm\desktop\dev\python-embree
Skipping wheel build for embree, due to binaries being disabled for it.
Installing collected packages: embree
Running setup.py install for embree ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"'; __file_
_='"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"
'"'exec'"'"'))' build_ext '-IC:\Program Files\Intel\Embree3\include' '-LC:\Program Files\Intel\Embree3\lib' install --record 'C:\Users\TTsesm\AppData\Local\Temp\pip-record-rqreguy3\install-record.txt' --single-version-externally-ma
naged --compile --install-headers 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\include\site\python3.8\embree'
cwd: C:\Users\TTsesm\AppData\Local\Temp\pip-req-build-dtrlbuol\
Complete output (14 lines):
running build_ext
building 'embree' extension
creating build
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I/opt/local/include "-IC:\Program Files\Intel\Embree3\include" -Ic:\users\ttsesm\desktop\dev\radiosity\r_venv\include -IC:\Users\TTsesm\AppData\Local\Programs\Python\Python38\include -IC:\Users\TTsesm\AppData\Local\Programs\Python\Python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Com
munity\VC\Tools\MSVC\14.26.28801\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x
86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\
10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcembree.c /Fobuild\temp.win-amd64-3.8\Release\embree.obj
embree.c
embree.c(9654): warning C4013: 'posix_memalign' undefined; assuming extern returning int
creating C:\Users\TTsesm\AppData\Local\Temp\pip-req-build-dtrlbuol\build\lib.win-amd64-3.8
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:/opt/local/lib "/LIBPATH:C:\Program F
iles\Intel\Embree3\lib" /LIBPATH:c:\users\ttsesm\desktop\dev\radiosity\r_venv\libs /LIBPATH:C:\Users\TTsesm\AppData\Local\Programs\Python\Python38\libs /LIBPATH:C:\Users\TTsesm\AppData\Local\Programs\Python\Python38 /LIBPAT
H:c:\users\ttsesm\desktop\dev\radiosity\r_venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio
\2019\Community\VC\Tools\MSVC\14.26.28801\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Wind
ows Kits\10\lib\10.0.18362.0\um\x64" embree3.lib /EXPORT:PyInit_embree build\temp.win-amd64-3.8\Release\embree.obj /OUT:build\lib.win-amd64-3.8\embree.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.li
b
Creating library build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\embree.cp38-win_amd64.exp
embree.obj : error LNK2001: unresolved external symbol posix_memalign
build\lib.win-amd64-3.8\embree.cp38-win_amd64.pyd : fatal error LNK1120: 1 unresolved externals
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build
-dtrlbuol\\setup.py'"'"'; __file__='"'"'C:\\Users\\TTsesm\\AppData\\Local\\Temp\\pip-req-build-dtrlbuol\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close(
);exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext '-IC:\Program Files\Intel\Embree3\include' '-LC:\Program Files\Intel\Embree3\lib' install --record 'C:\Users\TTsesm\AppData\Local\Temp\pip-record-rqreguy3\install-record.tx
t' --single-version-externally-managed --compile --install-headers 'c:\users\ttsesm\desktop\dev\radiosity\r_venv\include\site\python3.8\embree' Check the logs for full command output.
How did you install Embree?
Do you think that it might have to do with the Embree installation which is linked against the Visual Studio 2013 (VC12) and Visual Studio 2015/2017 (VC14) runtime?
I installed it by using the .msi
windows installer from here https://www.embree.org/downloads.html
No, your installation doesn't appear to be a problem.
It can't find posix_memalign
, which makes sense. It looks like I can use _aligned_malloc
instead on Windows. I'll need to try building this on Windows, which should be pretty easy for me right now.
If you want to try debugging this yourself, look for instances of posix_memalign
in embree.pyx and see if you can replace them with calls to _aligned_malloc
. It will be necessary to add an aligned malloc that's platform independent, where the appropriate function calls are selected using preprocessor macros.
Actually it seems that it is not a direct replacement, since the set of input parameters are different. I found this stackoverflow thread https://stackoverflow.com/questions/33696092/whats-the-correct-replacement-for-posix-memalign-in-windows where they discussing the same issue and the provided solution seems to do the trick.
That's what I was looking at, as well. :-)
To provide some context, when using Embree, things that are allocated on the heap need to be aligned appropriately for SIMD. This is explained pretty clearly in the Embree API, which I linked to in my first reply.
Regardless of the exact API of the function, what's needed is a malloc that can align allocated memory to the right byte boundary for use with SIMD.
@sampotter do you think that you could use the C's malloc
, realloc
and free
or the Python's C-API memory allocation functions instead of the posix one.
Based on this https://tianrunhe.wordpress.com/2012/04/23/aligned-malloc-in-c/ it seems that aligned_malloc
can be done with calls to malloc
.
Here are some other reference links:
https://cython.readthedocs.io/en/latest/src/tutorial/memory_allocation.html
https://github.com/linux-rdma/rdma-core/blob/master/pyverbs/mem_alloc.pyx
I couldn't find any way to use calls to _aligned_malloc
, which is valid for windows.
I also found the following link which could be usefull:
Hey @ttsesm, I just pushed a commit that should fix this. I switched from posix_memalign to aligned_alloc on POSIX, and _aligned_malloc on Windows, which has the same API. I was able to compile without any trouble on Windows but haven't tested this. Let me know how this works.
Hmm, thought I had fixed this, but my changed cause some problems on macOS, which I wasn't able to figure out (in the small amount of time I budgeted for it).
I created a new branch off of develop (aligned_allocation) where we can try to figure this out. The latest commit there is working on Mac, but haven't had a chance to try it on Windows or Linux yet.
Thanks @sampotter, I've managed to make it compile in Windows (I do not have access to a linux machine currently). I had to make the following change though:
if errno == EINVAL:
raise Exception('bad alignment')
elif errno == ENOMEM:
in line https://github.com/sampotter/python-embree/blob/802b5f46c1960193b026560596df8e381ba91d36/embree.pyx#L19 otherwise it was complaining in line https://github.com/sampotter/python-embree/blob/802b5f46c1960193b026560596df8e381ba91d36/embree.pyx#L803 that EINVAL
was not declared.
Thanks for catching that, and glad you were able to get it to compile. If you put together a pull request I'll be happy to accept it.
@sampotter I had some time to play a bit with the provided code in one of your first responses. However, while embree module is compiled and installed it doesn't seem that I can import it. Any idea what could be the issue?
I've compiled and installed it running the following command from the python-embree
root folder while being in my venv
:
pip install --global-option=build_ext --global-option="-IC:\Program Files\Intel\Embree3\include" --global-option="-LC:\Program Files\Intel\Embree3\lib" .
As you can see, I can find it in my site packages in pycharm:
but if I try to import it so that to be able to use it, it doesn't find it.
@ttsesm Sorry about the delay. Could you be a little more explicit about what's going wrong? It's hard for me to help with so little info.
@ttsesm Closing this due to lack of activity.
I also had trouble importhing this package. I've isolated the issue to pytorch_geometric
, importing works as long as I import embree first.
@sampotter I actually get the error:
ImportError: DLL load failed while importing embree: The specified module could not be found.
Any suggestions on how to fix this?
Very hard to say without more information. Could you provide more information about your setup, what version of Python you have, whether you're using vanilla Python or Anaconda, what version of Embree itself, how you installed it, etc etc
Hi @sampotter,
Setup: OS: Windows 10 x64 Professional, Build 1909 Python: 3.8.10 x64 (vanilla) Numpy: numpy‑1.21.1+mkl‑cp38‑cp38‑win_amd64.whl from Christoph Gohlke's collection Embree: 3.13.0 x64 (Standard Install Location: C:\Program Files\Intel\Embree3)
Steps to Reproduce: (I use MSYS2 bash terminal below)
project
venv
in subfolder .venv
(py -m venv .venv
) and activate itpy -m pip install numpy‑1.21.1+mkl‑cp38‑cp38‑win_amd64.whl
)py setup.py build_ext -I/c/Program\ Files/Intel/Embree3/include -L /c/Program\ Files/Intel/Embree3/lib
py setup.py install
embree
fails with:
ImportError: DLL load failed while importing embree: The specified module could not be found.
.egg
and .pyd
files are created. Looking further at embree.py
C:\project\.venv\Lib\site-packages\embree-0.0.0-py3.8-win-amd64.egg
│ embree.cp38-win_amd64.pyd
│ embree.py
│
├───EGG-INFO
│ dependency_links.txt
│ native_libs.txt
│ not-zip-safe
│ PKG-INFO
│ SOURCES.txt
│ top_level.txt
│
└───__pycache__
embree.cpython-38.pyc
So I open an ipython
terminal and step through the __build__()
function:
In [1]: cd C:\project\.venv\Lib\site-packages\embree-0.0.0-py3.8-win-amd64.egg
In [2]: import sys, pkg_resources, importlib.util
In [3]: # In this instance, `__name__` is `__main__`, not `embree`, just fyi
In [4]: __file__ = pkg_resources.resource_filename(__name__, 'embree.cp38-win_amd64.pyd')
In [5]: __file__
Out [5]: 'embree.cp38-win_amd64.pyd'
In [6]: spec = importlib.util.spec_from_file_location(__name__,__file__)
In [7]: spec
Out [7]: ModuleSpec(name='__main__', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x000002176D5FC7C0>, origin='C:\\project\\.venv\\Lib\\site-packages\\embree-0.0.0-py3.8-win-amd64.egg\\embree.cp38-win_amd64.pyd')
In [8]: mod = importlib.util.module_from_spec(spec)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
C:\project\.venv\Lib\site-packages\embree-0.0.0-py3.8-win-amd64.egg\embree.cp38-win_amd64.pyd in <module>
C:\Program Files\Python38\lib\importlib\_bootstrap.py in module_from_spec(spec)
C:\Program Files\Python38\lib\importlib\_bootstrap_external.py in create_module(self, spec)
C:\Program Files\Python38\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)
ImportError: DLL load failed while importing __main__: The specified module could not be found.
Hmm, I looked through what you sent, but nothing jumps out at me. Unfortunately, I don't have a Windows machine to test with at the moment, and I don't use Python on Windows very often (if at all). Of course, I would be very happy to get this issue resolved, but I'm going to need to punt and ask you to try to figure this out yourself... If you can determine what the problem is and put it together in a pull request, I'll trust your judgment and happily merge it. Sorry I can't be more help at the moment.
@sampotter Not to put you on the spot, but you provided Windows install instructions. How did you verify the install works properly on a Windows machine if you don't use Windows?
I had access to a Windows machine for work over a year ago, but no longer do. At the time, another person who was using this library (probably the OP for this thread) wanted some help, and I happened to have the time to give it a shot. Of course, this is an open source project. If these instructions don't work for you and you find a way to improve them so that they handle a broader set of cases, it would be very helpful to include them.
@sampotter I see now, that is completely understandable. Thank you for venturing into Windows for us! I think scikit-build
with a CMakeLists.txt
script might do nicely to fix the issue I'm seeing here. I'll see what magic I can conjure up and report back if I have any success.
Sure, please give it a shot if you have time and I'll be happy to merge any PRs if they seem reasonable. Make sure to @ me when you post so I don't miss anything.
@adam-grant-hendry Any luck with this?
@sampotter Not yet. Been busy with the day job. I got Embree2 to compile though. See pyvista #468
No problem, and nice job---that's good to know... it will be helpful to let people know there's another option on Windows in case they run into trouble.
@sampotter I would like to use the embree lib for raycasting since it provides a nice accelerated solution. However, I am not quite sure how to use your wrapper. Would be possible to write a small example showing some use case. For example considering that someone has the indices and faces of a mesh object in numpy arrays how to create the corresponding mesh in embree by using this wrapper?