search

pyvista / pyvista-support

[moved] Please head over to the Discussions tab of the PyVista repository
https://github.com/pyvista/pyvista/discussions
60 stars 4 forks source link

Install `pyembree` on Windows without Conda #468

Closed adamgranthendry closed 3 years ago

adamgranthendry commented 3 years ago

I am trying to install pyembree without conda (I have been successful using conda, but I would prefer to use poetry). I have placed a question here and am looking for some help. I am able to install trimesh and rtree with poetry, but not pyembree. I need all three libraries to use Vectorized Ray Tracing and am looking for some help here.

So far, my steps have been:

System: Windows 10 x64, Build 1909, Python 3.8.10

  1. Install Embree v3.13.0 via the msi installer
  2. Install Microsoft C++ Build Tools for Visual Studio Community 2019, including (a) Desktop development with C++ (b) .NET desktop build tools (c) Universal Windows Platform build tools (d) .NET build tools (e) Vistual Studio extension development
  3. Install the Windows 10.0.19041.685 SDK
  4. Add the following to the system path: 
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30037\bin\Hostx64\x64
C:\Program Files (x86)\Windows Kits\10
C:\Program Files (x86)\Windows Kits\10\Lib\10.0.10240.0\ucrt\x64\
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\
  5. Install poetry 1.1.7 via PowerShell
  6. Clone the pyembree 0.1.6 repo to %USERPROFILE%\Code\myproject.venv\Lib\site-packages\ (creates a subfolder pyembree): git clone https://github.com/scopatz/pyembree.git
  7. Open a Developer Command Prompt for VS 2019, which activates vcvarsall.bat for x64 builds
  8. Run the following: 
    set INCLUDE="C:\Program Files\Intel\Embree3\include\embree3\"
set LIB="C:\Program Files\Intel\Embree3\lib\"
cd "%USERPROFILE%\Code\myproject"
.venv\Scripts\activate.bat
cd ".venv\Lib\site-packages\pyembree
py setup.py install

However, no matter what, I get the following output:

Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running install
running bdist_egg
running egg_info
writing pyembree.egg-info\PKG-INFO
writing dependency_links to pyembree.egg-info\dependency_links.txt
writing top-level names to pyembree.egg-info\top_level.txt
reading manifest file 'pyembree.egg-info\SOURCES.txt'
adding license file 'LICENSE'
adding license file 'AUTHORS'
writing manifest file 'pyembree.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
running build_ext
building 'pyembree.mesh_construction' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

Any ideas what I'm doing wrong?

adamgranthendry commented 3 years ago

@akaszynski I think part of my problem might be that I'm trying to use Embree 3. Am I right? Does pyembree rely on Embree 2?

akaszynski commented 3 years ago

Main issue seems to be python isn't able to figure out where msbuild is. Surprising that it's not able to find it considering how you've made it available in your path. What about trying without activating the developer command prompt?

adamgranthendry commented 3 years ago

Agreed. Let me try that. I also downgraded Embree from v3 to v2.17.7.

adamgranthendry commented 3 years ago

@akaszynski Unfortunately, I have the same problem. From within a regular Command Prompt in my virtual environment, I ran

set INCLUDE="C:\Program Files\Intel\Embree v2.17.7 x64\include\embree2\"
set LIB="C:\Program Files\Intel\Embree v2.17.7 x64\lib\
cd .venv\Lib\site-packages\pyembree
poetry run py setup.py install

and I still get that it can't find Microsoft Visual C++ 14.0. I can run cl though and I get

Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30038.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

However (it's been a while), if I run msvc I get

'msvc' is not recognized as an internal or external command,
operable program or batch file.

I can't recall if msvc is an executable or not or if it is sufficient to have the cl.exe compiler.

adamgranthendry commented 3 years ago

@akaszynski Oh, I should also add that I've installed setuptools 57.4.0

adamgranthendry commented 3 years ago

@akaszynski I added C:\Program Files (x86)\MSBuild\15.0\.Net\.NetNative\15.0.24211\x64\ilc\tools\, so MSBuild is available now:

Microsoft (R) Build Engine version 16.10.2+857e5a733 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1003: Specify a project or solution file. The current working directory does not contain a project or solution file.

However, I still get the same error:

error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
adamgranthendry commented 3 years ago

@akaszynski I'm beginning to suspect that I must actually install Visual Studio 2015 for this to work, per this SO post and the fact that nothing else has worked so far.

Have you built this on Windows before?

akaszynski commented 3 years ago

I've never build pyembree on Windows. I'm a 90% linux developer and have always run into issues compiling anything other than "hello world" on windows.

You may want to checkout this SO question as well

adamgranthendry commented 3 years ago

@akaszynski Thank you! Yes, agreed. I am a Linux developer at home and a Windows developer at work out of necessity. I too have pretty much never been able to get Windows to compile anything well ever, haha!

adamgranthendry commented 3 years ago

Adding this SO link as well. Will report back with results shortly...

adamgranthendry commented 3 years ago

@akaszynski I was able to do it. I needed to install Visual Studio 2015 and the Windows 10 SDK and add their include and lib libraries with the Embree v2.17.7 x64 libraries by running the following in a VS2015 x64 Native Tools Command Prompt as Administrator (the carets were necessary because double-quotes wasn't properly handling spaces):

set INCLUDE=C:\Program^ Files\Intel\Embree^ v2.17.7^ x64\include;C:\Program^ Files\Intel\Embree^ v2.17.7^ x64\include\embree2;C:\Program^ Files^ (x86)\Microsoft^ Visual^ Studio^ 14.0\VC\include;C:\Program^ Files^ (x86)\Windows^ Kits\10\Include\10.0.19041.0
set LIB=C:\Program^ Files\Intel\Embree^ v2.17.7^ x64\lib;C:\Program^ Files^ (x86)\Microsoft^ Visual^ Studio^ 14.0\VC\lib;C:\Program^ Files^ (x86)\Windows^ Kits\10\Lib\10.0.19041.0
set DISTUTILS_USE_SDK=1
"%PROGRAMFILES(X86)%\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64

The output becomes (replacing my username and project path for security purposes):

Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
running install
running bdist_egg
running egg_info
writing pyembree.egg-info\PKG-INFO
writing dependency_links to pyembree.egg-info\dependency_links.txt
writing top-level names to pyembree.egg-info\top_level.txt
reading manifest file 'pyembree.egg-info\SOURCES.txt'
adding license file 'LICENSE'
adding license file 'AUTHORS'
writing manifest file 'pyembree.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
running build_ext
building 'pyembree.mesh_construction' extension
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include -IC:\Users\<user>\<project_folder>\.venv\include "-IC:\Program Files\Python38\include" "-IC:\Program Files\Python38\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include\embree2" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0" /EHsc /Tppyembree\mesh_construction.cpp /Fobuild\temp.win-amd64-3.8\Release\pyembree\mesh_construction.obj
mesh_construction.cpp
c:\users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
pyembree\mesh_construction.cpp(2069): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
pyembree\mesh_construction.cpp(2348): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
pyembree\mesh_construction.cpp(2357): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
pyembree\mesh_construction.cpp(2935): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
pyembree\mesh_construction.cpp(2944): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
pyembree\mesh_construction.cpp(3240): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
pyembree\mesh_construction.cpp(3249): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\<user>\<project_folder>\.venv\libs "/LIBPATH:C:\Program Files\Python38\libs" "/LIBPATH:C:\Program Files\Python38" /LIBPATH:C:\Users\<user>\<project_folder>\.venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files\Intel\Embree v2.17.7 x64\lib" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0" embree.lib /EXPORT:PyInit_mesh_construction build\temp.win-amd64-3.8\Release\pyembree\mesh_construction.obj /OUT:build\lib.win-amd64-3.8\pyembree\mesh_construction.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\pyembree\mesh_construction.cp38-win_amd64.lib
mesh_construction.obj : warning LNK4197: export 'PyInit_mesh_construction' specified multiple times; using first specification
   Creating library build\temp.win-amd64-3.8\Release\pyembree\mesh_construction.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\pyembree\mesh_construction.cp38-win_amd64.exp
Generating code
Finished generating code
building 'pyembree.rtcore' extension
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include -IC:\Users\<user>\<project_folder>\.venv\include "-IC:\Program Files\Python38\include" "-IC:\Program Files\Python38\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include\embree2" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0" /EHsc /Tppyembree\rtcore.cpp /Fobuild\temp.win-amd64-3.8\Release\pyembree\rtcore.obj
rtcore.cpp
c:\users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\<user>\<project_folder>\.venv\libs "/LIBPATH:C:\Program Files\Python38\libs" "/LIBPATH:C:\Program Files\Python38" /LIBPATH:C:\Users\<user>\<project_folder>\.venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files\Intel\Embree v2.17.7 x64\lib" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0" embree.lib /EXPORT:PyInit_rtcore build\temp.win-amd64-3.8\Release\pyembree\rtcore.obj /OUT:build\lib.win-amd64-3.8\pyembree\rtcore.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\pyembree\rtcore.cp38-win_amd64.lib
rtcore.obj : warning LNK4197: export 'PyInit_rtcore' specified multiple times; using first specification
   Creating library build\temp.win-amd64-3.8\Release\pyembree\rtcore.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\pyembree\rtcore.cp38-win_amd64.exp
Generating code
Finished generating code
building 'pyembree.rtcore_scene' extension
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include -IC:\Users\<user>\<project_folder>\.venv\include "-IC:\Program Files\Python38\include" "-IC:\Program Files\Python38\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include\embree2" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0" /EHsc /Tppyembree\rtcore_scene.cpp /Fobuild\temp.win-amd64-3.8\Release\pyembree\rtcore_scene.obj
rtcore_scene.cpp
c:\users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
pyembree\rtcore_scene.cpp(2253): warning C4996: 'rtcDeviceSetErrorFunction': was declared deprecated
c:\program files\intel\embree v2.17.7 x64\include\embree2\rtcore.h(233): note: see declaration of 'rtcDeviceSetErrorFunction'
pyembree\rtcore_scene.cpp(2550): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\<user>\<project_folder>\.venv\libs "/LIBPATH:C:\Program Files\Python38\libs" "/LIBPATH:C:\Program Files\Python38" /LIBPATH:C:\Users\<user>\<project_folder>\.venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files\Intel\Embree v2.17.7 x64\lib" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0" embree.lib /EXPORT:PyInit_rtcore_scene build\temp.win-amd64-3.8\Release\pyembree\rtcore_scene.obj /OUT:build\lib.win-amd64-3.8\pyembree\rtcore_scene.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\pyembree\rtcore_scene.cp38-win_amd64.lib
rtcore_scene.obj : warning LNK4197: export 'PyInit_rtcore_scene' specified multiple times; using first specification
   Creating library build\temp.win-amd64-3.8\Release\pyembree\rtcore_scene.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\pyembree\rtcore_scene.cp38-win_amd64.exp
Generating code
Finished generating code
building 'pyembree.triangles' extension
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include -IC:\Users\<user>\<project_folder>\.venv\include "-IC:\Program Files\Python38\include" "-IC:\Program Files\Python38\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include" "-IC:\Program Files\Intel\Embree v2.17.7 x64\include\embree2" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0" /EHsc /Tppyembree\triangles.cpp /Fobuild\temp.win-amd64-3.8\Release\pyembree\triangles.obj
triangles.cpp
c:\users\<user>\<project_folder>\.venv\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\<user>\<project_folder>\.venv\libs "/LIBPATH:C:\Program Files\Python38\libs" "/LIBPATH:C:\Program Files\Python38" /LIBPATH:C:\Users\<user>\<project_folder>\.venv\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.19041.0\um\x64" "/LIBPATH:C:\Program Files\Intel\Embree v2.17.7 x64\lib" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0" embree.lib /EXPORT:PyInit_triangles build\temp.win-amd64-3.8\Release\pyembree\triangles.obj /OUT:build\lib.win-amd64-3.8\pyembree\triangles.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\pyembree\triangles.cp38-win_amd64.lib
triangles.obj : warning LNK4197: export 'PyInit_triangles' specified multiple times; using first specification
   Creating library build\temp.win-amd64-3.8\Release\pyembree\triangles.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\pyembree\triangles.cp38-win_amd64.exp
Generating code
Finished generating code
creating build\bdist.win-amd64
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\mesh_construction.cp38-win_amd64.pyd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore.cp38-win_amd64.pyd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore.pxd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore_geometry.pxd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore_geometry_user.pxd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore_ray.pxd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore_scene.cp38-win_amd64.pyd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\rtcore_scene.pxd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\triangles.cp38-win_amd64.pyd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\__init__.pxd -> build\bdist.win-amd64\egg\pyembree
copying build\lib.win-amd64-3.8\pyembree\__init__.py -> build\bdist.win-amd64\egg\pyembree
byte-compiling build\bdist.win-amd64\egg\pyembree\__init__.py to __init__.cpython-38.pyc
creating stub loader for pyembree\mesh_construction.cp38-win_amd64.pyd
creating stub loader for pyembree\rtcore.cp38-win_amd64.pyd
creating stub loader for pyembree\rtcore_scene.cp38-win_amd64.pyd
creating stub loader for pyembree\triangles.cp38-win_amd64.pyd
byte-compiling build\bdist.win-amd64\egg\pyembree\mesh_construction.py to mesh_construction.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\pyembree\rtcore.py to rtcore.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\pyembree\rtcore_scene.py to rtcore_scene.cpython-38.pyc
byte-compiling build\bdist.win-amd64\egg\pyembree\triangles.py to triangles.cpython-38.pyc
creating build\bdist.win-amd64\egg\EGG-INFO
copying pyembree.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying pyembree.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying pyembree.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying pyembree.egg-info\not-zip-safe -> build\bdist.win-amd64\egg\EGG-INFO
copying pyembree.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
writing build\bdist.win-amd64\egg\EGG-INFO\native_libs.txt
creating dist
creating 'dist\pyembree-0.1.6-py3.8-win-amd64.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing pyembree-0.1.6-py3.8-win-amd64.egg
creating c:\users\<user>\<project_folder>\.venv\lib\site-packages\pyembree-0.1.6-py3.8-win-amd64.egg
Extracting pyembree-0.1.6-py3.8-win-amd64.egg to c:\users\<user>\<project_folder>\.venv\lib\site-packages
Adding pyembree 0.1.6 to easy-install.pth file

Installed c:\users\<user>\<project_folder>\.venv\lib\site-packages\pyembree-0.1.6-py3.8-win-amd64.egg
Processing dependencies for pyembree==0.1.6
Finished processing dependencies for pyembree==0.1.6
adamgranthendry commented 3 years ago

@akaszynski Should we add install instructions to pyvista? Might be useful for Windows users. Also, can we somehow give the .egg I generated to the pyembree-feedstock so they can make a pip installer?

akaszynski commented 3 years ago

Install instructions would be really helpful. If you can, add it to our optional features guide at:

https://github.com/pyvista/pyvista/blob/main/doc/user-guide/optional_features.rst

I think they would be interested in creating a pip package. Would be much easier than building from source. Really, they'll need to make a pipeline that does it.

adamgranthendry commented 3 years ago

Installation Instructions

System

OS: Windows 10 x64, Build 1909 Python: 3.8.10

Steps

  1. Install Embree v2.17.7 x64 via the msi installer

(NOTE: To date, pyembree still relies on Embree 2 and has not been updated for Embree 3.)

  1. Install Microsoft Visual Studio 2015

(NOTE: pyembree requires version 2015. The error 

error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": 
https://visualstudio.microsoft.com/visual-cpp-build-tools/

is incorrect. The version of Visual Studio MUST be exactly version 2015. Search for version 2015 here under "Still want an older version?", which will take you to https://my.visualstudio.com/Downloads. You must now have a Microsoft account to install older versions of Visual Studio). This will be an ISO file that you must mount and run the contained executable (be prepared for several hours of downloading and installing)

  1. Install the Windows 10.0.19041.685 SDK

  2. Install Poetry 1.1.7 via PowerShell

  3. Create your project and initialize a virtual environment with venv and package management with poetry.

(NOTE: In this example, the virtual environment is contained in a folder named .venv within the project folder itself)

  1. Open a VS2015 x64 Native Tools Command Prompt as an Administrator. Navigate to the <project_folder> and activate the virtual environment (.venv\Scripts\activate.bat)

(NOTE: The installation must be done in a command prompt terminal. Since we also need the build tools from VS2015, it is ideal to use the VS2015 x64 Native Tools Command Prompt. We need to set several environment variables and the set command is actually different in PowerShell than what it is in Command Prompt.)

  1. Navigate to <project_folder>\.venv\Lib\site-packages and clone the pyembree repo with the following command:
git clone https://github.com/scopatz/pyembree.git
  1. Activate appropriate build environment variables by running:
"%PROGRAMFILES(X86)%\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" amd64
  1. Set the path to the Embree, Visual Studio, and Windows SDK include and lib files and tell msbuild to use the Windows_SDK when compiling by running the following:
set INCLUDE=C:\Program^ Files\Intel\Embree^ v2.17.7^ x64\include;C:\Program^ Files\Intel\Embree^ v2.17.7^ x64\include\embree2;C:\Program^ Files^ (x86)\Microsoft^ Visual^ Studio^ 14.0\VC\include;C:\Program^ Files^ (x86)\Windows^ Kits\10\Include\10.0.19041.0
set LIB=C:\Program^ Files\Intel\Embree^ v2.17.7^ x64\lib;C:\Program^ Files^ (x86)\Microsoft^ Visual^ Studio^ 14.0\VC\lib;C:\Program^ Files^ (x86)\Windows^ Kits\10\Lib\10.0.19041.0
set DISTUTILS_USE_SDK=1

(NOTE: The carets are necessary so Windows properly handles the spaces in the file paths. At the time of this writing, double-quotes did not work, but your mileage may vary.)

  1. Lastly, run
poetry run py setup.py install

If all goes according to plan, python will install pyembree in your virtual environment site-packages folder and finish with the statement:

Installed <project_folder>\.venv\lib\site-packages\pyembree-0.1.6-py3.8-win-amd64.egg
Processing dependencies for pyembree==0.1.6
Finished processing dependencies for pyembree==0.1.6
adamgranthendry commented 3 years ago

@akaszynski I will try to do a PR as soon as I'm able. One thing for me though is that my code seems to run slower. It works, but I'm not sure why it's so much slower. Does conda include any kinds of optimizations?

akaszynski commented 3 years ago

I will try to do a PR as soon as I'm able.

If you forget to, I'll add it.

Does conda include any kinds of optimizations?

Depends on how they build it. It's possible they enable O3 or OpenMP by default.

adamgranthendry commented 3 years ago

@akaszynski Two questions:

  1. How can I see how they (conda-forge) build it?
  2. How fast does the example code run on your machine with the conda install version? Mine is stuck at data.multi_ray_trace and takes 10 minutes to render.
adamgranthendry commented 3 years ago

@akaszynski FYI, I'd wait to make a PR with install instructions. The build of pyembree I made is just too slow. The ray_trace example actually was still going after 30 minutes, after which point I killed the program. Conda comes pre-shipped with a lot of dll's, so that may contribute to its increased speed, but at this point I really don't know yet.

adamgranthendry commented 3 years ago

@akaszynski Okay, I digged a little deeper into the speed issue. I installed pyembree with conda and built it in a separate venv per my method above and found the only difference between the files in my venv was that the conda build includes dlls from the folders:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Remote Debugger\x64\
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT\

Any idea how I can build with those as well?

akaszynski commented 3 years ago

I'm not sure, but if you check the conda recipe for pyembree, it might shed some light on how they built it: https://github.com/conda-forge/pyembree-feedstock

adamgranthendry commented 3 years ago

@akaszynski I think I see what's going on: your suspicions were correct. numpy is built with MKL and openMP using O3 optimizations in the conda distribution. However, this is not done by default with the standard PyPI numpy. This is why my installation method works, but is running slower.

I'm running through the documentation right now to figure out how to build numpy with MKL support and will update the install instructions after.

akaszynski commented 3 years ago

I'm running through the documentation right now to figure out how to build numpy with MKL support and will update the install instructions after.

Good point. Didn't consider that. That's the nice thing about using conda, is that they can package this holistically.

adamgranthendry commented 3 years ago

@akaszynski Correct, agreed. Conda takes out the work of figuring out how to install such packages and their dependencies for heavy-lifting scientific applications.

adam-grant-hendry commented 3 years ago

@akaszynski I'm nearly there. I'm trying to build this with CMake and scikit-build using embree2 downloaded via vcpkg. Right now, I'm stuck at the following linking error:

mesh_construction.obj : error LNK2001: unresolved external symbol __imp_rtcMapBuffer
mesh_construction.obj : error LNK2001: unresolved external symbol __imp_rtcNewTriangleMesh
mesh_construction.obj : error LNK2001: unresolved external symbol __imp_rtcUnmapBuffer

I think __imp_ might have to do with DLL's, but I'm not sure. Any thoughts?

akaszynski commented 3 years ago

I think __imp_ might have to do with DLL's, but I'm not sure. Any thoughts?

It's absolutely due to the dlls, but as for which one, it's a mystery since symbol names tend to change with release and build toolset.

adamgranthendry commented 3 years ago

@akaszynski Well that's promising to know that that's indeed the problem. FYI, I think the pressure is going to be on for us (or more appropriately me) to get a working port of pyembree for Windows since today I saw that Windows support was dropped for pyembree from conda-forge (https://anaconda.org/conda-forge/pyembree)

adamgranthendry commented 3 years ago

@akaszynski Let me provide you my updated steps to-date:

Installation Instructions

System

Tested on:

OS: Windows 10 x64 Professional, Build 1909 Python: 3.8.10

Steps

  1. Install Microsoft Visual C++ 14.X and Windows 10 SDK. These are required for building cython code.

(NOTE: The version of Microsoft Visual Studio is not the same as the version of Microsoft Visual C++. Visual Studio 2015, 2017, and 2019 all have MSVCv14X build tools. At the time of this writing, installing the Visual Studio 2019 Build Tools with

MSVCv142 - VS 2019 C++ x64/x86 build tools and Windows 10 SDK (10.0.18362.0)

components will suffice (select the Desktop development with C++ Workload if installing Visual Studio 2019).

  1. Install the Google ninja build tool with choco (Also install choco if you don't have it):
choco install ninja
  1. Install vcpkg in C:\\vcpkg and add the path to your System Environment Variables:
C:
mkdir vcpkg
cd C:\\vcpkg
git clone https://github.com/Microsoft/vcpkg.git
bootstrap-vcpkg.bat
vcpkg integrate install
  1. Install embree2 64-bit:
vcpkg install embree2:x64-windows

NOTE: To date, pyembree still relies on Embree 2 and has not been updated for Embree 3.

  1. Install cmake.

  2. Create your project folder and initialize a virtual environment with venv (Use Python 3.6 - 3.8). In this example, Python 3.8.5 x64-bit is chosen as it is the Python version used in Miniconda py38_4.9.2.

  3. Install the following packages:

py -m pip install numpy cython cmake ninja scikit-build wheel setuptools pyvista pykdtree

  1. Add the following cmake modules to your system cmake modules folder (e.g. C:\Program Files\CMake\share\cmake-3.21\Modules\).

  2. Navigate to <virtual environment folder>\Lib\site-packages and clone the pyembree repo:

git clone https://github.com/scopatz/pyembree.git
  1. Change directories into the pyembree folder and create the following top-level CMakeLists.txt
cmake_minimum_required(VERSION 3.21.0)
project(pyembree
    VERSION 0.1.6
    LANGUAGES CXX
)

set(CMAKE_CXX_COMPILER "g++")
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_CXX_STANDARD_REQUIRED True)

if(NOT CMAKE_BUILD_TYPE)
    set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE)
endif()

set(CMAKE_TOOLCHAIN_FILE "C:/vcpkg/scripts/buildsystems/vcpkg.cmake")

find_package(PythonExtensions REQUIRED)
find_package(Cython REQUIRED)
find_package(embree 2 CONFIG REQUIRED)

add_subdirectory(${PROJECT_NAME})
  1. Move to the subfolder pyembree and create the following sub-level CMakeLists.txt:
add_cython_target(mesh_construction.pyx CXX)
add_cython_target(rtcore_scene.pyx CXX)
add_cython_target(rtcore.pyx CXX)
add_cython_target(triangles.pyx CXX)

add_library(${PROJECT_NAME} STATIC ${mesh_constructions} ${rtcore_scene} ${rtcore} ${triangles})

target_sources(${PROJECT_NAME} PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

target_include_directories(${PROJECT_NAME}
    PUBLIC "<system python path>/include"
    PUBLIC "C:/vcpkg/installed/x64-windows/include/embree2"
    PUBLIC "<virtual environment folder>/Lib/site-packages/numpy/core/include"
)

target_link_directories(${PROJECT_NAME}
    PUBLIC "<system python path>/libs"
    PUBLIC "C:/vcpkg/installed/x64-windows/lib"
    PUBLIC "C:/vcpkg/installed/x64-windows/bin"
    PUBLIC "<virtual environment folder>/Lib/site-packages/numpy/core/lib"
)

target_link_libraries(${PROJECT_NAME}
    embree
)

python_extension_module(${PROJECT_NAME})

install(TARGETS ${PROJECT_NAME} LIBRARY DESTINATION lib)

replacing <system python path> (e.g. C:/Program Files/Python38) and <virtual environment folder> accordingly.

  1. Move back to the top-level pyembree folder and create the following setup.py file:
from setuptools import find_packages
from skbuild import setup

import numpy as np
from Cython.Build import cythonize

include_path = [np.get_include()]

ext_modules = cythonize('pyembree/*.pyx', language_level=3, include_path=include_path)
for ext in ext_modules:
    ext.include_dirs = include_path
    ext.libraries = ["embree"]

setup(
    name="pyembree",
    version='0.1.6',
    ext_modules=ext_modules,
    zip_safe=False,
    packages=find_packages(),
    include_package_data=True
)
  1. Add the following line to the top of every *.pyx and *.pxd file in pyembree:
# distutils: language=c++
  1. Build and install pyembree by running the following from the top-level pyembree folder:
py setup.py build
py setup.py install
  1. Finally, install rtree and trimesh:
py -m pip install rtree trimesh
adamgranthendry commented 3 years ago

@akaszynski I am still very new to CMake, so if you can get the equivalent of the above CMake recipes to work for Linux, that would be a step in the right direction. I'm also very new to scikit-build, but I know it is used to build itk and vtk, so I feel it should work.

adamgranthendry commented 3 years ago

Currently, running

py setup.py build

yields the following output

--------------------------------------------------------------------------------
-- Trying "Ninja (Visual Studio 15 2017 Win64 v141)" generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/usr/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/usr/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Project/.venv/Lib/site-packages/pyembree/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying "Ninja (Visual Studio 15 2017 Win64 v141)" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    C:\Project\.venv\lib\site-packages\pyembree\_skbuild\win-amd64-3.8\cmake-build
  Command:
    cmake 'C:\Project\.venv\lib\site-packages\pyembree' -G Ninja '-DCMAKE_INSTALL_PREFIX:PATH=C:\Project\.venv\lib\site-packages\pyembree\_skbuild\win-amd64-3.8\cmake-install' '-DPYTHON_EXECUTABLE:FILEPATH=C:\Project\.venv\Scripts\python.exe' -DPYTHON_VERSION_STRING:STRING=3.8.5 '-DPYTHON_INCLUDE_DIR:PATH=C:\Program Files\Python38\Include' '-DPYTHON_LIBRARY:FILEPATH=C:\Program Files\Python38\libs\python38.lib' -DSKBUILD:BOOL=TRUE '-DCMAKE_MODULE_PATH:PATH=C:\Project\.venv\lib\site-packages\skbuild\resources\cmake' -DCMAKE_BUILD_TYPE:STRING=Release

-- The CXX compiler identification is GNU 10.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/usr/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: C:/Project/.venv/Scripts/python.exe (found version "3.8.5") 
-- Found PythonLibs: optimized;C:/Program Files/Python38/libs/python38.lib;debug;C:/Program Files/Python38/libs/python38_d.lib (found version "3.8.5") 
-- Found Cython: C:/Project/.venv/Scripts/cython.exe  
-- Found PythonLibs: optimized;optimized;optimized;C:/Program Files/Python38/libs/python38.lib;optimized;debug;optimized;C:/Program Files/Python38/libs/python38_d.lib;debug;C:/Program Files/Python38/libs/python38_d.lib (found version "3.8.5") 
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    SKBUILD

-- Build files have been written to: C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build
[1/5] Building CXX object pyembree/CMakeFiles/pyembree.dir/rtcore_scene.cxx.obj
FAILED: pyembree/CMakeFiles/pyembree.dir/rtcore_scene.cxx.obj
C:\msys64\usr\bin\c++.exe  -I"C:/Program Files/Python38/include" -IC:/vcpkg/installed/x64-windows/include/embree2 -IC:/vcpkg/installed/x64-windows/include/tbb -IC:/Project/.venv/Lib/site-packages/numpy/core/include -I"C:/Program Files/Python38/Include" -O3 -DNDEBUG -std=c++17 -MD -MT pyembree/CMakeFiles/pyembree.dir/rtcore_scene.cxx.obj -MF pyembree\CMakeFiles\pyembree.dir\rtcore_scene.cxx.obj.d -o pyembree/CMakeFiles/pyembree.dir/rtcore_scene.cxx.obj -c C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/rtcore_scene.cxx
In file included from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/rtcore_scene.cxx:634:
C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/rtcore_scene.cxx:642:10: fatal error: embree2/rtcore.h: No such file or directory
  642 | #include "embree2/rtcore.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
[2/5] Building CXX object pyembree/CMakeFiles/pyembree.dir/rtcore.cxx.obj
FAILED: pyembree/CMakeFiles/pyembree.dir/rtcore.cxx.obj
C:\msys64\usr\bin\c++.exe  -I"C:/Program Files/Python38/include" -IC:/vcpkg/installed/x64-windows/include/embree2 -IC:/vcpkg/installed/x64-windows/include/tbb -IC:/Project/.venv/Lib/site-packages/numpy/core/include -I"C:/Program Files/Python38/Include" -O3 -DNDEBUG -std=c++17 -MD -MT pyembree/CMakeFiles/pyembree.dir/rtcore.cxx.obj -MF pyembree\CMakeFiles\pyembree.dir\rtcore.cxx.obj.d -o pyembree/CMakeFiles/pyembree.dir/rtcore.cxx.obj -c C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/rtcore.cxx
In file included from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/rtcore.cxx:634:
C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/rtcore.cxx:642:10: fatal error: embree2/rtcore.h: No such file or directory
  642 | #include "embree2/rtcore.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
[3/5] Building CXX object pyembree/CMakeFiles/pyembree.dir/triangles.cxx.obj
FAILED: pyembree/CMakeFiles/pyembree.dir/triangles.cxx.obj
C:\msys64\usr\bin\c++.exe  -I"C:/Program Files/Python38/include" -IC:/vcpkg/installed/x64-windows/include/embree2 -IC:/vcpkg/installed/x64-windows/include/tbb -IC:/Project/.venv/Lib/site-packages/numpy/core/include -I"C:/Program Files/Python38/Include" -O3 -DNDEBUG -std=c++17 -MD -MT pyembree/CMakeFiles/pyembree.dir/triangles.cxx.obj -MF pyembree\CMakeFiles\pyembree.dir\triangles.cxx.obj.d -o pyembree/CMakeFiles/pyembree.dir/triangles.cxx.obj -c C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/triangles.cxx
In file included from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/triangles.cxx:634:
C:/Project/.venv/Lib/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
C:/Project/.venv/Lib/site-packages/pyembree/_skbuild/win-amd64-3.8/cmake-build/pyembree/triangles.cxx:642:10: fatal error: embree2/rtcore.h: No such file or directory
  642 | #include "embree2/rtcore.h"
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\Project\.venv\lib\site-packages\skbuild\setuptools_wrap.py", line 589, in setup
    cmkr.make(make_args, env=env)
  File "C:\Project\.venv\lib\site-packages\skbuild\cmaker.py", line 496, in make
    raise SKBuildError(

An error occurred while building with CMake.
  Command:
    cmake --build . --target install --config Release --
  Source directory:
    C:\Project\.venv\lib\site-packages\pyembree
  Working directory:
    C:\Project\.venv\lib\site-packages\pyembree\_skbuild\win-amd64-3.8\cmake-build
Please see CMake's output for more information.

I need cmake to copy the embree2 subfolder that lives at C:/vcpkg/installed/x64-windows/include/embree2 to my pyembree/pyembree folder and to have it copy the DLLs over to pyembree/pyembree as well, but my cmake script isn't doing that.

Have I written it incorrectly?

adamgranthendry commented 3 years ago

@akaszynski I am also reaching out for further help on StackOverflow here.

akaszynski commented 3 years ago

Looks like cmake can't find the embree headers. Sadly, I'm not enough of an expert with cmake to be of help here.

adamgranthendry commented 3 years ago

@akaszynski I updated a bit more on the SO question. Now I'm stuck at a linker error.

adamgranthendry commented 3 years ago

@akaszynski Also seeing if I can post question on scikit-build

adamgranthendry commented 3 years ago

@akaszynski I solved the problem. I needed to manually copy and paste the DLLs over to the generated .egg folder in my site-packages. CMake is not required, but may be nice to have in the future. Would be good to turn this into a pip install on PyPI at some point. Here are the final instructions for working code on Windows:

Installation Instructions

Tested on:

OS: Windows 10 x64 Professional, Build 1909 Python: 3.8.5 x64 Date: 31-JULY-2021

Steps

  1. Install Microsoft Visual Studio 2017 Build Tools and the Windows 10 SDK. These are required for building cython code.

(NOTE: FYI, Microsoft Visual Studio version numbers are not the same as Microsoft Visual C++ Compiler versions. MSVCv14X x64/x86 build tools are required for Python 3.8. Installing the Visual Studio 2017 Build Tools with the Windows 10 SDK. If you wish to install Visual Studio 2017, select the Desktop development with C++ Workload and make sure the required Windows 10 SDK build tool is selected under Individual Components. Refer to the Python Wiki for the right build tool versions to install).

  1. Install vcpkg in C:\vcpkg and add the path to your System Environment Variables. Be sure to bootstrap and integrate vcpkg:
cd C:\vcpkg
git clone https://github.com/Microsoft/vcpkg.git
bootstrap-vcpkg.bat
vcpkg integrate install

Alternatively, you can download and install the embree-2.17.7.x64.msi Installer on Embree's GitHub page (to learn more about Embree, visit their website).

  1. Install embree2 64-bit. If using vcpkg, this is done from the command-line as:
vcpkg install embree2:x64-windows

NOTE: To date, pyembree relies on Embree 2 and has not been updated version 3. The above command will install version 2.17.7, which is the required version.

  1. Create your project folder and initialize a virtual environment with venv. In this example, Python 3.8.5 x64-bit is chosen, simply as it is the Python version used in Miniconda py38_4.9.2. (It may work for Python 3.6 and 3.7, but at the time of this writing, one should not go to higher than 3.8)

  2. Install the following packages:

py -m pip install numpy cython wheel setuptools pyvista pykdtree rtree trimesh
  1. Navigate to <virtual environment folder>\Lib\site-packages and clone the pyembree repo:
git clone https://github.com/scopatz/pyembree.git

and remove the underlying .git subfolder.

  1. Move into pyembree folder and modify the setup.py file to:
import os
from setuptools import find_packages, setup

import numpy as np
from Cython.Build import cythonize
from Cython.Distutils import build_ext

include_path = [
    np.get_include(),
]

ext_modules = cythonize("pyembree/*.pyx", language_level=3, include_path=include_path)

for ext in ext_modules:
    ext.include_dirs = include_path
    ext.libraries = [
        "pyembree/embree2/lib/embree",
        "pyembree/embree2/lib/tbb",
        "pyembree/embree2/lib/tbbmalloc",
    ]

setup(
    name="pyembree",
    version="0.1.6",
    cmdclass={"build_ext": build_ext},
    ext_modules=ext_modules,
    zip_safe=False,
    packages=find_packages(),
    include_package_data=True,
    package_data={"pyembree": ["*.cpp", "*.dll"]},
)
  1. Move into the pyembree subfolder. Copy the folder C:\vcpkg\installed\x64-windows\include\embree2 into this folder (if you installed from the .msi, this will be the folder C:\Program Files\Intel\Embree v2.17.7 x64\include\embree2). In addition, copy the libraries from C:\vcpkg\installed\x64-windows\lib into this embree2 folder (if you installed from the .msi, this will be the folder C:\Program Files\Intel\Embree v2.17.7 x64\lib\). You only need the (static) libraries
embree.lib
tbb.lib
tbbmalloc.lib

and dynamic libraries

embree.dll
tbb.dll
tbbmalloc.dll
  1. Add the following line to the top of every *.pyx and *.pxd file in pyembree:
# distutils: language=c++

and change every relative import of an rtcore module to an absolute import in each file. For example, change:

cimport rtcore as rtc

to

cimport pyembree.rtcore as rtc
  1. Create the following MANIFEST.in file in the top-level pyembree folder:
include *.h *.pxd *.dll
recursive-include embree2 *.h *.isph
recursive-include embree2/lib *.lib
  1. At the end, your folder should look like this:
<virtual environment folder>\Lib\site-packages\pyembree
│   .authors.yml
│   .gitignore
│   .mailmap
│   .travis-install.sh
│   .travis.yml
│   AUTHORS
│   CHANGELOG.rst
│   LICENSE
│   MANIFEST.in
│   README.rst
│   rever.xsh
│   setup.py
│   
├───examples
│       attenuate.py
│       intersection.py
│       
├───news
│       TEMPLATE.rst
│       
├───pyembree
│   │   embree.dll
│   │   mesh_construction.h
│   │   mesh_construction.pyx
│   │   rtcore.pxd
│   │   rtcore.pyx
│   │   rtcore_geometry.pxd
│   │   rtcore_geometry_user.pxd
│   │   rtcore_ray.pxd
│   │   rtcore_scene.pxd
│   │   rtcore_scene.pyx
│   │   tbb.dll
│   │   tbbmalloc.dll
│   │   triangles.pyx
│   │   __init__.pxd
│   │   __init__.py
│   │
│   └───embree2
│       │   rtcore.h
│       │   rtcore.isph
│       │   rtcore_builder.h
│       │   rtcore_geometry.h
│       │   rtcore_geometry.isph
│       │   rtcore_geometry_user.h
│       │   rtcore_geometry_user.isph
│       │   rtcore_ray.h
│       │   rtcore_ray.isph
│       │   rtcore_scene.h
│       │   rtcore_scene.isph
│       │   rtcore_version.h
│       │
│       └───lib
│               embree.lib
│               tbb.lib
│               tbbmalloc.lib
│
├───recipes
│   └───pyembree
│           build.sh
│           meta.yaml
│
└───tests
        test_intersection.py
  1. Build pyembree by running the following from the top-level pyembree folder (TIP: To help debug errors, pipe stderr to a local file errors.txt that you can look at):
py setup.py build_ext -i 2> errors.txt
  1. Test that everything is working correctly by opening a python terminal in your virtual environment and running
>>> import pyembree
>>> from pyembree import rtcore_scene
>>>

If you get no errors, then pyembree has been installed correctly. You can also create a .egg and install the library with

py setup.py install

The setup.py we created will copy the dlls and cpp files to the correct folder.

  1. As a last optional step, create a main.py in your virtual environment and copy and paste the code from Project to Finite Plane demo. If it runs without errors in approximately 1-2 seconds, then pyembree is working properly with trimesh for vectorized ray tracing (NOTE: trimesh will work without pyembree, but it will be very slow. It should not take a long time to run this example code if pyembree is properly installed. trimesh resorts back to trimesh.ray.ray_triangle.RayMeshIntersector if it cannot import pyembree (see pyembree ray casting acceleration #875), so you won't see any errors; just slow running code):
import numpy as np
from pykdtree.kdtree import KDTree

import pyvista as pv
from pyvista import examples

# Load data
data = examples.load_random_hills()
data.translate((10, 10, 10))

# Create triangular plane (vertices [10, 0, 0], [0, 10, 0], [0, 0, 10])
size = 10
vertices = np.array([[size, 0, 0], [0, size, 0], [0, 0, size]])
face = np.array([3, 0, 1, 2])

planes = pv.PolyData(vertices, face)

# Subdivide plane so we have multiple points to project to
planes = planes.subdivide(8)

# Get origins and normals
origins = planes.cell_centers().points
normals = planes.compute_normals(cell_normals=True, point_normals=False)["Normals"]

# Vectorized Ray trace
points, pt_inds, cell_inds = data.multi_ray_trace(
    origins, normals
)  # Must have rtree, trimesh, and pyembree installed

# Filter based on distance threshold, if desired (mimics VTK ray_trace behavior)
# threshold = 10  # Some threshold distance
# distances = np.linalg.norm(origins[inds] - points, ord=2, axis=1)
# inds = inds[distances <= threshold]

tree = KDTree(data.points.astype(np.double))
_, data_inds = tree.query(points)

elevations = data.point_arrays["Elevation"][data_inds]

# Mask points on planes
planes.cell_arrays["Elevation"] = np.zeros((planes.n_cells,))
planes.cell_arrays["Elevation"][pt_inds] = elevations
planes.set_active_scalars("Elevation")  # Probably not necessary, but just in case

# Create axes
axis_length = 20
tip_length = 0.25 / axis_length * 3
tip_radius = 0.1 / axis_length * 3
shaft_radius = 0.05 / axis_length * 3
x_axis = pv.Arrow(
    direction=(axis_length, 0, 0),
    tip_length=tip_length,
    tip_radius=tip_radius,
    shaft_radius=shaft_radius,
    scale="auto",
)
y_axis = pv.Arrow(
    direction=(0, axis_length, 0),
    tip_length=tip_length,
    tip_radius=tip_radius,
    shaft_radius=shaft_radius,
    scale="auto",
)
z_axis = pv.Arrow(
    direction=(0, 0, axis_length),
    tip_length=tip_length,
    tip_radius=tip_radius,
    shaft_radius=shaft_radius,
    scale="auto",
)
x_label = pv.PolyData([axis_length, 0, 0])
y_label = pv.PolyData([0, axis_length, 0])
z_label = pv.PolyData([0, 0, axis_length])
x_label.point_arrays["label"] = [
    "x",
]
y_label.point_arrays["label"] = [
    "y",
]
z_label.point_arrays["label"] = [
    "z",
]

# Plot results
p = pv.Plotter()
p.add_mesh(x_axis, color="r")
p.add_point_labels(x_label, "label", show_points=False, font_size=24)
p.add_mesh(y_axis, color="r")
p.add_point_labels(y_label, "label", show_points=False, font_size=24)
p.add_mesh(z_axis, color="r")
p.add_point_labels(z_label, "label", show_points=False, font_size=24)
p.add_mesh(data)
p.add_mesh(planes)
p.show()
adamgranthendry commented 3 years ago

Closing as the solution has been found. 🥇

akaszynski commented 3 years ago

This will be super helpful for anyone wanting to install pyembree outside of conda. Thanks for posting this, and I'll upload these docs to our extras section.

adamgranthendry commented 3 years ago

@akaszynski Thank you very much! I went through the steps again to make sure I had documented everything correctly. I made some minor changes (e.g. I forgot to include that the rtcore modules must be imported using absolute imports). I also updated the procedure so the package will work without needing to run install (i.e. py setup.py install) and made it so that the appropriate cpp and dll files are copied over to the install directory should one want to run py setup.py install after building.

adamgranthendry commented 3 years ago

@akaszynski One last thing: The "Project to Finite Plane" demo, though on the GitHub docs, doesn't show up on the pyvista website. Can you help make sure it can be seen on the website?

akaszynski commented 3 years ago

It's here on the development docs: https://dev.pyvista.org/user-guide/optional_features.html#project-to-finite-plane

We're still working on the next release, should be relatively soon.

adamgranthendry commented 3 years ago

Excellent! Thanks @akaszynski !

adamgranthendry commented 3 years ago

@akaszynski As a recommendation, would you be able to test this install process for Mac/Linux? I do not have a Mac, so I wouldn't be able to confirm. However, I believe if you replace *.dll with *.so in the above that the process should work for Mac/Linux as well. User @ead on my SO post linked me to two helpful SO posts in this area: here and here.

It would be nice if it worked for all users, not just those on Windows.

venugovh commented 3 years ago

@adamgranthendry Hello! I am following your step-by-step instructions to install pyembree without conda. Just a precursor, I am fairly new to programming, so I am following the instructions to the dot. Currently, I am stuck at step number 12.

py setup.py build_ext -i 2> errors.txt

I am getting 

Traceback (most recent call last):
  File "C:\Users\CGDM12\Documents\ray_trace_proj\venv\Lib\site-packages\pyembree\setup.py", line 3, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

I am not sure where I went wrong. I don't know if this is relevant, there are a couple of things I am not sure about.

  1. I am running this command py setup.py build_ext -i 2> errors.txt in a command prompt opened in the top level of the pyembree folder.
  2. The .dll files named embree.dll, tbb.dll, and tbbmalloc.dll were copied from C:\Program Files\Intel\Embree v2.17.7 x64\bin rather than C:\Program Files\Intel\Embree v2.17.7 x64\lib as per your instructions. The \lib folder did not have .dll files.

Apart from that my folder structure looks exactly like you showed in the instructions.

Could you please help me figure this out? Thank you very much!

adam-grant-hendry commented 3 years ago

@Vysakh5391 Glad to hear you're trying it out!

  1. It looks like you don't have numpy installed, which is a prerequisite. You can install this with pip from the command line like so
> py -m pip install numpy

that should solve your problem.

  1. Yes, the dlls are in the bin folder. If I wrote they were in lib, that was a mistake and I will fix the instructions.

Please reply back and let me know you got it working!

venugovh commented 3 years ago

@adam-grant-hendry Thank you so much for responding and confirming this! I was able to install it without any errors. The code in https://docs.pyvista.org/user-guide/optional_features.html#vectorised-ray-tracing seems to be working fine for me. I was even able to run the test_intersection script in C:\...\venv\Lib\site-packages\pyembree\tests and two scripts in C:\...\venv\Lib\site-packages\pyembree\examples without any errors.

Surprisingly, the 'https://dev.pyvista.org/user-guide/optional_features.html#project-to-finite-plane' is taking more than the 1-2 seconds as you mentioned in the instructions.

I saw there was a mention of generating an installation of pyembree using pip. Is that ready or is it WIP?

Thanks!

adamgranthendry commented 3 years ago

@Vysakh5391

  1. Be sure that step 13 works properly:
>>> import pyembree
>>> from pyembree import rtcore_scene
>>>

If it works, then the code may be slower than 1-2 seconds depending on how powerful your hardware is, but it will be much faster than the non-pyembree code.

  1. Sadly, the creator of pyembree is not interested in making a pip installer, which is why this workaround is necessary. If you would like him to create a pip installer, I suggest you make a feature request on his GitHub page and have as many people as you can give it a thumbs up to push him to develop one. I wish it weren't the case, but sadly it is.
venugovh commented 3 years ago

Sure, I will make a feature request on the Github page.

Just to be clear, I am not getting any errors when I run the two commands from step 13. Does that mean my pyembree installation is ok?

I am using a workstation with 64 GB RAM on Intel Xeon Processor. For 6 million points on a STL file, the ray tracing is taking around 7.5 minutes. Not that it is bad, but do you think that is ok? I am afraid if I have made any error installing pymebree and it is running the ray tracing without embree.

Could you please let me know? Thank you very much!

adamgranthendry commented 3 years ago

@Vysakh5391 I'm unsure of benchmarks. The only one I have currently is for the example I wrote projecting the Hills Data to a Finite Plane. I have a similar system (192 GB RAM, dual Xeon Processors (24 cores each)).

If you run step 13 and can import without getting errors, then yes, that means it is installed correctly.

If you try the Hills Data example, it should only take a couple seconds on your system. Compared to with it turned off, for me, the example takes about 30 minutes.

venugovh commented 3 years ago

Surprisingly, it is taking more than 1-2 seconds even though I am not getting any errors. It has been 10 minutes and the code is still running. I believe my pyembree installation is not accurate.

adamgranthendry commented 3 years ago

@Vysakh5391 Yes, the installation is incorrect then. The best I can tell you is to go through all the steps in order again and keep trying.