Houdini in Windows 10

[Idclip]

I'm trying to build openvdb against Houdini in Windows 10, using vcpkg (and then Visual Studio 2019).

I've run into the same error, which I haven't been able to resolve:

CMake Error at C:/Program Files/CMake/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find IlmBase (missing: IlmBase_LIB_COMPONENTS Half) (found
  suitable version "2.2", minimum required is "2.2")
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindIlmBase.cmake:285 (find_package_handle_standard_args)
  C:/vcpkg/scripts/buildsystems/vcpkg.cmake:263 (_find_package)
  openvdb/CMakeLists.txt:61 (find_package)

Firstly, it seems odd that it says "found suitable version" when it's also saying "could not find" it.

Also, inside C:\vcpkg\installed\x64-windows\lib I can see Half:

Half-2_3.lib
Half-s2_3_s.lib

So, it seems that it's there but CMake just can't find it. I get the same error even if I explicitly point to this lib directory with my CMake command:

-DILMBASE_ROOT=”C:/vcpkg/installed/x64-windows/lib”

I've been making a guide document for myself as I go, which I've attached to show the detail of what I've done so far. OpenVDB installation using vcpkg - github query.pdf

In summary, my steps have been:

.\vcpkg install --triplet x64-windows ilmbase blosc boost zlib openexr tbb boost-python cppunit 
.\vcpkg integrate install

If I run vcpkg list, it shows that ilmbase is installed:

ilmbase:x64-windows              2.3.0            empty package, linking to newer one

I've cloned the openvdb git:

git clone git@github.com:AcademySoftwareFoundation/openvdb.git

My CMake command in Windows Powershell is:

cmake `
-S"C:/vcpkg/openvdb" `
-G"Visual Studio 16 2019" `
-DOPENVDB_ABI_VERSION_NUMBER=5 `
-DCMAKE_TOOLCHAIN_FILE="C:/vcpkg/scripts/buildsystems/vcpkg.cmake" `
-DHoudini_ROOT="C:/Program Files/Side Effects Software/Houdini 17.5.425" `
-DUSE_HOUDINI=ON `
-DOPENVDB_BUILD_HOUDINI_PLUGIN=ON `
-DUSE_DEFAULT_HOUDINI_INSTALL=ON `
-DOPENVDB_BUILD_PYTHON_MODULE=ON `
-DOPENVDB_BUILD_UNITTESTS=ON `
-DOPENEXR_ROOT="C:/vcpkg/installed/x64-windows" `
-DILMBASE_ROOT=”C:/vcpkg/installed/x64-windows/lib” `
-DTBB_ROOT="C:/vcpkg/installed/x64-windows" `
-DUSE_BLOSC=OFF `
-DCPPUNIT_ROOT="C:/vcpkg/installed/x64-windows" `
-DBOOST_ROOT="C:/vcpkg/installed/x64-windows" `
-DBOOST-PYTHON_ROOT="C:/vcpkg/installed/x64-windows" `
-DBOOST_LIBRARYDIR="C:/vcpkg/installed/x64-windows" `
-DZLIB_ROOT="C:/vcpkg/installed/x64-windows" `
..

I'm wondering if the error is because CMake is expecting to find Half in a discrete ilmbase libary but Half is now actually built inside OpenEXR, which the empty ilmbase package should be pointing to but isn't.

I'd really appreciate some guidance on resolving this!

Originally posted by @ianww in https://github.com/AcademySoftwareFoundation/openvdb/issues/429#issuecomment-571989625

[ianww]

That is correct. Perhaps I missed this in the discussion above, but what is the reason for requiring Houdini 17.5 instead of 18.0?

It's a bit lost in the mists of time but I think it was because early on I was getting CMake build errors (which are now fixed) with 18.0.287 and I thought I should revert to the 'official' supported version, which I took to be 17.5: https://www.openvdb.org/documentation/doxygen/build.html , and I've been continuing with 17.5 because I didn't want to risk undoing all the progress we've made.

I'm very happy to be able to move onto 18.0 when the 18.0.365 daily build appears!

[e4lam]

@ianww Here's my attempt at a set of instructions to build everything from scratch using the latest vcpkg, openvdb, and Houdini. I've repeated some of the same steps from before so that these instructions can be stand-alone. Apply these as appropriate to your setup.

Setup vcpkg

I'm going to assume for the rest of these instructions that you're working with a git clone of vcpkg as that's what I'm currently doing and inside a cygwin bash shell. This is just to make it easier for me to write these instructions.

If starting from scratch:

cd $HOME                                     # or wherever you want to put vcpkg
git clone git@github.com:microsoft/vcpkg.git # creates vcpkg subdirectory
cd vcpkg
chmod +x bootstrap-vcpkg.bat
cmd /c bootstrap-vcpkg.bat

If you've already got a git clone and running vcpkg:

cd $HOME/vcpkg
git stash # save and undo all modifications

Set up vcpkg triplet file

In vcpkg/triplets add a new triplet file called x64-windows-houdini.cmake but you if had made your own already you can reuse the same triplet file name to avoid rebuilding the world. The contents of this file is:

# x64-windows-houdini.cmake
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)
set(VCPKG_BUILD_TYPE release)
set(VCPKG_HOUDINI_VERSION "18.0.365")

Notice that this is just the same as x64-windows.cmake except that I've added the last two lines. Change the VCPKG_HOUDINI_VERSION value as appropriate. This is used by the changes to the openvdb package.

Set up boost-python

Change boost-python to build against python2 instead of python3. In vcpkg/ports/boost-python, search and replace python3 with python2 in both the CONTROL and portfile.cmake files.

Instead of doing this manually, you can just apply this patch: vcpkg_ports_boost-python.patch.txt Sample steps:

cd $HOME/vcpkg
patch -p1 < /path/to/vcpkg_ports_boost-python.patch.txt

Set up openvdb

Apply this (new) patch for openvdb: vcpkg_ports_openvdb.patch.txt

cd $HOME/vcpkg
patch -p1 < /path/to/vcpkg_ports_openvdb.patch.txt

Build

cd $HOME/vcpkg
./vcpkg install openvdb[python,houdini]:x64-windows-houdini

Install

cd $HOME
mkdir -p houdini18.0/bin houdini18.0/dso houdini18.0/python2.7libs
cp vcpkg/packages/openexr_x64-windows-houdini/bin/Half-2_3.dll houdini18.0/bin
cp vcpkg/packages/openvdb_x64-windows-houdini/bin/openvdb.dll houdini18.0/bin
cp vcpkg/packages/openvdb_x64-windows-houdini/bin/openvdb_houdini.dll houdini18.0/bin
cp vcpkg/packages/openvdb_x64-windows-houdini/dso/*.dll houdini18.0/dso
cp vcpkg/packages/openvdb_x64-windows-houdini/python2.7libs/pyopenvdb.dll houdini18.0/python2.7libs/pyopenvdb.pyd
cp vcpkg/packages/boost-python_x64-windows-houdini/bin/boost_python27-vc142-mt-x64-1_72.dll houdini18.0/bin

Run

cd "C:/Program Files/Side Effects Software/Houdini 18.0.365"
source houdini_setup
export HOUDINI_DSO_ERROR=1
export PATH="$PATH:$HOME/houdini18.0/bin"
houdini

[ianww]

Thanks @e4lam, I'll give it a go as soon as the 18.0.365 daily build becomes available.

Some questions:

1. Do I need Python2.7 separately installed? Previously, I've been pointing CMake to Houdini's python2.7.exe. If I don't install it separately, will CMake find Houdini's copy?

2. Do I need Python3.6 separately installed? I recall seeing somewhere that some of the dependencies need 3.x, even though Houdini uses 2.7. (I'll install 3.6 anyway but I wanted to check in case other readers might not have it.)

3. After doing a fresh install of vcpkg I looked at the C:\vcpkg\ports\openvdb directory and noticed that there are 3 patches already there:

Your 5 patches were added in, to make 8:

I presume they'll all work happily together?

Also - FYI at the end of your vcpkg_ports_openvdb.patch.txt it's saying:

+set(VCPKG_HOUDINI_VERSION "18.0.363")

In my copy I've changed this to:

+set(VCPKG_HOUDINI_VERSION "18.0.365").

[ianww]

FYI - In the vcpkg build and install, I got a build error with python2, due to a missing debug lib:

CMake Error at ports/python2/portfile.cmake:67 (file):
  file COPY cannot find
  "C:/vcpkg/buildtrees/python2/src/Python-2-2a4970fdfb/PCBuild/amd64/python27_d.lib":

I got around this by commenting out the debug lib and bin file copies in python2's portfile.cmake:

#file(COPY ${SOURCE_PATH}/PCBuild/${OUT_DIR}/python${PYTHON_VERSION_MAJOR}${PYTHON_VERSION_MINOR}_d.lib DESTINATION ${CURRENT_PACKAGES_DIR}/debug/lib)
#file(COPY ${SOURCE_PATH}/PCBuild/${OUT_DIR}/python${PYTHON_VERSION_MAJOR}${PYTHON_VERSION_MINOR}_d.dll DESTINATION ${CURRENT_PACKAGES_DIR}/debug/bin)

After that fix, I have 96 out of 97 packages successfully installed.

Just waiting on the Houdini 18.0.365 release to run the openvdb package install.

[e4lam]

@ianww the patch is relative to a clean vcpkg so you need to first remove previous changes. If you're building with the modified triplet, then it should not build any debug libraries at all. Have you tried updating your vcpkg yet?

[ianww]

@e4lam Yeah, the triplet says release only and I was surprised when it looked for debug too.

I had done a complete new install of vcpkg, and also ran vcpkg update (it said no packages need updating).

It's not a problem for me - my hack got around it - but others may run into it.

Looking at the python2 portfile.cmake, there's no IF statement to test for BUILD_TYPE. The way I read it (perhaps wrongly) is that it will always try the file copy for debug.

If that's not the answer, there are 7 patch files in the python2 port:

and maybe one of them is doing something to override the triplet?

[e4lam]

@ianww Hmm, you're right I don't know why I didn't run into it as I'm pretty sure it also had to build 97 packages when I added the new triplet and built with it. I'd need to try again.

Sorry, I missed your earlier questions:

1. Do I need Python2.7 separately installed? Previously, I've been pointing CMake to Houdini's python2.7.exe.  If I don't install it separately, will CMake find Houdini's copy?

No, we don't need any python interpreter at all. This is only needed when building the unit tests. In openvdb/python/CMakeLists.txt, I've patched away this requirement (see #621).

2. Do I need Python3.6 separately installed?  I recall seeing somewhere that some of the dependencies need 3.x, even though Houdini uses 2.7.  (I'll install 3.6 anyway but I wanted to check in case other readers might not have it.)

Also, no. AFAIR, none of openvdb's dependencies require python other than boost-python.

3. After doing a fresh install of vcpkg I looked at the `C:\vcpkg\ports\openvdb` directory and noticed that there are 3 patches already there:

Your 5 patches were added in, to make 8:

I presume they'll all work happily together?

That is right. Note that the original patches in ports/openvdb have also been modified.

Also - FYI at the end of your vcpkg_ports_openvdb.patch.txt it's saying:

+set(VCPKG_HOUDINI_VERSION "18.0.363")

That's my bad. When I created the diff I accidentally included my triplet file which I didn't intend on doing. I've updated the original comment to have a corrected vcpkg_ports_openvdb.patch.txt.

[ianww]

@e4lam Got a couple of errors.

The first one may be due to what I've done previously. For some reason the build preferentially chose my Houdini 17.5.425 installation instead of 18.0.365, and then failed. Unfortunately, the error log file has since been overwritten, so I can't copy the message into here. For previous try-it-and-see reasons I had installed 17.5 at the root C:/Houdini17.5.425 level. Perhaps CMake scans from top down and found this before it got into C:/Program Files? Anyway, I deleted 17.5.425 and the error went.

After fixing that and rebuilding, the second error was caused by Unable to find boost_python36, boost_python3 or boost_python:

CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:72 (message):
    Command failed: ninja -v
    Working Directory: C:/vcpkg/buildtrees/openvdb/x64-windows-houdini-rel/vcpkg-parallel-configure
    Error code: 1
    See logs for more information:
      C:\vcpkg\buildtrees\openvdb\config-x64-windows-houdini-out.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_cmake.cmake:295 (vcpkg_execute_required_process)
  ports/openvdb/portfile.cmake:54 (vcpkg_configure_cmake)
  scripts/ports.cmake:94 (include)

Error log

``` CMake Warning at C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1147 (message): New Boost version may have incorrect or missing dependencies and imported targets Call Stack (most recent call first): C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1269 (_Boost_COMPONENT_DEPENDENCIES) C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1907 (_Boost_MISSING_DEPENDENCIES) C:/vcpkg/scripts/buildsystems/vcpkg.cmake:249 (_find_package) openvdb/CMakeLists.txt:113 (find_package) CMake Warning at C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1147 (message): New Boost version may have incorrect or missing dependencies and imported targets Call Stack (most recent call first): C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1269 (_Boost_COMPONENT_DEPENDENCIES) C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1907 (_Boost_MISSING_DEPENDENCIES) C:/vcpkg/scripts/buildsystems/vcpkg.cmake:249 (_find_package) openvdb/CMakeLists.txt:113 (find_package) -- Found Boost: C:/vcpkg/installed/x64-windows-houdini/include (found suitable version "1.72.0", minimum required is "1.61") found components: iostreams system regex -- ---------------------------------------------------- -- ------------ Configuring OpenVDBPython ------------- -- ---------------------------------------------------- -- Found Python: optimized;C:/Python36/libs/python36.lib;debug;C:/Python36/libs/python36_d.lib (found version "3.6") found components: Development CMake Warning at C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1147 (message): New Boost version may have incorrect or missing dependencies and imported targets Call Stack (most recent call first): C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1269 (_Boost_COMPONENT_DEPENDENCIES) C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1907 (_Boost_MISSING_DEPENDENCIES) C:/vcpkg/scripts/buildsystems/vcpkg.cmake:249 (_find_package) openvdb/python/CMakeLists.txt:152 (find_package) CMake Warning at C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1147 (message): New Boost version may have incorrect or missing dependencies and imported targets Call Stack (most recent call first): C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1269 (_Boost_COMPONENT_DEPENDENCIES) C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1907 (_Boost_MISSING_DEPENDENCIES) C:/vcpkg/scripts/buildsystems/vcpkg.cmake:249 (_find_package) openvdb/python/CMakeLists.txt:155 (find_package) CMake Warning at C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1147 (message): New Boost version may have incorrect or missing dependencies and imported targets Call Stack (most recent call first): C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1269 (_Boost_COMPONENT_DEPENDENCIES) C:/Program Files/CMake/share/cmake-3.16/Modules/FindBoost.cmake:1907 (_Boost_MISSING_DEPENDENCIES) C:/vcpkg/scripts/buildsystems/vcpkg.cmake:249 (_find_package) openvdb/python/CMakeLists.txt:158 (find_package) CMake Error at openvdb/python/CMakeLists.txt:174 (message): Unable to find boost_python36, boost_python3 or boost_python. -- Configuring incomplete, errors occurred! See also "C:/vcpkg/buildtrees/openvdb/x64-windows-houdini-rel/CMakeFiles/CMakeOutput.log". ninja: build stopped: subcommand failed. ```

That's a familiar (fixed) error from my old method early build attempts.

I can't find openvdb/python/CMakeLists.txt to look for the error cause more closely. With my old method, I had a C:/vcpkg/openvdb directory from my openvdb git clone but with this new method there's no such directory to look in.

In the absence of C:/vcpkg/openvdb, I've looked for it in C:\vcpkg\buildtrees\openvdb\x64-windows-houdini-rel\openvdb\python\CMakeFiles but that directory is empty.

However, in C:\vcpkg\buildtrees\openvdb\x64-windows-houdini-rel\CMakeCache.txt there is:

//Boost python36 library (debug)
Boost_PYTHON36_LIBRARY_DEBUG:FILEPATH=Boost_PYTHON36_LIBRARY_DEBUG-NOTFOUND

//Boost python36 library (release)
Boost_PYTHON36_LIBRARY_RELEASE:FILEPATH=Boost_PYTHON36_LIBRARY_RELEASE-NOTFOUND

//Boost python3 library (debug)
Boost_PYTHON3_LIBRARY_DEBUG:FILEPATH=Boost_PYTHON3_LIBRARY_DEBUG-NOTFOUND

//Boost python3 library (release)
Boost_PYTHON3_LIBRARY_RELEASE:FILEPATH=Boost_PYTHON3_LIBRARY_RELEASE-NOTFOUND

//Boost python library (debug)
Boost_PYTHON_LIBRARY_DEBUG:FILEPATH=Boost_PYTHON_LIBRARY_DEBUG-NOTFOUND

//Boost python library (release)
Boost_PYTHON_LIBRARY_RELEASE:FILEPATH=Boost_PYTHON_LIBRARY_RELEASE-NOTFOUND

[e4lam]

@ianww Is C:\Python36 somewhere in your PATH? We're running up into the same problem as always on your system. The openvdb source when using this method is unpacked into vcpkg/buildtrees/openvdb/src/... Check the time stamps inside there to see the last one used. If in doubt, you can first remove all of vcpkg/buildtrees/openvdb and try again:

cd $HOME/vcpkg
./vcpkg install openvdb[python,houdini]:x64-windows-houdini

But I suspect that we need to first make sure that vcpkg finds the python2 installation within itself first. As before, adding -DPython_ROOT_DIR=/SOME/VCPKG/PATH to the cmake options inside vcpkg/ports/openvdb/portfile.cmake might help if you try some reasonable looking paths. eg. path to vcpkg/installed/x64-windows-houdini.

[ianww]

I couldn't make using Python_ROOT_DIR help.

In C:\vcpkg\buildtrees\openvdb\src\a0d51d97d8-d297da0650\openvdb\python\CMakeLists.txt (at line 47) I found this comment next to the find python function, so it seems it's a recognised issue:

# Small function which mimics basic output (bar components) of
# FindPackageHandleStandardArgs. This is required as we want to ensure
# the minimum python version is MINIMUM_PYTHON_VERSION - however this cannot
# be provided to find_package(Python) with differing major versions. e.g.
# calls to find_package(Python 2.7) fails if python3 is found on the system.
function(OPENVDB_CHECK_PYTHON_VERSION _PY_VERSION _PY_PATH)
  if(_PY_VERSION VERSION_LESS MINIMUM_PYTHON_VERSION)
    message(FATAL_ERROR "Could NOT find Python: Found unsuitable version \"${_PY_VERSION}\""
      "but required is at least \"${MINIMUM_PYTHON_VERSION}\" (found ${_PY_PATH})"
    )
  else()
    message(STATUS "Found Python: ${_PY_PATH}) (found suitable version \"${_PY_VERSION}\", "
      "minimum required is \"${MINIMUM_PYTHON_VERSION}\")"
    )
  endif()
endfunction()

I therefore uninstalled python 3.6 from my system, and the CMake build worked!

Now, onto the next step...which I think is to copy the dso folder (and any others?) from:

C:\vcpkg\installed\x64-windows-houdini

to:

C:\Users\Owner\Documents\houdini18.0

and launch Houdini 18.0.365.

Is that right?

Trouble is, I can't try that just at the moment as I've run into a licensing issue with that version of Houdini when I try to open it. I have to sort that before I can try.

[e4lam]

I therefore uninstalled python 3.6 from my system, and the CMake build worked!

You couldn't find Python 3.6 listed anywhere in either the system value of PATH nor the user value of PATH?

Now, onto the next step...which I think is to copy the dso folder (and any others?) from: C:\vcpkg\installed\x64-windows-houdini to: C:\Users\Owner\Documents\houdini18.0

and launch Houdini 18.0.365.

Is that right?

Yes, but they are some details in what the commands that you need to pay attention.

• pyopenvdb.dll needs to be renamed to pyopenvdb.pyd in the destination
• you need to copy a few of the DLL's into a "bin" directory that is then added to the PATH
• set HOUDINI_DSO_ERROR to at least 1 so that we can tell if things fail to load

Trouble is, I can't try that just at the moment as I've run into a licensing issue with that version of Houdini when I try to open it. I have to sort that before I can try.

Odd, but remember that you can always just redeem a free Apprentice license just to test things out if need be.

[ianww]

I've got Houdini 18.0.365 working now - it was my error - when I removed earlier installations I inadvertently removed the licence server too.

You couldn't find Python 3.6 listed anywhere in either the system value of PATH nor the user value of PATH?

Ah, sorry, forgot to look - yes, Python36 is (now was) in my system PATH.

you need to copy a few of the DLL's into a "bin" directory that is then added to the PATH

Not sure what you mean by this, ie:

1. Which of the dll's?

2. Where would I place the bin directory? Inside the dso directory: C:\Users\Owner\Documents\houdini18.0\dso ?

3. Which PATH? Windows System PATH or Houdini DSO PATH?

[e4lam]

you need to copy a few of the DLL's into a "bin" directory that is then added to the PATH

Not sure what you mean by this, ie:

1. Which of the dll's?

See the cp (copy) commands in the Install section in the comment.

2. Where would I place the bin directory? Inside the dso directory: C:\Users\Owner\Documents\houdini18.0\dso ?

Those commands I wrote put them in houdini18.0/bin just so that everything is in one place. However, it's not strictly necessary.

3. Which PATH? Windows System PATH or Houdini DSO PATH?

Windows PATH. Probably SYSTEM if you want any user logged into the computer to use it.

[e4lam]

PS. I take that back. Modify the Windows USER PATH instead because these libraries are only going into a user-specific location.

[ianww]

@e4lam @Idclip @danrbailey

SUCCESS!!! IT'S WORKED!!!

All custom dso's loaded with no errors, and I can see them as tools in the Edit Shelf menu, eg:

Label: VDB Vector Split NAME: sop_DW_OpenVDBVectorSplit

etc

And I can import pyopenvdb into the python shell.

Thanks for your fantastic help @e4lam over a long (3 weeks!) and bumpy road. I've learnt heaps. You've always been ultra-responsive and I've often wondered whether you ever sleep!

Your consolidated instructions a few posts back (here) plus the comments that follows them should be enough for anyone starting from scratch to succeed too.

FYI I've been keeping detailed notes for myself as we've gone along and I'll polish them up into a pdf, which I'll post back here in the next few days as a guide too.

Thanks to you also @e4lam and @danrbailey!

[e4lam]

That is great to hear, @ianww ! I admire your persistence. :)

I'll try to see if I can upstream these patches in a way so that we can more easily keep the openvdb vcpkg up to date in the coming weeks. That way, new people coming in won't have to do so much work.

[ianww]

@e4lam I've attached my pdf guide to building OpenVDB against Houdini, based on what we've done above. I've added more details to cover all the preliminary steps etc, so that hopefully a user can start from (almost) zero knowledge with it. It's the sort of guide that I wish had been available when I started!

If you find any mistakes, please let me know and I'll do an update.

Guide to installing OpenVDB into Houdini in Windows ver 1.1.pdf

[ianww]

@e4lam Another query if I may.

As well as running pyopenvdb inside Houdini, I'd like to be able to import the pyopenvdb.dll (which we renamed pyopenvdb.pyd) as a module into my Python interpreter (PyCharm).

I've tried simply copying the pyopenvdb.pyd file into my PyCharm environment's DLLs folder (where it sits with lots of other .pyd files) but if I try import pyopenvdb I get a 'no such module' error.

Googling says that c++ objects need to be put inside a python wrapper with boost.python to be able to import them into a python interpreter. Before I try to learn how to do this, can I please check: is pyopenvdb.pyd raw c++ or has it already been wrapped in python in order for Houdini to use it (in which case I must be doing something else wrong)?

[e4lam]

@ianww The .pyd is already wrapped with boost.python. Offhand, it's supposed work if you put it somewhere in PYTHONPATH so that it shows up in PyCharm's python interpreter to load it, and that PATH has the directories to all the libraries that pyopenvdb.pyd is built against. A problem though is that if PyCharm's python interpreter was built with a compiler version older than Visual Studio 2015, then it likely won't work. The Windows Python 2 binaries that you download from the official site are built with Visual Studio 2008 or something old like that making it incompatible.

[e4lam]

PS. A quick google revealed this plugin which may work by telling PyCharm to use Houdini's python interpreter (I think, YMMV)

https://plugins.jetbrains.com/plugin/12877-houdini-python-support

[ianww]

Thanks @e4lam - no luck so far. I'll keep on trying a few things but I suspect that it is an incompatibility. I can't seem to make Pycharm see pyopenvdb and conversely I can't make Houdini see yt without it throwing R6034 errors.

I may just need to work using Houdini for pyopenvdb and PyCharm for yt separately.

[e4lam]

Have you talked with the ytini people yet? Given that the python modules and interpreter all need to built with the same version of Visual Studio on Windows, you might be able get the pip instructions working by using hython instead of python (see the ytni.com installation instructions). However, I don't know if the hython python distro has all the packages that yt requires. On Linux/macOS, Houdini is able to use the system's python packages with hython. So if you install yt on Linux into the system python, it's much easier to just import it from within Houdini.

[ianww]

Ah, interesting, thanks. I'll look into that (I have been talking to Jill at ytini but not yet about this per se).

Another (possibly silly) thought I had was making a fresh python virtual environment with conda and replacing the python interpreter files in that with a copy and paste of Houdini's python interpreter files, and then installing yt again into that env.

Could some thing like that work or is it too simplistic? If it might work, what python files would I need to copy?

eg. python2.7.exe, python27.dll, python27.lib, ...

[e4lam]

Another (possibly silly) thought I had was making a fresh python virtual environment with conda and replacing the python interpreter files in that with a copy and paste of Houdini's python interpreter files, and then installing yt again into that env.

I don't think that will work because all of yt's dependencies need to be built with the same compiler even though yt itself doesn't seem to have any native code. The big one will be numpy, which at least Houdini ships with. Note that hython is just the python interpreter with Houdini's hou module imported and some additional command line options.

[VfxTDwiki]

@ianww I hopefuly found your documentation to build the openvdb_houdini and it's pretty useful! Great job ! I am contacting you because I have some issues with the diff file. I'm losing some indent when I copy/paste the text from the pdf file. It seams that the patch file above is not the last one and I was wondering if you can share with us the final patch file here to fix this issue. I have already some comments to improve your documentation, if you want them. Thanks in advance

[ianww]

Hi Nicholas – I haven’t worked with the Windows version for a while now. I ended up going over to Linux because of further issues (see below).

I expect that the patch file that I used is now well and truly redundant because the OpenVDB team that was helping me were incorporating the findings in the daily builds, so those patches should now be incorporated. If you haven’t done so already, I recommend that you download the latest daily build. If that doesn’t help, send me a direct email and I’ll send you a Word version of my guide, which should solve the copy and paste issue.

FYI, the reasons that I went over to Linux were as follows:

The OpenVDB build was complicated and frustrating because of the need to compile it and its dependencies to match the msvc compiler used for Windows Houdini’s native python – unlike with UNIX/linux or Mac, in Windows python 2.7 comes internal to Houdini, so there’s no choice – the compilers have to match.

Although I got there in the end (after many tens of hours) with the help of the OpenVDB team, I then moved on to what I thought would be nice and simple – pip installing yt, a package that I wanted to use for astronomical visualisations.

The first problem that arose was a fnv_hash DLL installation error with yt on Windows. Although I found a way around that (by importing an fnv_hash for python module), it turns out that fnv_hash is just the tip of a binary incompatibility iceberg. I found binary incompatibilities with other dependencies too, notably numpy and matplotlib – in fact, any that rely on compiled pyd files. I found myself facing the prospect of having to build and compile numerous dependencies from scratch, and maybe their sub-dependencies as well.

I’d finally had enough! I therefore surrendered and moved across to Linux, something I’d been contemplating anyway.

Looking back, I wish I’d done this from the start but of course I was not to know until I’d tried, and as I progressively solved so many problems it got harder and harder to abandon the chase, but that’s now what I’ve ultimately done!

In summary, based on my experience, for any Houdini for Windows newcomers to OpenVDB and yt, unless they’re gluttons for punishment I would recommend:

1. Not attempting a Windows build of Openvdb or yt into Houdini while it’s reliant on python 2.7 or even into a python 3.7 version if python stays internal to Windows Houdini.

2. Instead, dual boot and install Houdini in Linux.

3. Not bothering with trying to build OpenVDB or yt into Linux Houdini itself while it’s still python 2.7, but instead build them into a Pycharm (or similar) python 3.7 environment, and do all your python processing there.

4. When python 3.7 Houdini eventually comes out, you could move your python activities across to it.

Hope this helps!

Cheers

Ian

From Ian Woodward @ Home

From: Nicolas H notifications@github.com Sent: Thursday, 7 May 2020 9:12 PM To: AcademySoftwareFoundation/openvdb openvdb@noreply.github.com Cc: Ian Woodward ianww@iinet.net.au; Mention mention@noreply.github.com Subject: Re: [AcademySoftwareFoundation/openvdb] Houdini in Windows 10 (#603)

@ianww https://github.com/ianww I hopefuly found your documentation https://groups.google.com/forum/#!topic/openvdb-forum to build the openvdb_houdini and it's pretty useful! Great job ! I am contacting you because I have some issues with the diff file. I'm losing some indent when I copy/paste the text from the pdf file. It seams that the patch file above is not the last one and I was wondering if you can share with us the final patch file here to fix this issue. I have already some comments to improve your documentation, if you want them. Thanks in advance

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AcademySoftwareFoundation/openvdb/issues/603#issuecomment-625189724 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIZTGD4DXYLPK3JWLO73TF3RQKJRVANCNFSM4KEG667A . https://github.com/notifications/beacon/AIZTGD2Y72SSMTWCJZURBQDRQKJRVA5CNFSM4KEG667KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEVB2GXA.gif

[VfxTDwiki]

Thanks Ian for your answer and for sharing with me your experience on that subject. I really appreciate. Looks like you brought me back from the brink and that it is the end of my quest on this subject. I just wanted to make the green nodes work in the openvdb_examples for Houdini.... So I will just skip these examples ! Thanks a lot for your help!!!