Cannot build repo - Githubissues

WaliaRohan commented 1 year ago

Hello.

I have all dependencies installed as shown here: https://github.com/nachovizzo/openvdb (namely Boost, TBB and BLOSC). I followed the instructions on your page by running the following commands individually:

git clone https://github.com/nachovizzo/openvdb.git -b nacho/fix_background_inactive \ && cd openvdb \ && mkdir build && cd build \ && cmake -DOPENVDB_BUILD_PYTHON_MODULE=ON -DUSE_NUMPY=ON .. \ && sudo make -j$(nproc) all install

However, I keep getting the following error:

[rwalia@ah-qemachine1-l:~/Documents/openvdb/build] ... $ sudo make -j$(nproc) all install CMake Error: The source directory "/home/rwalia/Documents/openvdb" does not exist. Specify --help for usage, or press the help button on the CMake GUI. make: *** [Makefile:268: cmake_check_build_system] Error 1

You can see that the directory 'home/rwalia/Documents/openvdb' exists since I am running the command from its subdirectory. Have you seen this error / can you suggest how to resolve this?

Also, I noticed that the third command above (&& cmake -DOPENVDB_BUILD_PYTHON_MODULE=ON -DUSE_NUMPY=ON .. \) might be incorrect. Shouldn't it be && cmake -D OPENVDB_BUILD_PYTHON_MODULE=ON -D USE_NUMPY=ON .. \ ?

Thank you in advance for your help.

nachovizzo commented 1 year ago

Hello, sorry is not working out on your end.

Would you please elaborate more on your system, docker, etc?

WaliaRohan commented 1 year ago

Update Hi @nachovizzo . Sorry for the delay in my response. I tried installing the repo on a different system (Ubuntu 22.04.2) and it worked.

However, please note that people might face the following issue while installing make_it_dense: pyyaml fails while building make_it_dense because cython 3.0 (one of the dependencies of pyyaml) is broken (please see https://github.com/yaml/pyyaml/issues/601 and https://github.com/yaml/pyyaml/pull/702). I got around this problem by following the steps below:

Install cython 0.29.36 (basically anything less than 3.0) by using pip install Cython==0.29.36
Install pyyaml from source as shown here: https://github.com/yaml/pyyaml/pull/702#issuecomment-1638930830
Then install make_it_dense as shown here: https://github.com/PRBonn/make_it_dense

Current issue I cannot run https://github.com/PRBonn/make_it_dense/blob/main/apps/train.py. My system cannot find pyopenvdb. The error I get is:

(python_venv) speedracer1702@workstation:~/Projects/surface_reconstruction/make_it_dense/apps$ python3 train.py 
Traceback (most recent call last):
  File "/home/speedracer1702/Projects/surface_reconstruction/make_it_dense/apps/train.py", line 8, in <module>
    from make_it_dense.dataset import KITTIVDBDataModule
  File "/home/speedracer1702/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages/make_it_dense/dataset/__init__.py", line 1, in <module>
    from .kitti_sequence import KITTIOdometrySequence
  File "/home/speedracer1702/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages/make_it_dense/dataset/kitti_sequence.py", line 7, in <module>
    from make_it_dense.utils.cache import get_cache, memoize
  File "/home/speedracer1702/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages/make_it_dense/utils/__init__.py", line 4, in <module>
    from .vdb_utils import *
  File "/home/speedracer1702/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages/make_it_dense/utils/vdb_utils.py", line 5, in <module>
    import pyopenvdb as vdb
ModuleNotFoundError: No module named 'pyopenvdb'

I built openvdb from your fork as you suggested in the README. I noticed that the last supported version of python for pyopenvdb was python 3.8. I am running python 3.10. Could that be a problem?

I am using a virtual environment. All I am trying to do is get to a point where I can get a model of make_it_dense. Basically I want to run this line: https://github.com/PRBonn/make_it_dense/blob/8dd65be8b573a8a8e61dfc8232ad700804532b1c/apps/train.py#L21

Please let me know what more info you need from me to help me troubleshoot this problem. Also, please let me know if you would like me to create a new issue and close this one.

WaliaRohan commented 1 year ago

I was looking at https://github.com/PRBonn/make_it_dense/issues/5 and searched for "pyopenvdb.so". I looked at your comment regarding checking if the path of the shared library for pyopenvdb exists on the interpreter path.

Following are the values for PYTHONPATH and sys.path in my virtual environment:

(python_venv) speedracer1702@workstation:~/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages$ echo $PYTHONPATH
/opt/ros/humble/lib/python3.10/site-packages:/opt/ros/humble/local/lib/python3.10/dist-packages

>>> sys.path
['', '/opt/ros/humble/lib/python3.10/site-packages', '/opt/ros/humble/local/lib/python3.10/dist-packages', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/speedracer1702/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages']

I searched for pyopenvdb.so and found that it exists at the following locations:

/usr/lib/x86_64-linux-gnu/grilo-0.3/libgrlthetvdb.so
/usr/local/lib/libopenvdb.so.9.0.1
/usr/local/lib/libopenvdb.so
/usr/local/lib/libopenvdb.so.9.0
/usr/local/lib/python3.10/site-packages/pyopenvdb.so
/usr/local/lib/python3.10/dist-packages/pyopenvdb.so
/usr/local/lib/libopenvdb.a
/usr/local/include/openvdb
/usr/local/include/openvdb/python/pyopenvdb.h
/usr/local/include/openvdb/openvdb.h
/usr/local/bin/vdb_print

Do you know if there is a way to make python recognize that this shared object library exists? I can try adding these to my PYTHONPATH or sys.path but I've read online (https://stackoverflow.com/questions/1893598/pythonpath-vs-sys-path) that this might be a bad solution.

vnmsklnk commented 1 year ago

I was looking at #5 and searched for "pyopenvdb.so". I looked at your comment regarding checking if the path of the shared library for pyopenvdb exists on the interpreter path.

Following are the values for PYTHONPATH and sys.path in my virtual environment:
(python_venv) speedracer1702@workstation:~/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages$ echo $PYTHONPATH
/opt/ros/humble/lib/python3.10/site-packages:/opt/ros/humble/local/lib/python3.10/dist-packages
>>> sys.path
['', '/opt/ros/humble/lib/python3.10/site-packages', '/opt/ros/humble/local/lib/python3.10/dist-packages', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/home/speedracer1702/Projects/surface_reconstruction/python_venv/lib/python3.10/site-packages']
I searched for pyopenvdb.so and found that it exists at the following locations:
/usr/lib/x86_64-linux-gnu/grilo-0.3/libgrlthetvdb.so
/usr/local/lib/libopenvdb.so.9.0.1
/usr/local/lib/libopenvdb.so
/usr/local/lib/libopenvdb.so.9.0
/usr/local/lib/python3.10/site-packages/pyopenvdb.so
/usr/local/lib/python3.10/dist-packages/pyopenvdb.so
/usr/local/lib/libopenvdb.a
/usr/local/include/openvdb
/usr/local/include/openvdb/python/pyopenvdb.h
/usr/local/include/openvdb/openvdb.h
/usr/local/bin/vdb_print
Do you know if there is a way to make python recognize that this shared object library exists? I can try adding these to my PYTHONPATH or sys.path but I've read online (https://stackoverflow.com/questions/1893598/pythonpath-vs-sys-path) that this might be a bad solution.

I had the same issue, this helped me.

nachovizzo commented 1 year ago

Thanks @vnmsklnk . @WaliaRohan could you check if that solves your problem? I'm not a big fan of virtual env/condas in general, since this stuff typically breaks :)

WaliaRohan commented 1 year ago

Thanks @vnmsklnk . @nachovizzo I'll give that a shot. Meanwhile, is it possible for you to add a docker image to this repo (if that is a better option)? Thanks

nachovizzo commented 1 year ago

@WaliaRohan I will, but I can't anytime soon :) I'll try to make some room in August to do this.

vnmsklnk commented 1 year ago

@WaliaRohan a little bit dirty version of my Dockerfile, but it works. It uses your fix for pyyaml installation Dockerfile.zip

WaliaRohan commented 1 year ago

@vnmsklnk I get the following error while building the dockerfile:

62.12 ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
62.12     torch>=1.11.0 from https://files.pythonhosted.org/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl#sha256=8ced00b3ba471856b993822508f77c98f48a458623596a4c43136158781e306a (from pytorch_lightning>=1.6->make_it_dense==0.1):
62.12         Expected sha256 8ced00b3ba471856b993822508f77c98f48a458623596a4c43136158781e306a
62.12              Got        fe8504f0795ae8a7414a20d65d0dff45d585779787e2ccc03495f80c3f3edab1
62.12 
------
Dockerfile:30
--------------------
  29 |     
  30 | >>> RUN git clone https://github.com/PRBonn/vdbfusion.git && \
  31 | >>>     cd vdbfusion && \
  32 | >>>     sed -i 's/"-c"/"-I" "-c"/g' src/vdbfusion/pybind/CMakeLists.txt && \
  33 | >>>     pip install . && \
  34 | >>>     pip install Cython==0.29.36 && \
  35 | >>>     cd .. && \
  36 | >>>     git clone https://github.com/yaml/pyyaml.git && \
  37 | >>>     cd pyyaml && \
  38 | >>>     git checkout release/5.4.1 && \
  39 | >>>     sed -i.bak 's/Cython/Cython<3.0/g' pyproject.toml && \
  40 | >>>     python3 setup.py sdist && \
  41 | >>>     pip install --pre dist/PyYAML-5.4.1.tar.gz && \
  42 | >>>     cd .. && \
  43 | >>>     git clone https://github.com/PRBonn/make_it_dense && \
  44 | >>>     cd make_it_dense && \
  45 | >>>     pip install .
  46 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c git clone https://github.com/PRBonn/vdbfusion.git &&     cd vdbfusion &&     sed -i 's/\"-c\"/\"-I\" \"-c\"/g' src/vdbfusion/pybind/CMakeLists.txt &&     pip install . &&     pip install Cython==0.29.36 &&     cd .. &&     git clone https://github.com/yaml/pyyaml.git &&     cd pyyaml &&     git checkout release/5.4.1 &&     sed -i.bak 's/Cython/Cython<3.0/g' pyproject.toml &&     python3 setup.py sdist &&     pip install --pre dist/PyYAML-5.4.1.tar.gz &&     cd .. &&     git clone https://github.com/PRBonn/make_it_dense &&     cd make_it_dense &&     pip install ." did not complete successfully: exit code: 1

Any ideas?

vnmsklnk commented 1 year ago

@WaliaRohan I've just built image from this dockerfile successfully. Maybe something happened to the internet connection while downloading the pip package?

WaliaRohan commented 1 year ago

@vnmsklnk I had to execute pip cache purge and it worked. Thank you so much for sharing the docker file! @nachovizzo I am avoiding the solution @vnmsklnk shared in this comment above to prevent breaking my current build. You can close this issue for now if you want to.

nachovizzo commented 1 year ago

@WaliaRohan great nows!. Sorry you had to struggle a lot with this repo.

ingeehong commented 1 year ago

@vnmsklnk hi! thanks for sharing your docker file while building you docker file, i got the following error(build openvdb). do you have any idea how to fix it?

4.778 -- ----------------------------------------------------
4.778 -- ------------ Configuring OpenVDBPython -------------
4.778 -- ----------------------------------------------------
5.077 -- Found Python: /usr/bin/python3.10 (found version "3.10.12") found components: Development.Module Interpreter NumPy
5.078 CMake Error at openvdb/openvdb/python/CMakeLists.txt:110 (find_package):
5.078   By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
5.078   asked CMake to find a package configuration file provided by "pybind11",
5.078   but CMake did not find one.
5.078
5.078   Could not find a package configuration file provided by "pybind11"
5.078   (requested version 2.9.1) with any of the following names:
5.078
5.078     pybind11Config.cmake
5.078     pybind11-config.cmake
5.078
5.078   Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
5.078   "pybind11_DIR" to a directory containing one of the above files.  If
5.078   "pybind11" provides a separate development package or SDK, be sure it has
5.078   been installed.
5.078
5.078
5.079 -- Configuring incomplete, errors occurred!
5.079 See also "/workspace/openvdb/build/CMakeFiles/CMakeOutput.log".
------
Dockerfile:16
--------------------
  15 |
  16 | >>> RUN pip install numpy && \
  17 | >>>     git clone https://github.com/nachovizzo/openvdb.git -b nacho/fix_background_inactive && \
  18 | >>>     cd openvdb && \
  19 | >>>     mkdir build && cd build && \
  20 | >>>     cmake \
  21 | >>>     -DOPENVDB_BUILD_PYTHON_MODULE=ON \
  22 | >>>     -DUSE_NUMPY=ON \
  23 | >>>     .. && \
  24 | >>>     make -j4 all install
  25 |
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install numpy &&     git clone https://github.com/nachovizzo/openvdb.git -b nacho/fix_background_inactive &&     cd openvdb &&     mkdir build && cd build &&     cmake     -DOPENVDB_BUILD_PYTHON_MODULE=ON     -DUSE_NUMPY=ON     .. &&     make -j4 all install" did not complete successfully: exit code: 1

sorry solved with

sudo apt-get install pybind11-dev

PRBonn / make_it_dense

Cannot build repo #7