search

autonomousvision / monosdf

[NeurIPS'22] MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
MIT License
565 stars 52 forks source link

RuntimeError: Error building extension '_hash_encoder' (Has referred to issue#19) #40

Closed raynehe closed 1 year ago

raynehe commented 1 year ago

Hi, thanks for your wonderful work!

When running command to train monosdf, the following error is reported. I'm running it on Ubuntu 20.04 machine.

I have looked at issue #19 and command the line, but the error still remains (as shown below).

I have installed cudatoolkit-11.3 and cudatoolket-dev-11.3. You might refer to the conda list below.

The error log:

(nf22) rayne@phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx:~/code/monosdf/code$ CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf confs/dtu_mlp_3views.conf  --scan_id 65
/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
RANK and WORLD_SIZE in environ: 0/1
0
shell command : training/exp_runner.py --local_rank=0 --conf confs/dtu_mlp_3views.conf --scan_id 65
Loading data ...
Finish loading data. Data-set size: 49
Detected CUDA files, patching ldflags
Emitting ninja build file ./tmp_build/build.ninja...
Building extension module _hash_encoder...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /home/rayne/anaconda3/envs/nf22/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem /home/rayne/anaconda3/envs/nf22/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
FAILED: hashencoder.cuda.o 
/home/rayne/anaconda3/envs/nf22/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem /home/rayne/anaconda3/envs/nf22/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
In file included from /home/rayne/anaconda3/envs/nf22/include/thrust/system/cuda/detail/execution_policy.h:33:0,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/iterator/detail/device_system_tag.h:23,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/iterator/iterator_traits.h:111,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/detail/type_traits/pointer_traits.h:23,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/type_traits/is_contiguous_iterator.h:27,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/type_traits/is_trivially_relocatable.h:19,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/detail/complex/complex.inl:20,
                 from /home/rayne/anaconda3/envs/nf22/include/thrust/complex.h:1031,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/util/complex.h:8,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/util/Half.h:15,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/core/ScalarType.h:5,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/core/StorageImpl.h:4,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/core/Storage.h:3,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/core/TensorImpl.h:8,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/c10/core/GeneratorImpl.h:12,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/ATen/core/Generator.h:22,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
                 from /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/ATen/cuda/CUDAContext.h:14,
                 from /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu:5:
/home/rayne/anaconda3/envs/nf22/include/thrust/system/cuda/config.h:78:2: error: #error The version of CUB in your include path is not compatible with this release of Thrust. CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB. Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this.
 #error The version of CUB in your include path is not compatible with this release of Thrust. CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB. Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this.
  ^~~~~
[2/3] c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem /home/rayne/anaconda3/envs/nf22/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -c /home/rayne/code/monosdf/code/hashencoder/src/bindings.cpp -o bindings.o 
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1740, in _run_ninja_build
    subprocess.run(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "training/exp_runner.py", line 58, in <module>
    trainrunner = MonoSDFTrainRunner(conf=opt.conf,
  File "/home/rayne/code/monosdf/code/../code/training/monosdf_train.py", line 107, in __init__
    self.model = utils.get_class(self.conf.get_string('train.model_class'))(conf=conf_model)
  File "/home/rayne/code/monosdf/code/../code/utils/general.py", line 17, in get_class
    m = __import__(module)
  File "/home/rayne/code/monosdf/code/../code/model/network.py", line 140, in <module>
    from hashencoder.hashgrid import _hash_encode, HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/__init__.py", line 1, in <module>
    from .hashgrid import HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/hashgrid.py", line 12, in <module>
    from .backend import _backend
  File "/home/rayne/code/monosdf/code/../code/hashencoder/backend.py", line 10, in <module>
    _backend = load(name='_hash_encoder',
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1144, in load
    return _jit_compile(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1357, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1469, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1756, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension '_hash_encoder'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1326120) of binary: /home/rayne/anaconda3/envs/nf22/bin/python
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run
    elastic_launch(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
training/exp_runner.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-11-20_14:10:06
  host      : phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1326120)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

The conda list:

#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   1.3.0                    pypi_0    pypi
addict                    2.4.0                    pypi_0    pypi
anyio                     3.6.2                    pypi_0    pypi
argon2-cffi               21.3.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
asttokens                 2.1.0                    pypi_0    pypi
attrs                     22.1.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
beautifulsoup4            4.11.1                   pypi_0    pypi
blas                      1.0                         mkl  
bleach                    5.0.1                    pypi_0    pypi
brotlipy                  0.7.0           py38h27cfd23_1003  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2022.10.11           h06a4308_0  
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.9.24        py38h06a4308_0  
cffi                      1.15.1           py38h74dc2b5_0  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.3                    pypi_0    pypi
configargparse            1.5.3                    pypi_0    pypi
cryptography              38.0.1           py38h9ce1e76_0  
cudatoolkit               11.3.1               h2bc3f7f_2  
cudatoolkit-dev           11.3.1           py38h497a2fe_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
dash                      2.7.0                    pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
debugpy                   1.6.3                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fastjsonschema            2.16.2                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
flask                     2.2.2                    pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
freetype                  2.12.1               h4a9f257_0  
fvcore                    0.1.5.post20210915            py38    fvcore
giflib                    5.2.1                h7b6447c_0  
gmp                       6.2.1                h295c915_3  
gnutls                    3.6.15               he1e5248_0  
google-auth               2.14.1                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
grpcio                    1.50.0                   pypi_0    pypi
idna                      3.4              py38h06a4308_0  
imageio                   2.16.1                   pypi_0    pypi
importlib-metadata        5.0.0                    pypi_0    pypi
importlib-resources       5.10.0                   pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561  
iopath                    0.1.9                      py38    iopath
ipykernel                 6.17.1                   pypi_0    pypi
ipython                   8.6.0                    pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                8.0.2                    pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.1                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9e                   h7f8727e_0  
jsonschema                4.17.0                   pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            7.4.7                    pypi_0    pypi
jupyter-console           6.4.4                    pypi_0    pypi
jupyter-core              5.0.0                    pypi_0    pypi
jupyter-server            1.23.2                   pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab-widgets        3.0.3                    pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
kornia                    0.6.4                    pypi_0    pypi
lame                      3.100                h7b6447c_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libdeflate                1.8                  h7f8727e_5  
libffi                    3.4.2                h295c915_4  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.2                h7f8727e_0  
libpng                    1.6.37               hbc83047_0  
libstdcxx-ng              11.2.0               h1234567_1  
libtasn1                  4.16.0               h27cfd23_0  
libtiff                   4.4.0                hecacb30_2  
libunistring              0.9.10               h27cfd23_0  
libuv                     1.40.0               h7b6447c_0  
libwebp                   1.2.4                h11a3e52_0  
libwebp-base              1.2.4                h5eee18b_0  
lpips                     0.1.4                    pypi_0    pypi
lz4-c                     1.9.3                h295c915_1  
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.1                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mistune                   2.0.4                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py38h7f8727e_0  
mkl_fft                   1.3.1            py38hd3c417c_0  
mkl_random                1.2.2            py38h51133e4_0  
msgpack                   1.0.4                    pypi_0    pypi
msgpack-numpy             0.4.8                    pypi_0    pypi
nbclassic                 0.4.8                    pypi_0    pypi
nbclient                  0.7.0                    pypi_0    pypi
nbconvert                 7.2.5                    pypi_0    pypi
nbformat                  5.5.0                    pypi_0    pypi
ncurses                   6.3                  h5eee18b_3  
nest-asyncio              1.5.6                    pypi_0    pypi
nettle                    3.7.3                hbbd107a_1  
networkx                  2.8.8                    pypi_0    pypi
ninja                     1.11.1                   pypi_0    pypi
notebook                  6.5.2                    pypi_0    pypi
notebook-shim             0.2.2                    pypi_0    pypi
numpy                     1.21.5                   pypi_0    pypi
nvidiacub                 1.10.0                        0    bottler
oauthlib                  3.2.2                    pypi_0    pypi
open3d                    0.16.0                   pypi_0    pypi
opencv-python             4.5.5.64                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openssl                   1.1.1s               h7f8727e_0  
packaging                 21.3                     pypi_0    pypi
pandas                    1.3.5                    pypi_0    pypi
pandocfilters             1.5.0                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
partio                    1.0.0                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.1.0                    pypi_0    pypi
pip                       22.2.2           py38h06a4308_0  
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              2.5.4                    pypi_0    pypi
plotly                    5.11.0                   pypi_0    pypi
plyfile                   0.7.4                    pypi_0    pypi
portalocker               2.3.0            py38h06a4308_0  
prometheus-client         0.15.0                   pypi_0    pypi
prompt-toolkit            3.0.32                   pypi_0    pypi
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.4                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0  
pygments                  2.13.0                   pypi_0    pypi
pyhocon                   0.3.59                   pypi_0    pypi
pymcubes                  0.1.2                    pypi_0    pypi
pyopenssl                 22.0.0             pyhd3eb1b0_0  
pyparsing                 2.4.7                    pypi_0    pypi
pyquaternion              0.9.9                    pypi_0    pypi
pyrsistent                0.19.2                   pypi_0    pypi
pysocks                   1.7.1            py38h06a4308_0  
python                    3.8.15               h3fd9d12_0  
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.8                      2_cp38    conda-forge
pytorch                   1.11.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytorch3d                 0.7.1                     dev_0    <develop>
pytz                      2022.6                   pypi_0    pypi
pywavelets                1.4.1                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     24.0.1                   pypi_0    pypi
qtconsole                 5.4.0                    pypi_0    pypi
qtpy                      2.3.0                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.28.1           py38h06a4308_0  
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-image              0.19.2                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scipy                     1.7.3                    pypi_0    pypi
send2trash                1.8.0                    pypi_0    pypi
setuptools                65.5.0           py38h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
sniffio                   1.3.0                    pypi_0    pypi
soupsieve                 2.3.2.post1              pypi_0    pypi
sqlite                    3.39.3               h5082296_0  
stack-data                0.6.1                    pypi_0    pypi
tabulate                  0.8.10           py38h06a4308_0  
tenacity                  8.1.0                    pypi_0    pypi
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
termcolor                 2.1.0            py38h06a4308_0  
terminado                 0.17.0                   pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2022.10.10               pypi_0    pypi
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
torchaudio                0.11.0               py38_cu113    pytorch
torchvision               0.12.0               py38_cu113    pytorch
tornado                   6.2                      pypi_0    pypi
tqdm                      4.64.1           py38h06a4308_0  
traitlets                 5.5.0                    pypi_0    pypi
trimesh                   3.10.8                   pypi_0    pypi
typing_extensions         4.3.0            py38h06a4308_0  
urllib3                   1.26.12          py38h06a4308_0  
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.4.2                    pypi_0    pypi
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
widgetsnbextension        4.0.3                    pypi_0    pypi
xz                        5.2.6                h5eee18b_0  
yacs                      0.1.6              pyhd3eb1b0_1  
yaml                      0.2.5                h7b6447c_0  
zipp                      3.10.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0  
zstandard                 0.19.0                   pypi_0    pypi
zstd                      1.5.2                ha4553b6_0

Could you give me a hint about that? Thanks for your help!

niujinshuchong commented 1 year ago

Hi, from the error

error: #error The version of CUB in your include path is not compatible with this release of Thrust. CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB. Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this.
 #error The version of CUB in your include path is not compatible with this release of Thrust. CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB. Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this.

Maybe you could run like

THRUST_IGNORE_CUB_VERSION_CHECK=1 CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf confs/dtu_mlp_3views.conf  --scan_id 65
raynehe commented 1 year ago

Thanks very much! Sadly, I still got the error message as below:

(nf22) rayne@phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx:~/code/monosdf/code$ THRUST_IGNORE_CUB_VERSION_CHECK=1 CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf confs/dtu_mlp_3views.conf  --scan_id 65
/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
RANK and WORLD_SIZE in environ: 0/1
opt.local_rank 0
shell command : training/exp_runner.py --local_rank=0 --conf confs/dtu_mlp_3views.conf --scan_id 65
Loading data ...
Finish loading data. Data-set size: 49
Detected CUDA files, patching ldflags
Emitting ninja build file ./tmp_build/build.ninja...
Building extension module _hash_encoder...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] :/usr/local/cuda-11.3/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.3/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
FAILED: hashencoder.cuda.o 
:/usr/local/cuda-11.3/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.3/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
/bin/sh: 1: :/usr/local/cuda-11.3/bin/nvcc: not found
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1740, in _run_ninja_build
    subprocess.run(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "training/exp_runner.py", line 58, in <module>
    trainrunner = MonoSDFTrainRunner(conf=opt.conf,
  File "/home/rayne/code/monosdf/code/../code/training/monosdf_train.py", line 107, in __init__
    self.model = utils.get_class(self.conf.get_string('train.model_class'))(conf=conf_model)
  File "/home/rayne/code/monosdf/code/../code/utils/general.py", line 17, in get_class
    m = __import__(module)
  File "/home/rayne/code/monosdf/code/../code/model/network.py", line 140, in <module>
    from hashencoder.hashgrid import _hash_encode, HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/__init__.py", line 1, in <module>
    from .hashgrid import HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/hashgrid.py", line 12, in <module>
    from .backend import _backend
  File "/home/rayne/code/monosdf/code/../code/hashencoder/backend.py", line 10, in <module>
    _backend = load(name='_hash_encoder',
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1144, in load
    return _jit_compile(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1357, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1469, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1756, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension '_hash_encoder'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1367592) of binary: /home/rayne/anaconda3/envs/nf22/bin/python
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run
    elastic_launch(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
training/exp_runner.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-11-20_16:44:50
  host      : phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1367592)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
raynehe commented 1 year ago

Maybe I should try a different way to "Define THRUST_IGNORE_CUB_VERSION_CHECK"?

niujinshuchong commented 1 year ago

Could you remove the temp folder tmp_build and try again?

raynehe commented 1 year ago

I remove the tmp_build under hashencoder and run: THRUST_IGNORE_CUB_VERSION_CHECK=1 CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf confs/dtu_mlp_3views.conf --scan_id 65 The error remains the same😟

niujinshuchong commented 1 year ago

I think you could try to modify it here: https://github.com/autonomousvision/monosdf/blob/main/code/hashencoder/backend.py#L14

raynehe commented 1 year ago

I'm thinking whether it's due to the version of PyTorch?

The different between my installation and README is that,

If I use:conda install pytorch torchvision cudatoolkit=11.3 -c pytorch This would install a cpu version of pytorch:

# Name                    Version                   Build  Channel
pytorch                   1.13.0              py3.8_cpu_0    pytorch

Thus when installing, I use pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 instead.

There's no other difference between my installation and README.

raynehe commented 1 year ago

I think you could try to modify it here: https://github.com/autonomousvision/monosdf/blob/main/code/hashencoder/backend.py#L14

After adding THRUST_IGNORE_CUB_VERSION_CHECK=1 at the place you've mentioned, the error changes😟 (I have deleted the tmp_build)

Error:

(nf22) rayne@phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx:~/code/monosdf/code$ python -m torch.distributed.launch --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf confs/dtu_mlp_3views.conf  --scan_id 65
/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
RANK and WORLD_SIZE in environ: 0/1
opt.local_rank 0
shell command : training/exp_runner.py --local_rank=0 --conf confs/dtu_mlp_3views.conf --scan_id 65
Loading data ...
Finish loading data. Data-set size: 49
Detected CUDA files, patching ldflags
Emitting ninja build file ./tmp_build/build.ninja...
Building extension module _hash_encoder...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] :/usr/local/cuda-11.3/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.3/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ THRUST_IGNORE_CUB_VERSION_CHECK=1 -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
FAILED: hashencoder.cuda.o 
:/usr/local/cuda-11.3/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.3/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ THRUST_IGNORE_CUB_VERSION_CHECK=1 -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
/bin/sh: 1: :/usr/local/cuda-11.3/bin/nvcc: not found
[2/3] c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.3/include -isystem /home/rayne/anaconda3/envs/nf22/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -c /home/rayne/code/monosdf/code/hashencoder/src/bindings.cpp -o bindings.o 
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1740, in _run_ninja_build
    subprocess.run(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "training/exp_runner.py", line 58, in <module>
    trainrunner = MonoSDFTrainRunner(conf=opt.conf,
  File "/home/rayne/code/monosdf/code/../code/training/monosdf_train.py", line 107, in __init__
    self.model = utils.get_class(self.conf.get_string('train.model_class'))(conf=conf_model)
  File "/home/rayne/code/monosdf/code/../code/utils/general.py", line 17, in get_class
    m = __import__(module)
  File "/home/rayne/code/monosdf/code/../code/model/network.py", line 140, in <module>
    from hashencoder.hashgrid import _hash_encode, HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/__init__.py", line 1, in <module>
    from .hashgrid import HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/hashgrid.py", line 12, in <module>
    from .backend import _backend
  File "/home/rayne/code/monosdf/code/../code/hashencoder/backend.py", line 10, in <module>
    _backend = load(name='_hash_encoder',
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1144, in load
    return _jit_compile(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1357, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1469, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1756, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension '_hash_encoder'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1372359) of binary: /home/rayne/anaconda3/envs/nf22/bin/python
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/run.py", line 715, in run
    elastic_launch(
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/rayne/anaconda3/envs/nf22/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
training/exp_runner.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-11-20_17:23:18
  host      : phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1372359)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
niujinshuchong commented 1 year ago

Hi, now the error is 

/bin/sh: 1: :/usr/local/cuda-11.3/bin/nvcc: not found

I would suggest you re-install conda environment follow our readme and install pytorch with

pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
raynehe commented 1 year ago

Sorry to bother again, I'm wondering did you mean that I should do as follows:

(instead of using conda install pytorch torchvision cudatoolkit=11.3 -c pytorch)

conda create -y -n monosdf python=3.8
conda activate monosdf
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
conda install cudatoolkit-dev=11.3 -c conda-forge
niujinshuchong commented 1 year ago

Yes

raynehe commented 1 year ago

Hello, I re-install a new environment, and the error changed again:

nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified

Could you please help me about how to solve this?

The whole error is:

(nf2) rayne@phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx:~/code/monosdf/code$ CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf confs/dtu_mlp_3views.conf  --scan_id 65
/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
RANK and WORLD_SIZE in environ: 0/1
opt.local_rank 0
shell command : training/exp_runner.py --local_rank=0 --conf confs/dtu_mlp_3views.conf --scan_id 65
Loading data ...
Finish loading data. Data-set size: 49
Detected CUDA files, patching ldflags
Emitting ninja build file ./tmp_build/build.ninja...
Building extension module _hash_encoder...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/rayne/anaconda3/envs/nf2/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ THRUST_IGNORE_CUB_VERSION_CHECK -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
FAILED: hashencoder.cuda.o 
/usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/rayne/anaconda3/envs/nf2/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -O3 -std=c++14 -allow-unsupported-compiler -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ THRUST_IGNORE_CUB_VERSION_CHECK -c /home/rayne/code/monosdf/code/hashencoder/src/hashencoder.cu -o hashencoder.cuda.o 
nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified
[2/3] c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=_hash_encoder -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/TH -isystem /home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/rayne/anaconda3/envs/nf2/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -c /home/rayne/code/monosdf/code/hashencoder/src/bindings.cpp -o bindings.o 
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1808, in _run_ninja_build
    subprocess.run(
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "training/exp_runner.py", line 58, in <module>
    trainrunner = MonoSDFTrainRunner(conf=opt.conf,
  File "/home/rayne/code/monosdf/code/../code/training/monosdf_train.py", line 107, in __init__
    self.model = utils.get_class(self.conf.get_string('train.model_class'))(conf=conf_model)
  File "/home/rayne/code/monosdf/code/../code/utils/general.py", line 17, in get_class
    m = __import__(module)
  File "/home/rayne/code/monosdf/code/../code/model/network.py", line 140, in <module>
    from hashencoder.hashgrid import _hash_encode, HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/__init__.py", line 1, in <module>
    from .hashgrid import HashEncoder
  File "/home/rayne/code/monosdf/code/../code/hashencoder/hashgrid.py", line 12, in <module>
    from .backend import _backend
  File "/home/rayne/code/monosdf/code/../code/hashencoder/backend.py", line 10, in <module>
    _backend = load(name='_hash_encoder',
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1202, in load
    return _jit_compile(
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1425, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1537, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1824, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension '_hash_encoder'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1377802) of binary: /home/rayne/anaconda3/envs/nf2/bin/python
Traceback (most recent call last):
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/rayne/anaconda3/envs/nf2/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
training/exp_runner.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-11-20_18:56:51
  host      : phil-OMEN-by-HP-45L-Gaming-Desktop-GT22-0xxx
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1377802)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

The coda list is:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   1.3.0                    pypi_0    pypi
ca-certificates           2022.9.24            ha878542_0    conda-forge
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
charset-normalizer        2.1.1                    pypi_0    pypi
contourpy                 1.0.6                    pypi_0    pypi
cudatoolkit-dev           11.3.1           py38h497a2fe_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
google-auth               2.14.1                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
grpcio                    1.50.0                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imageio                   2.22.4                   pypi_0    pypi
importlib-metadata        5.0.0                    pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.2                h295c915_4  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.6.2                    pypi_0    pypi
ncurses                   6.3                  h5eee18b_3  
networkx                  2.8.8                    pypi_0    pypi
ninja                     1.11.1                   pypi_0    pypi
numpy                     1.23.5                   pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.6.0.66                 pypi_0    pypi
openssl                   1.1.1s               h7f8727e_0  
packaging                 21.3                     pypi_0    pypi
pillow                    9.3.0                    pypi_0    pypi
pip                       22.2.2           py38h06a4308_0  
protobuf                  3.20.3                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyhocon                   0.3.59                   pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.8.15               h3fd9d12_0  
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.8                      2_cp38    conda-forge
pywavelets                1.4.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scipy                     1.9.3                    pypi_0    pypi
setuptools                65.5.0           py38h06a4308_0  
six                       1.16.0                   pypi_0    pypi
sqlite                    3.39.3               h5082296_0  
tensorboard               2.11.0                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tifffile                  2022.10.10               pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
torch                     1.12.1+cu113             pypi_0    pypi
torchvision               0.13.1+cu113             pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
trimesh                   3.16.4                   pypi_0    pypi
typing-extensions         4.4.0                    pypi_0    pypi
urllib3                   1.26.12                  pypi_0    pypi
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
xz                        5.2.6                h5eee18b_0  
zipp                      3.10.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0

Thank you for your help!

niujinshuchong commented 1 year ago

I think you need to remove THRUST_IGNORE_CUB_VERSION_CHECK or try to change it as -DTHRUST_IGNORE_CUB_VERSION_CHECK=1

raynehe commented 1 year ago

That works. Thanks for your kind help!