search

hugoycj / torch-bakedsdf

An unofficial pytorch implementation of BakedSDF
https://bakedsdf.github.io/
MIT License
193 stars 8 forks source link

The code is stuck. #10

Closed hzhao1997 closed 10 months ago

hzhao1997 commented 10 months ago

Thanks for your great work. I tried to run your code on my data but the code is stuck here:

python launch.py --config configs/bakedsdf-colmap.yaml --gpu 0 --train   dataset.root_dir=./data/360_v2/garden/
Global seed set to 42
/home/zh/.conda/envs/bakedsdf/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
Using 16bit None Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
[rank: 0] Global seed set to 42
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------

You are using a CUDA device ('NVIDIA GeForce RTX 4090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name  | Type          | Params
----------------------------------------
0 | model | BakedSDFModel | 41.9 M
----------------------------------------
41.9 M    Trainable params
0         Non-trainable params
41.9 M    Total params
83.865    Total estimated model params size (MB)
Epoch 0: : 0it [00:00, ?it/s]

When I cancel this task, it exits from this line:

Epoch 0: : 0it [00:00, ?it/s]^C^C/home/zh/.conda/envs/bakedsdf/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py:48: UserWarning: Detected KeyboardInterrupt, attempting graceful shutdown...
  rank_zero_warn("Detected KeyboardInterrupt, attempting graceful shutdown...")

This is my environment:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
absl-py                   2.0.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
ca-certificates           2023.12.12           h06a4308_0    defaults
cachetools                5.3.2                    pypi_0    pypi
certifi                   2023.11.17               pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
fonttools                 4.47.0                   pypi_0    pypi
fsspec                    2023.12.2                pypi_0    pypi
google-auth               2.26.1                   pypi_0    pypi
google-auth-oauthlib      1.0.0                    pypi_0    pypi
grpcio                    1.60.0                   pypi_0    pypi
idna                      3.6                      pypi_0    pypi
imageio                   2.33.1                   pypi_0    pypi
imageio-ffmpeg            0.4.9                    pypi_0    pypi
importlib-metadata        7.0.1                    pypi_0    pypi
importlib-resources       6.1.1                    pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1    defaults
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
lightning-utilities       0.10.0                   pypi_0    pypi
markdown                  3.5.1                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.4                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0    defaults
nerfacc                   0.3.3                    pypi_0    pypi
networkx                  3.1                      pypi_0    pypi
ninja                     1.11.1.1                 pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
nvidia-cudnn-cu12         8.9.2.26                 pypi_0    pypi
nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
nvidia-nccl-cu12          2.18.1                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.3.101                 pypi_0    pypi
nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
omegaconf                 2.2.3                    pypi_0    pypi
opencv-python             4.9.0.80                 pypi_0    pypi
openssl                   3.0.12               h7f8727e_0    defaults
packaging                 23.2                     pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       23.3.1           py38h06a4308_0    defaults
protobuf                  4.25.1                   pypi_0    pypi
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pybind11                  2.11.1                   pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pymcubes                  0.1.4                    pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
pyransac3d                0.6.0                    pypi_0    pypi
python                    3.8.18               h955ad1f_0    defaults
python-dateutil           2.8.2                    pypi_0    pypi
pytorch-lightning         1.9.5                    pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0    defaults
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rich                      13.7.0                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
setuptools                68.2.2           py38h06a4308_0    defaults
six                       1.16.0                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0    defaults
sympy                     1.12                     pypi_0    pypi
tensorboard               2.14.0                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
tinycudann                1.7                      pypi_0    pypi
tk                        8.6.12               h1ccaba5_0    defaults
torch                     2.1.2                    pypi_0    pypi
torch-efficient-distloss  0.1.3                    pypi_0    pypi
torchmetrics              1.2.1                    pypi_0    pypi
torchvision               0.16.2                   pypi_0    pypi
tqdm                      4.66.1                   pypi_0    pypi
trimesh                   4.0.8                    pypi_0    pypi
triton                    2.1.0                    pypi_0    pypi
typing-extensions         4.9.0                    pypi_0    pypi
urllib3                   2.1.0                    pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2           py38h06a4308_0    defaults
xz                        5.4.5                h5eee18b_0    defaults
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0    defaults

Could you give me a hand : )

hzhao1997 commented 10 months ago

I find that it is a issue of nerfacc and I solve it by this: https://github.com/nerfstudio-project/nerfacc/issues/70#issuecomment-1279782194