yewzijian
commented
2 years ago Hi, I'm using the sparse tensors from Minkowski Engine as a quick way to perform the downsampling. Your issue might be due to a bug in your version of Minkowski Engine, however I am not able to replicate it on my machine even with a fresh environment.
I provide my installation code here which hopefully can help (I just tested this on PyTorch 1.10.0 with cuda 11.1):
conda install openblas-devel -c anaconda
export CUDA_HOME=/usr/local/cuda-11.1
pip install -U git+https://github.com/NVIDIA/MinkowskiEngine -v --no-deps --install-option="--blas_include_dirs=${CONDA_PREFIX}/include" --install-option="--blas=openblas"Alternatively, if you cannot resolve the issue with Minkowski Engine, you can use the (slower) CPU version of the processing code by replacing the processing code with its CPU version. You can do so by replacing PreprocessorGPU with Preprocessor here, taking care to change the imports properly, and compiling the helper files using models/backbone_kpconv/compile_wrappers.sh.
Zi Jian
tranceok
caijillx
qsisi
huk112739
ubuntu18.04 RTX3090 cuda11.1 MinkowskiEngine 0.5.4
The following error occurred when I tried to run your model。
(RegTR) ➜ src git:(main) ✗ python test.py --dev --resume ../trained_models/3dmatch/ckpt/model-best.pth --benchmark 3DMatch
/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/init.py:36: UserWarning: The environment variable
OMP_NUM_THREADSnot set. MinkowskiEngine will automatically setOMP_NUM_THREADS=16. If you want to setOMP_NUM_THREADSmanually, please export it on the command line before running a python script. e.g.export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24. warnings.warn( /home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") 04/23 20:06:22 [INFO] root - Output and logs will be saved to ../logdev 04/23 20:06:22 [INFO] cvhelpers.misc - Command: test.py --dev --resume ../trained_models/3dmatch/ckpt/model-best.pth --benchmark 3DMatch 04/23 20:06:22 [INFO] cvhelpers.misc - Source is from Commit 64e5b3f0 (2022-03-28): Fixed minor typo in Readme.md and demo.py 04/23 20:06:22 [INFO] cvhelpers.misc - Arguments: benchmark: 3DMatch, config: None, logdir: ../logs, dev: True, name: None, num_workers: 0, resume: ../trained_models/3dmatch/ckpt/model-best.pth 04/23 20:06:22 [INFO] root - Using config file from checkpoint directory: ../trained_models/3dmatch/config.yaml 04/23 20:06:22 [INFO] data_loaders.threedmatch - Loading data from ../data/indoor 04/23 20:06:22 [INFO] RegTR - Instantiating model RegTR 04/23 20:06:22 [INFO] RegTR - Loss weighting: {'overlap_5': 1.0, 'feature_5': 0.1, 'corr_5': 1.0, 'feature_un': 0.0} 04/23 20:06:22 [INFO] RegTR - Config: d_embed:256, nheads:8, pre_norm:True, use_pos_emb:True, sa_val_has_pos_emb:True, ca_val_has_pos_emb:True 04/23 20:06:25 [INFO] CheckPointManager - Loaded models from ../trained_models/3dmatch/ckpt/model-best.pth 0%| | 0/1623 [00:00<?, ?it/s] ** On entry to cusparseSpMM_bufferSize() parameter number 1 (handle) had an illegal value: bad initialization or already destroyedTraceback (most recent call last): File "test.py", line 75, in
main()
File "test.py", line 71, in main
trainer.test(model, test_loader)
File "/home/lileixin/work/Point_Registration/RegTR/src/trainer.py", line 204, in test
test_out = model.test_step(test_batch, test_batch_idx)
File "/home/lileixin/work/Point_Registration/RegTR/src/models/generic_reg_model.py", line 132, in test_step
pred = self.forward(batch)
File "/home/lileixin/work/Point_Registration/RegTR/src/models/regtr.py", line 117, in forward
kpconv_meta = self.preprocessor(batch['src_xyz'] + batch['tgt_xyz'])
File "/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/lileixin/work/Point_Registration/RegTR/src/models/backbone_kpconv/kpconv.py", line 489, in forward
pool_p, pool_b = batch_grid_subsampling_kpconv_gpu(
File "/home/lileixin/work/Point_Registration/RegTR/src/models/backbone_kpconv/kpconv.py", line 232, in batch_grid_subsampling_kpconv_gpu
sparse_tensor = ME.SparseTensor(
File "/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiSparseTensor.py", line 275, in init
coordinates, features, coordinate_map_key = self.initialize_coordinates(
File "/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiSparseTensor.py", line 338, in initialize_coordinates
features = spmm_avg.apply(self.inverse_mapping, cols, size, features)
File "/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/sparse_matrix_functions.py", line 183, in forward
result, COO, vals = spmm_average(
File "/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/sparse_matrix_functions.py", line 93, in spmm_average
result, COO, vals = MEB.coo_spmm_average_int32(
RuntimeError: CUSPARSE_STATUS_INVALID_VALUE at /home/lileixin/MinkowskiEngine/src/spmm.cu:590
(RegTR) ➜ src git:(main) ✗ python test.py --dev --resume ../trained_models/3dmatch/ckpt/model-best.pth --benchmark 3DMatch
/home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/init.py:36: UserWarning: The environment variable
OMP_NUM_THREADSnot set. MinkowskiEngine will automatically setOMP_NUM_THREADS=16. If you want to setOMP_NUM_THREADSmanually, please export it on the command line before running a python script. e.g.export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24. warnings.warn( /home/lileixin/anaconda3/envs/RegTR/lib/python3.8/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.")But when I cross out this line of code, the program can run. [` sparse_tensor = ME.SparseTensor( features=points, coordinates=coord_batched,
quantization_mode=ME.SparseTensorQuantizationMode.UNWEIGHTED_AVERAGE