Closed Outlying3720 closed 3 months ago
Tai-Wang
commented
3 months ago Thanks for your feedback and careful observation. The different data_time may come from the different properties of these raw data, and we will have a closer look at this issue.
For now, you can reduce the time by increasing the batch size during inference.
jiachen0212
commented
3 months ago Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
main branch https://github.com/open-mmlab/mmdetection3d
Environment
/opt/conda/lib/python3.7/site-packages/MinkowskiEngine/__init__.py:42: UserWarning: The environment variable `OMP_NUM_THREADS` not set. MinkowskiEngine will automatically set `OMP_NUM_THREADS=16`. If you want to set `OMP_NUM_THREADS` manually, please export it on the command line before running a python script. e.g. `export OMP_NUM_THREADS=12; python your_program.py`. It is recommended to set it below 24. "It is recommended to set it below 24.", sys.platform: linux Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0] CUDA available: True MUSA available: False numpy_random_seed: 2147483648 GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 4090 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.3, V11.3.109 GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.12.1 PyTorch compiling details: PyTorch built with: - GCC 9.3 - C++ Version: 201402 - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 11.3 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37 - CuDNN 8.3.2 (built against CUDA 11.5) - Magma 2.5.2 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, TorchVision: 0.13.1 OpenCV: 4.6.0 MMEngine: 0.10.3 MMDetection: 3.3.0 MMDetection3D: 1.4.0+962f093 spconv2.0: TrueReproduces the problem - code sample
Hi, I found it is extremely slow to run a test. So much time is used in data loading.
My environment: 4*RTX4090 datasets storage in local SSD
test batchsize:
test_dataloader = dict(batch_size=8, num_workers=8, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict(type=dataset_type, data_root=data_root, ann_file='embodiedscan_infos_test.pkl', vg_file='embodiedscan_test_vg.json', metainfo=metainfo, pipeline=test_pipeline, test_mode=True, filter_empty_gt=True, box_type_3d='Euler-Depth'))the data_time to load different datasets is very different. data_time of scannet is about 3.4 data_time of 3rscan is about 0.4 data_time of matterport is about 14
which makes the test time so long. Does it normal to have more than 3 hours to run a test? and How can I reduce data_time, especially in loading scannet and matterport data?
Reproduces the problem - command or script
Start test.py:
python3 -m torch.distributed.run --nproc_per_node=4 tools/test.py configs/grounding/mv-grounding_8xb12_embodiedscan-vg-9dof.py work_dirs/mv-grounding/epoch_12.pth --launcher="pytorch"
Hello, the download of the 3rscan data set failed. Could you please give me the data connection~ ！！！ thanks!!!
Tai-Wang
commented
3 months ago @Outlying3720 Hi, we have just looked into this problem and found that the key point is the resolution of depth images. The depth images of ScanNet vs. 3RScan vs. Matterport3D have a resolution of 640x480 vs. 224x172 vs. 1280x1024, leading to up to 50x differences in the number of loaded point clouds. Now we just convert them in the same way and do the downsampling for point clouds. After printing the time for each data transform, we find this results in much larger data_time when processing Matterport3D data.
To alleviate this problem, you can try to downsample the depth images before converting them to point clouds. Note that the depth camera intrinsic matrix should be adjusted accordingly.
Tai-Wang
commented
3 months ago @jiachen0212 We cannot distribute such links without the permission of 3RScan data hosts. I recommend you contact their official hosts for the downloading issue.
Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
main branch https://github.com/open-mmlab/mmdetection3d
Environment
Reproduces the problem - code sample
Hi, I found it is extremely slow to run a test. So much time is used in data loading.
My environment: 4*RTX4090 datasets storage in local SSD
test batchsize:
the data_time to load different datasets is very different. data_time of scannet is about 3.4 data_time of 3rscan is about 0.4 data_time of matterport is about 14
which makes the test time so long. Does it normal to have more than 3 hours to run a test? and How can I reduce data_time, especially in loading scannet and matterport data?
Reproduces the problem - command or script
Start test.py: