Pointcept / PointTransformerV3

[CVPR'24 Oral] Official repository of Point Transformer V3 (PTv3)
MIT License
583 stars 30 forks source link

Only One GPU is Active When Using Four GPUs for Testing on a Different Dataset with nuscene.pth from PTV3 #28

Open LSu666 opened 2 months ago

LSu666 commented 2 months ago

Hello,

I've encountered an issue while testing the nuscene.pth model from PTV3 on a different dataset. Despite setting the number of GPUs to 4, it appears that only one GPU is actively running during the test. I'm unsure if this behavior is expected or if it indicates a potential problem with the multi-GPU setup or model compatibility.

Is it normal for only one GPU to be active under these settings, or should all four GPUs be utilized for testing? Any guidance or insights on this matter would be greatly appreciated.

Thank you.

image

Gofinge commented 2 months ago

Hi, all four GPUs should engage with the testing process. Did you set "CUDA_VISIABLE_DEVICES=0,1,2,3" or start the test process with our script/test.sh?

LSu666 commented 2 months ago

Thank you for your previous response; it has been very helpful. However, I've encountered a new issue while continuing my work with the ptv3 project and attempting to apply nuScenes dataset parameters to other datasets.

Specifically, when I apply the parameters from nuScenes to other datasets, I notice that the number of points in the results is less than in the original data. Additionally, the structures of origin_segment and segment do not match, leading me to suspect that some points may have been lost during the data processing or augmentation phase.

I appreciate any guidance or suggestions you could offer on this new problem. Thank you once again for your support.

Gofinge commented 2 months ago

Hi, if you mean cached results in the exp folder, for the current version of Pointcept, we only save the first downsampled predictions for the dual grid-sampling testing strategy. When we need to resume testing, these downsampled results will be mapped back to the shape of orgin_segment. While for future versions, we will add more options.

LSu666 commented 2 months ago

Thank you for your reply. Just to confirm, does this mean that full testing capabilities are not yet available in the current version?

Gofinge commented 2 months ago

I mean that the results from the test process are already "full testing capabilities". Just the cached prediction is downsampled for saving the local disk.

Gofinge commented 2 months ago

The discussion here might help you understand our testing process: https://github.com/Pointcept/Pointcept?tab=readme-ov-file#testing https://github.com/Pointcept/Pointcept/issues/108#issuecomment-1893709522