search

JMoonr / LATR

[ICCV2023 Oral] LATR: 3D Lane Detection from Monocular Images with Transformer
https://arxiv.org/abs/2308.04583
MIT License
173 stars 31 forks source link

CUDA out of memory #9

Closed hzm-January closed 10 months ago

hzm-January commented 10 months ago

Why does the training code throw exceptions about 'CUDA out of memory'.

I'd tried runing the code after setting 2-layer decoder in LATR without any other code changes, but the problem is still not solved!

I trained models on 4 x NVIDIA RTX A4000 with 16G memory per GPU.

hzm-January commented 10 months ago

training the models on apollo Balanced Scene. CUDA_VISIBLE_DEVICES='0,1,2,3' python -m torch.distributed.launch --nproc_per_node 4 main.py --config config/release_iccv/apollo_standard.py

hzm-January commented 10 months ago

cuda 11.1 python 3.8.18
torch 1.8.0+cu111 pypi_0 pypi torchaudio 0.8.0 pypi_0 pypi torchvision 0.9.0+cu111 pypi_0 pypi mmcv-full 1.5.0 pypi_0 pypi mmdet 2.24.0 pypi_0 pypi mmdet3d 1.0.0rc3 pypi_0 pypi mmengine 0.9.0 pypi_0 pypi mmsegmentation 0.24.0 pypi_0 pypi

JMoonr commented 10 months ago

Thank you for your interest in our work, as stated in #7 , the provided configuration file with batch_size=8 requires gpu memory larger than 20G, but you can always reduce batch_size in config file to fit in your gpu.

hzm-January commented 10 months ago

Thank you for your response, it's very helpful, and based on it, I have resolved my issue.