search

OpenDriveLab / OpenLane-V2

[NeurIPS 2023 Track Datasets and Benchmarks] OpenLane-V2: The First Perception and Reasoning Benchmark for Road Driving
https://proceedings.neurips.cc/paper_files/paper/2023/hash/3c0a4c8c236144f1b99b7e1531debe9c-Abstract-Datasets_and_Benchmarks.html
Apache License 2.0
541 stars 65 forks source link

CUDA out of memory #64

Closed Bochicchio3 closed 1 year ago

Bochicchio3 commented 1 year ago

Hello, I have 32GB v100 gpus, but I still can't fit batch size 1 for the large baseline. I was wondering how do you train it and on which gpus. I didn't find any option to lower image resolution for training, am I wrong? Do you train with half precision?
Thank you for the clarifications

sephyli commented 1 year ago

Hi, @Bochicchio3,

I understand the challenge you're facing, and I apologize for any inconvenience. The memory usage issue you're encountering is primarily due to the large resolution of the input images. In our setup, we're dealing with 7 surround-view images, each with a resolution of $2048 \times 1550$. To handle this, we utilize A100 GPUs with 80GB of GPU memory to meet the requirements effectively.

You can consider adding a resize function within the train_pipeline. By downscaling the input image resolution, you can potentially reduce the memory demands. However, the specific functionality isn't included in the current codebase. Maybe you can adapt one from BEV-Toolbox or BEVFormer.

We do not utilize half precision during our training process.

Bochicchio3 commented 1 year ago

I see, thank you!