search

PJLab-ADG / 3DTrans

An open-source codebase for exploring autonomous driving pre-training
https://bobrown.github.io/Team_3DTrans.github.io/
Apache License 2.0
585 stars 72 forks source link

About training Bi3D Adaptation stage 1: active source domain data. #5

Closed YannnWanggg closed 1 year ago

YannnWanggg commented 1 year ago

Hello. I'm running the Bi3D Adaptation stage 1. After training the first epoch, the algorithm begins to Active Evaluate. However, after the evaluation is finished, I noticed the memory usage of GPU is abnormal. My hardware is RTX 4090 24GB, however when the evaluation is done, the program reports CUDA error: out of memory. I wonder have you ever met this problem before and how did you solve it? What's more, https://github.com/PJLab-ADG/3DTrans/blob/c3bf52ff0f4f46bd898dbd3014fbb65e1714f043/tools/train_utils/train_active_source_utils.py#L518 set oss to True, which seems not suitable for me, so I set it to False.

JiakangYuan commented 1 year ago

Thanks for your attention to our work. We will test the memory usage of GPUs soon. Further, set oss=True when you use s3 ceph. We will revise our code soon.

RhythmOfTheRain-Byte commented 10 months ago

Thanks for your attention to our work. We will test the memory usage of GPUs soon. Further, set oss=True when you use s3 ceph. We will revise our code soon.

When I set OSS=TRue, some error occurs source_list = active_learning_utils.get_dataset_list(source_file_path, oss=False) def get_target_list(target_pkl_file, oss): if oss == True: from petrel_client.client import Client ModuleNotFoundError：Nomodule named “petrel_client” Any one else konw the reason?