search

zyw-stu / CPA-Enhancer

This is the official repository of the paper: CPA-Enhancer: Chain-of-Thought Prompted Adaptive Enhancer for Object Detection under Unknown Degradations
Apache License 2.0
40 stars 1 forks source link

High VRAM Usage #8

Open ducnt1210 opened 1 month ago

ducnt1210 commented 1 month ago

Thank you for sharing your work! I’ve been using your code with some modifications for my own dataset, and I’m encountering significantly higher VRAM usage during training. I’m hoping you could provide some insights.

I’ve used your cpa_config file and only modified the dataset section to use my custom dataset. The only other change I made was adjusting the output layer to 6 classes instead of the original 5. In your paper, you mentioned using a batch size of 16 for training on an RTX 4090 (24GB VRAM), but I can only fit a batch size of 2 on an A100 (40GB VRAM). If I increase the batch size to 4, I run into an OOM error.

Could the higher memory usage be due to differences in the dataset? Also, could you provide more details about your VRAM usage and training time during your experiments? Any advice or insights would be greatly appreciated. Thank you!

zyw-stu commented 1 week ago

Sorry for the late reply! I’ve been tied up with some other research projects recently. First, the increased VRAM usage is likely due to differences in the dataset. If your custom dataset has larger image sizes, it will require more memory during training. Additionally, I highly recommend using gradient accumulation. It allows you to simulate larger batch sizes without increasing memory usage, which is especially helpful when you're running into VRAM limitations. I hope these suggestions help! Feel free to reach out if you have further questions.