search

ywyue / RoomFormer

[CVPR 2023] RoomFormer: Two-level Queries for Single-stage Floorplan Reconstruction
https://ywyue.github.io/RoomFormer/
MIT License
163 stars 22 forks source link

Training got stuck #20

Closed Linya-Peng closed 1 month ago

Linya-Peng commented 8 months ago

Hi, ywyue! Thank you for your wonderful work. I tried to train on the Structured3D dataset, however, the training got stuck midway without any error being reported. image I tried to set --num_workers=0, but the problem hasn't been resolved. I've tried terminating and resuming training multiple times, but the epoch at which it gets stuck varies each time. Do you have any suggestions for a solution?

I'm using 1.9.0+cu111 and running the main.py in a WSL2 Ubuntu 20.04

ywyue commented 8 months ago

Hi @Linya-Peng, thank you for your interest in our work! The problem you said is weird to me (I didn't meet that). What's your GPU card (and memory)? Could you just try a smaller batch size (e.g. 2) and see whether it will solve your problem?

Linya-Peng commented 7 months ago

Hi @Linya-Peng, thank you for your interest in our work! The problem you said is weird to me (I didn't meet that). What's your GPU card (and memory)? Could you just try a smaller batch size (e.g. 2) and see whether it will solve your problem?

Hi ywyue, thank you for your reply! I've tried set batch size as 2, but the problem still existed. After several attemps and searching, I have not solve this problem. I'm working on TRX 4070 Ti GPU, with 12GB GPU memory. Do you have any suggestions for the hardware capacity needed for training?

ywyue commented 7 months ago

Hi @Linya-Peng, our code is tested on NVIDIA TITAN RTX with 24 GB memory! Do you also meet similar issues when running other programs with your current GPU?

Linya-Peng commented 6 months ago

Hi @Linya-Peng, our code is tested on NVIDIA TITAN RTX with 24 GB memory! Do you also meet similar issues when running other programs with your current GPU?

Thank you for your reply! I didn't meet similar issues when running other programs. I'm not sure if it's because I'm using WSL, but I've run other projects successfully with the same configuration.

ywyue commented 2 months ago

Hey, sorry for the late follow-up. I cannot reproduce your issue without WSL. Have you tried training on native Ubuntu or other Linux systems (not WSL)?

ywyue commented 1 month ago

Close for now. Feel free to reopen it if you have any concerns.