search

szymanowiczs / splatter-image

Official implementation of `Splatter Image: Ultra-Fast Single-View 3D Reconstruction' CVPR 2024
https://szymanowiczs.github.io/splatter-image
BSD 3-Clause "New" or "Revised" License
795 stars 54 forks source link

why killed? #30

Closed johnren-code closed 5 months ago

johnren-code commented 5 months ago

The training process will be killed in the half

johnren-code commented 5 months ago

If I understand correctly, the main reason for being killed is due to insufficient memory during the training process. I noticed that memory usage increases continuously at the beginning of training and stabilizes at around a certain value. What is the reason for this?

szymanowiczs commented 5 months ago

The dataset is being read into working memory - see the data loader where new images are appended to the dataset object attributes https://github.com/szymanowiczs/splatter-image/blob/98b465731c3273bf8f42a747d1b6ce1a93faf3d6/scene/co3d.py#L144C13-L145C1 This is done so that data loading doesn't bottleneck training.

It is possible that the entire dataset does not fit into RAM, depending on the machine you're using.

If this is an issue for you, you will need to modify the data loader so that the data is being read just for the current iteration - make sure to provide enough workers so that the training process does not become cpu-bottlenecked.

johnren-code commented 5 months ago

Yeah, you are right. Perhaps because my dataset is too large (512*512 image), it is not enough to load all of it into memory. If I want to load the dataset during each iteration, could you please provide me with some reference code? This will really be of great help to me, thank you!

szymanowiczs commented 5 months ago

Unfortunately I don't have any reference for this. It might be included in the coming weeks / months but I'm afraid I don't have anything that could help you right now. Best of luck!

johnren-code commented 5 months ago

Okay, thank you! I will try to modify the part of loading data myself. In summary, thank you for your help, which has answered my confusion!