search

autonomousvision / sdfstudio

A Unified Framework for Surface Reconstruction
Apache License 2.0
1.9k stars 179 forks source link

Question about training and validation split #280

Open davidceka opened 5 months ago

davidceka commented 5 months ago

Hi, i'm training some bakedangelo models which gave very nice outputs, but I can't seem to wrap my head around one thing: image

The comment on row 117 states: # set to a very large model so we don't eval with all images So i was wondering that there's not a "complete" train/validation split like happens on Nerfstudio (90-10) Correct me if i'm wrong: Let's say I have 100 images: in sdfstudio these models use all the 100 images for training set and also the same 100 images as validation set, but the 100 images aren't all used because the model is set to a very large number of iterations (in this case 1 million). I tipically stop the training at 200k~ iterations so that should mean that it's at 1/5 of the maximum allowed steps for evaluating all images. Does that mean that I've used 1/5 of all the evaluation dataset -> 20 images?

Second question: why are we using the same images for training and validation?

Sorry if I've been messy but this part has been bugging me for the past week. Can you explain it to me better?

Thanks in advance

niujinshuchong commented 5 months ago

Hi, line 117 is just because the evaluation on all images is very slow. For the train test split, you need to check with data parser you used.

For reconstruction, since we care more about the reconstruction quality and using more images is generally help so we use all available images for training.

davidceka commented 5 months ago

Hi, line 117 is just because the evaluation on all images is very slow. For the train test split, you need to check with data parser you used.

For reconstruction, since we care more about the reconstruction quality and using more images is generally help so we use all available images for training.

I used the sdfstudio-data parser, that's the one using train data also as eval. But doesn't that implicate some level of overfitting??

I was thinking that because we want to reproduce well just the train scenario, overfitting might not be that bad. But correct me if I'm wrong

niujinshuchong commented 5 months ago

Yes, It might be. But I think since the SDF has strong regulations with eikonal loss, using 10 % more images is complete fine. In general, if you use more images for training NeRF, you will also get better results. Of course, you will need train/test split if you want to do some bench mark.

davidceka commented 5 months ago

Ok, but by default in the sdfstudio-data parser train and validation splits DO everlap, right? in this part of the sdfstudio data parser there's this flag i found: image So, by default training and validation set overlap and are the same, but if i set it to "true" a portion of the training is reserved for validation so it's not seen during training, right?