search

NVIDIA / vid2vid

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.
Other
8.6k stars 1.2k forks source link

max_dataset_size Not Utilized in Temporal Dataset #66

Closed heliotl closed 6 years ago

heliotl commented 6 years ago

In base_options.py there is the option 'max_dataset_size' which has the description: 'Maximum number of samples allowed per dataset. If the dataset directory contains more than max_dataset_size, only a subset is loaded.'

Based on the description, it seems as if though if your entire dataset is 1000 video series and you set max_dataset_size = 200, then the model will only train on the same 200/1000 samples throughout it's training cycle. However, as of now, this is not the case. From what I see in temporal_dataset.py, all of the contents in 'dataroot/train_A' is loaded thenself.n_of_seqs = len(self.A_paths), allowing all content in A_paths to be indexed. This happened to me, I have a dataset of 4500 sequences, and specified max_dataset_size to be 128, however, by the end of training the model had been exposed to 2026 different sequences rather than only 128 specified by max_dataset_size. This dismisses the use of max_dataset_size.

However, if you are to do the following: self.n_of_seqs = min(len(self.A_paths), opt.max_dataset_size) the issue is resolved.

If the purpose of max_dataset_size is to specify the number of samples you want to train on (have a fixed subset), then this a potential fix. However, if the purpose of max_dataset_size is not to have a fixed subset of samples, then could you please make that more clear in the description.

tcwang0509 commented 6 years ago

Thanks for pointing it out. It will be updated in the future release.