Labeled and Unlabeled dataloaders

[nysp78]

Hello, I want to ask how you handle the dataloading when the unlabeled data is more than the labeled. I have read a couple of approaches for this. The first is to define an epoch as the passing of all unlabeled data from the network, but with this the labeled data will be passed from the network multiple times in an epoch. The second approach is to use a sampler to sample at each training step equal amount of unlabeled data to match the size of the labeled data, so to have 2 dataloaders with the same size. Which of these 2 techniques would force the model to perform better? Generally , I'm a bit confused on how to construct the dataloaders of labeled and unlabeled data in a semi supervised setting. How do you handle this? Any hints will be appreciated!

Thanks in advance.

[Haochen-Wang409]

Hi, thanks for your attention of our work!

In fact, we follow both 2 settings you have mentioned in building labeled/unlabeled loaders, which means 1) one epoch is defined as passing through all unlabeled images and 2) for each training iteration, labeled data and unlabeled data are equally sampled.

Taking 1464 labeled data and 9118 unlabeled data with batch_size=16 as an instance, each batch consists of 8 labeled images and 8 unlabeled images, and the total iterations of one epoch is 9118 // 8 = 1139.

More details and the implementation can be found in here.

[nysp78]

Thanks for your reply, But let me ask something. As the labeled data are less than the unlabeled, the labeled will be forward passed the network multiple times during an epoch, provided that we have defined an epoch as the passing of all unlabeled data. Am I right to this?

[Haochen-Wang409]

Yes, you are right. Taking 1464 labeled data and 9118 unlabeled data as an instance again, all labeled images are expected to be passed 9118 // 1464 = 7 times.

[nysp78]

Perfect. As the labeled are passed num_unlabeled // num_labeled times from the network during a "semi-supervised" epoch, it means that it is like we train the network num_unlabeled // num_labeled "supervised" epochs. So, if we want to compare the performance of the model in those 2 settings (supervised and semi-supervised) how can we do that? It is certain that for the first "semi-supervised" epoch the supervised loss will be much smaller and maybe the accuracy much higher compared with the "supervised" one.

[Haochen-Wang409]

That's a really great question.

To be honest, we simply follow previous works to conduct comparisons with the supervised baseline (SupOnly) in our paper, and we do not think too much on this issue. Using iterations instead of epochs might be a more fair comparison.

[nysp78]

I see, taking into account the gradient updates it makes sense. Thank you very much for your replies and congrats for your work!