search

google-research / fixmatch

A simple method to perform semi-supervised learning with limited data.
Apache License 2.0
1.11k stars 177 forks source link

Question about RandAugment Implementation #65

Open psellcam opened 3 years ago

psellcam commented 3 years ago

In the paper you mention that flips are applied with 50 percent probability. Is it also the case that each randaugment sample is also applied with 50 percent probability or are two randaugment choices always applied to each unlabelled image?

The reason I ask is that in the main pytorch reimplementation of your work there is a 50 percent chance for each randaugment policy to be applied but I didn't see this in your paper

Many Thanks

carlini commented 3 years ago

When we say in our paper that we apply a "weak augmentation" then this means (among other things): "flip, with 50% probability, the image across the vertical axis". When we say "strong augmentation" then this means (for the case of RA): "apply the RA function 100% of the time". I'm not sure what any other repository is doing but the 50% number in our paper means only that flips are done 50% of the time, RA is treated as a black box that can do whatever it wants to augment. Similarly, when say we use CTA, this means we use the CTA function to augment 100% of the time and let it do whatever it wants.

psellcam commented 3 years ago

Thank you for this fast response,

I believed it was 100% but had to double check

JoyHuYY1412 commented 3 years ago

Hi, @carlini, I want to get the weak and strong augmented images of labeled images as the unlabeled in each iteration, but after I change https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/cta/lib/train.py#L49 to False, I meet the AssertionError. Could you give me some advice? Is the https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L88 the strong augmented images of labeled images?

carlini commented 3 years ago

Sorry I don't understand quite what you want. Could you expand?

JoyHuYY1412 commented 3 years ago

Sorry I don't understand quite what you want. Could you expand?

Hi @carlini, I mean how to get the strong augmented labeled images? are they 'x_in'?

carlini commented 3 years ago

y_in here gets set with the weakly and strongly augmented images.

https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L89

What do you want to do with the augmentations?

JoyHuYY1412 commented 3 years ago

y_in here gets set with the weakly and strongly augmented images.

https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L89

What do you want to do with the augmentations?

Hi @carlini, thank you for your reply. I want the weakly and strongly augmented images of only the labeled images. Are they 'xt_in' and 'x_in'? https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L87-L88

carlini commented 3 years ago

The labeled images will get passed as part of the unlabeled set as well, and that's how the labeled images are strongly augmented.

I'm still confused what your overall objective is, though. If you just want to weakly and strongly augment a particular set of images then it would be easier to call directly into the augmentation libraries. Do you ant something else?

JoyHuYY1412 commented 3 years ago

The labeled images will get passed as part of the unlabeled set as well, and that's how the labeled images are strongly augmented.

I'm still confused what your overall objective is, though. If you just want to weakly and strongly augment a particular set of images then it would be easier to call directly into the augmentation libraries. Do you ant something else?

Hi @carlini, I just want the strong augmented images for only the labeled images in a batch. For example, in each batch, I have 64 labeled images L and 64*7*2 unlabeled images U, and I want the corresponding 64 strong augmented images of L.

I know that labeled images will get passed as part of the unlabeled set, but I just want the strong augmented labeled images in each batch. I want to know what is ''x_in'', because I think they are the strong augmented version of "xt_in". Am I right?

If not, what augmentation libraries should I call to get the strong augmented images of L in each iteration?

Thank you so much.

carlini commented 3 years ago

FixMatch does not strongly augment each of the labeled images in the batch on every iteration. It only weakly augments those images. The only time the labeled images become strongly augmented is when they are passed as unlabeled images. (See Algorithm 1, line 2.)

So if you want strongly augmented labeled images, you will need to get collect them from the unlabeled dataset when they happen to appear there.

JoyHuYY1412 commented 3 years ago

FixMatch does not strongly augment each of the labeled images in the batch on every iteration. It only weakly augments those images. The only time the labeled images become strongly augmented is when they are passed as unlabeled images. (See Algorithm 1, line 2.)

So if you want strongly augmented labeled images, you will need to get collect them from the unlabeled dataset when they happen to appear there.

Hi @carlini, thank you for your reply. I have just one last question: what is the 'x_in' here in the following code? I am a little bit confused. I think they are not the validation data. https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L87-L88

And I found some explanation in https://github.com/google-research/fixmatch/issues/4#issuecomment-580120773, but I still don't understand how to generate x_in. Thank you so much.

carlini commented 3 years ago

xt_in has the training examples, and x_in has the evaluation images. You can see that x_in is only used in the classification ops:

https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L137-L138

JoyHuYY1412 commented 3 years ago

xt_in has the training examples, and x_in has the evaluation images. You can see that x_in is only used in the classification ops:

https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L137-L138

Hi @carlini, the code in https://github.com/google-research/fixmatch/blob/master/fixmatch.py#L38 seems to give augmentation to training labeled images and generate x_in.

The code is a little complicated for me, and I hope you don't mind my bother.

carlini commented 3 years ago

Yeah, the code is a bit convoluted. It evolved from three projects. If you'd like to find a simpler implementation David wrote one here https://github.com/google/objax/tree/master/examples/fixmatch

However, for your question, no those augmentations aren't going into x_in. I'm not quite sure why you think they are. If you look at the code then xin https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L88 is only ever used here, and exported as x https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/fixmatch.py#L136-L138 and the x gets fed here as the prediction op https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/libml/train.py#L228-L233

JoyHuYY1412 commented 3 years ago

Thank you so much and thank you for the simplified version @carlini . I find the code here feeds x_in as x['probe']: https://github.com/google-research/fixmatch/blob/d4985a158065947dba803e626ee9a6721709c570/cta/lib/train.py#L63-L69

Are x['probe'] here the evaluation images, or the training labeled images? I am still not very clear about that... sorry

carlini commented 3 years ago

Sorry for the late reply. If you haven't figured it out already, probe here is used to check how accurate the model is on these probe images as a way to tune the CTAugment scheme. If you're using randaugment then you don't need probes.