Questions about search and retrain.

[Shaw95]

@ildoonet @sublee Hi, it is so nice of you guys to release the search code and will be much appreciated. While running with your search code and retrain with found policies, I still got some problems and hopefully you can help me figure them out.

1. Some bugs in the search code after policy search, seems like the policy cannot be fed into the final retrain stage？
2. About your search process, for me it looks like during each trial you select 5 sub-policies containing 2 operations, while in your eval_tta, you just sampled 5 val_loaders by randomly select 1 sub-policy from your 5 sub-policies and report the avg acc? I'm wondering what's the intuition behind this?
3. After running search with your code, we can get a policy list with around 750 sub-polices. I'm wondering how many sub-policies you used to get the results reported in the paper.
4. I've retrained the model with randomly selected polices, p and magnitude many times, I can consistently get the comparable error rates with the results trained by best policies. This makes me very confused, so I'm very much desired to listen to your opinions.

Looking forward to your reply Thanks again.

[Shaw95]

One more question. What does the T represent in your paper. from your code, it seems to be the target_lb, however your paper refers T to search width and set to 2. Really hoping for your reply. It should be very helpful to my current work.

Thanks!

[rodrigoberriel]

@Shaw95 I think T is supposed to be this hard-coded 1:

https://github.com/kakaobrain/fast-autoaugment/blob/e79d0c78a0fbb5126cc1bbb2cbf4d001d6f680d6/FastAutoAugment/search.py#L225-L226

For this to be consistent with the Algorithm 1, probably these two for loops should be inverted (I guess K=cv_num). If I understood correctly, the Figure 2 and the Algorithm 1 are also inconsistent. In Figure 2, the top-N selection is performed out of the T-loop while in the Algorithm 1 it is inside the loop. This inconsistency seems to be minor and these numbers seem completely arbitrary, therefore I would tune them anyway (ok, it would require minor changes in the code...).

I would recommend authors to explicitly provide the mapping between the inputs of Algorithm 1 and variables in the code. It would be extremely helpful.

[Shaw95]

@rodrigoberriel Thank you. I think they have updated their search coed This is the original one.

[RenShuhuai-Andy]

@Shaw95 @rodrigoberriel The code is really confusing... I think T supposed to be augment['num_policy'] in https://github.com/kakaobrain/fast-autoaugment/blob/e79d0c78a0fbb5126cc1bbb2cbf4d001d6f680d6/FastAutoAugment/search.py#L88-L91 and 'stop': {'training_iteration': args.num_policy}, in https://github.com/kakaobrain/fast-autoaugment/blob/e79d0c78a0fbb5126cc1bbb2cbf4d001d6f680d6/FastAutoAugment/search.py#L232-L244 If not, could you tell me why it need to be repeated args.num_policy times? btw, I still don't know what for _ in range(1): means, this line happens more than one place.

[KhrystynaFaryna]

Hi, @RenShuhuai-Andy and @Shaw95 did you manage to find out why calling the generator needs to be repeated args.num_policy of times? Technically they should get the same batch every time they call a generator and do one iteration of it (it is seeded). I looked at accuracies from 5 repetitions and they are not the same. Is it because of the fact that whether a particular transform will be applied is uncertain?