We cannot obtain your claimed result on ImageNet after trying many configurations

[NoOneUST]

Hello, we have sent you several emails before asking about this problem. We first try to use 10% training set for training and 2.5% training set for validation and use a linear learning rate scheduler, with SGDM we train 250 epochs but the result is horrible. Then we try to use the whole ImageNet and get an even much more horrible result. Later we change the optimizer and use the whole ImageNet, now we still have clear accuracy gap. May you kindly give us more details about how you obtained the results in your paper? We really need your response, or we may have to follow our "wrong" configuration, do fair comparisons, and report our real observations to the conferences. Thank you so much!

[yuhuixu1993]

@NoOneUST , sorry for the late reply, I was quite busy those days. From your description, I guess your disappointing results may caused by taking the search results as the final evaluate result. First, you need to make sure that the whole search process is as followed: 1. search stage: use as mentioned 10% training set for training and 2.5% training set for validation which will takes about 50 epochs to get the searched architectures (corresponding python file traing_search_imagenet.py). 2. evaluate stage: change the architecture in genotypes.py by the searched architecture in the first stage and use the whole dataset which will takes about 250 epochs to get the final accuracy (corresponding python file traing_imagenet.py, genotypes.py). Please note that the python files are different in different stages.

If you have more questions, please let me know.

[yuhuixu1993]

@Christian-lyc Does the validate phase refer to evaluating the searched architectures on imagenet (traing_imagenet.py)? If yes, 1% is the training acc or the validate acc or both?

[yuhuixu1993]

@Christian-lyc, what I know is that if you use torch.utils.data.sampler.SubsetRandomSampler to generate the subset of imagenet which I have noted in the readme.md file, the results are similar. You need to write your own sampling file. I have not encountered other situations.

[yuhuixu1993]

Hi,guys, I run the train_imagenet.py myself. The code runs well in the first two epochs and I have attached the log. I only change the batchsize and lr into 256 and 0.1 due to the limit of GPU. log.txt