zhangjiajin
commented
2 years ago @impulsecorp
CARS will evolve and retain only the best 8 network architectures. The current 8 best network architectures may not be the best.
Very good issue.
We intend to bring the best network architecture into every evolution, not just abandon them. We'll do that as soon as possible.
I am using a custom dataset with CARS and it runs fine, but I am confused because it shows a different accuracy at the end of the search part than during the search itself. During the 500 epochs of searching it for example finds this NN with 89% Val accuracy: 2022-06-08 03:07:34.605 INFO ############## KEEP ############ 2022-06-08 03:07:34.609 INFO KEEP Valid_acc for invidual 1 0.890625, size 0.111890, genotype Genotype(normal=[['max_pool_3x3', 2, 0], ['max_pool_3x3', 2, 1], ['max_pool_3x3', 3, 0], ['max_pool_3x3', 3, 1], ['max_pool_3x3', 4, 2], ['max_pool_3x3', 4, 0], ['avg_pool_3x3', 5, 0], ['avg_pool_3x3', 5, 1]], normal_concat=range(2, 6), reduce=[['sep_conv_5x5', 2, 0], ['max_pool_3x3', 2, 1], ['avg_pool_3x3', 3, 0], ['skip_connect', 3, 1], ['max_pool_3x3', 4, 1], ['max_pool_3x3', 4, 0], ['avg_pool_3x3', 5, 0], ['max_pool_3x3', 5, 1]], reduce_concat=range(2, 6))
but then at the end of the search it shows the best NN had 84% accuracy: 022-06-08 03:11:18.300 INFO worker id [0], epoch [500/500], train step [10/17], loss [ 0.004, 0.100], lr [ 0.0010004], time pre batch [0.582s] , total mean time per batch [0.569s] 2022-06-08 03:11:23.176 INFO Finished the unified trainer successfully. 2022-06-08 03:11:23.191 INFO Update Success. step_name=nas, worker_id=0 2022-06-08 03:11:23.194 INFO Best values: [{'worker_id': 2, 'performance': {'accuracy': 0.84375}}] 2022-06-08 03:11:23.218 INFO Clean worker folder /content/tasks/0608.012736.396/workers/nas.
Is this because it is picking the best NNs based on the Val score, and then at the end showing what the Test accuracy was for it?