changlin31
commented
2 years ago Hi @zhy0860 , thanks for your interest!
-
Sorry for the confusion. In Eq. (5) and (6), the notation S is only used to denote that we only use one single network (either online or the target) for evaluation loss, in contrast to the training stage, where a pair of siamese networks are used. In practice, the target network usually have better performance. So we directly use it as our supernet for eval. (In self-supervised learning, it's also common to use the target network for evaluation results)
-
We don't use EA as our search space is small enough for extensive traversal search, in each stage, as mentioned in the paper:
-
This is just a heuristic setting to reduce the search space. You can set them back to [4,4,4,4] if you want to explore the whole search space. Actually, these networks with larger scales is not likely to be chosen even if the whole search space is enabled.
zhy0860
Hi, great work!
I have some question about the code and paper:
In section 3.3 of the paper which is about the searching phase, when calculating the evaluation loss in equation(5) and (6) the probability ensemble of the architecture population is from the online network, but in the code it's from the target network, which makes me confused.
Still in section 3.3, it is mentioned that the searching are with an evolutionary algorithm, I read the references[12] and [54] but still have no clue how the evolutionary algorithm is implemented in the code, to be specific, how the architecture population is evolved?
In the code of hytra_supernet.py, the stage depths are set to [4,3,2,2], is there a particular reason to set so? why not use [4,4,4,4] so that all possible pathes can be chosen?
Thanks a lot for your time and I'm looking forward to your reply!