Confusion with 'status' and 'phase'

[mfriendly]

Hi:)

First of all, I thank you so much for the codes and the insightful research. Im doing a study on time series using Neural Architecture Search and your repository is helping greatly .

---

I have some questions regarding phases and status in your code though:(

Firstly, Where it says, (in BM-NAS/models/search/train_searchable/ego.py)

phases = [] if status == 'search': phases = ['train', 'dev'] else: phases = ['train', 'test']

model.load_state_dict(best_test_model_sd)

-- shouldnt the phases be just ['test'] for 'else'? (=status being 'Eval') I couldnt understand why 'train' phase was included.:(

Another similar point that confused me was that in the scripts training the found architectures,

(For example, in the arguments of https://github.com/Somedaywilldo/BM-NAS/blob/master/main_darts_found_mmimd.py)

# loading searching experiment, if not None, perform evalution
parser.add_argument('--search_exp_dir', type=str, help='evaluate which search exp', default=None)
# loading evaluation experiment, if not None, perform test
parser.add_argument('--eval_exp_dir', type=str, help='test which eval exp', default=None)

What do 'perform evaluation' and 'perform test' mean here, respectively?

'Evaluation' seemed to actually mean 'train' (thay is, training the found architecture) if I look at the codes, so i was confused again with the use of the words.:(

Thank you again for the great model, and Id be very grateful for your answer.:)

[Somedaywilldo]

Hi, sorry for letting you confuse. In the searching stage, the training means to train the hypernet, the evaluation means evaluating the hypernet, when searching stage is finished, you will get a genotype, which is the description of the structure. Then after the genotype is found, the evaluation here means evaluating the genotype, that means to build a fusion network based on this genotype and train it, so you are getting it right. Then the test is just the evaluation of the trained fusion network.

[mfriendly]

Thanks for the reply! so to clarify, there are 1. searching stage(seraching for best architectur) /2. eval stage (meaning training the structure found in the searching stage)

Eval stage is divided into two cases , one is when there exists 'eval exp directory' -> train/ val steps of 'eval'is already performed, and we need to test it, And when there's no 'eval exp directory' -> we need to train / validate the found structure .

Am i getting it right?/

[Somedaywilldo]

Yes, you're getting it right, if you don't have further question I may close this issue.