Parse search results to get schedules

[snownus]

I try to run the search example, the result is show in as follows: experiment_state-2019-07-04_11-18-12.json pbt_global.txt pbt_policy_0.txt pbt_policy_10.txt pbt_policy_11.txt pbt_policy_12.txt pbt_policy_13.txt pbt_policy_14.txt pbt_policy_15.txt pbt_policy_1.txt pbt_policy_2.txt pbt_policy_3.txt pbt_policy_4.txt pbt_policy_5.txt pbt_policy_6.txt pbt_policy_7.txt pbt_policy_8.txt pbt_policy_9.txt RayModel_0_2019-07-04_11-18-12X5ljyD RayModel_10_2019-07-04_11-19-03P76U0K RayModel_11_2019-07-04_11-19-05drsWQC RayModel_1_2019-07-04_11-18-12AArW21 RayModel_12_2019-07-04_11-19-06UP0tWZ RayModel_13_2019-07-04_11-19-076AxfVX RayModel_14_2019-07-04_11-19-070uF1rg RayModel_15_2019-07-04_11-19-26GJtjgM RayModel_2_2019-07-0411-18-12QQx2i RayModel_3_2019-07-04_11-18-12KnTGqv RayModel_4_2019-07-04_11-18-12Wdvaqo RayModel_5_2019-07-04_11-18-38bTvB_m RayModel_6_2019-07-04_11-18-39YjVHQe RayModel_7_2019-07-04_11-18-39e7EF_T RayModel_8_2019-07-04_11-18-39YhkcS0 RayModel_9_2019-07-04_11-18-39_Q4q5a

The readme file hasn't mentioned how to use these generated results to get the schedules. The log file in RayModel*** is empty. Can you mention more details how to use the pba/util.py to parse the result file? @arcelien Many Thanks!

[arcelien]

You can use the raw pbt_policy_*.txt files in train.py for training by passing the flag: --hp_policy=[path to pbt_policy_(trial_id).txt]. The trial_id corresponds to the trial of the schedule you want to use and also the index in tensorboard. It will be parsed automatically.

If you want to visualize the schedule, you can see the end of pba/utils.py for an example, and just change the path from 'schedules/rsvhn_16_wrn.txt' to your schedule file.

[snownus]

@arcelien , Thanks for your instant reply! When I investigate the contents of pbt_policy_*.txt, it looks like: ["0", "14", 5, 3, {"test_batch_size": 25, "data_path": "", "use_hp_policy": true, "dataset": "svhn", "no_aug": false, "gradient_clipping_by_global_norm": 5.0, "weight_decay_rate": 0.005, "num_epochs": 160, "init_epoch": 0, "main_scope": "Student", "train_size": 1000, "lr": 0.1, "wrn_depth": 40, "wrn_size": 32, "batch_size": 128, "validation_size": 7325, "explore": "cifar10", "recompute_dset_stats": false, "aug_policy": "cifar10", "no_cutout": true, "train_dir": null, "Distillation": "DWL", "model_name": "wrn"}, {"data_path": "", "use_hp_policy": true, "dataset": "svhn", "no_aug": false, "gradient_clipping_by_global_norm": 5.0, "weight_decay_rate": 0.005, "num_epochs": 160, "init_epoch": 0, "test_batch_size": 25, "train_size": 1000, "lr": 0.1, "wrn_depth": 40, "wrn_size": 32, "batch_size": 128, "validation_size": 7325, "explore": "cifar10", "hp_policy": [0, 3, 9, 0, 2, 0, 2, 0, 0, 1, 2, 4, 0, 0, 6, 0, 0, 7, 0, 0, 10, 0, 1, 0, 0, 3, 0, 0, 0, 0, 4, 1, 9, 1, 3, 0, 2, 0, 0, 0, 3, 1, 0, 0, 0, 2, 3, 0, 1, 2, 0, 2, 0, 2, 0, 2, 0, 0, 0, 0], "recompute_dset_stats": false, "aug_policy": "cifar10", "no_cutout": true, "train_dir": null, "model_name": "wrn"}],

I think the pbt_policy_*.txt is the input of pba/utils.py, am I right? When I use pbt_policy_*.txt as the input of pba/utils.py, it has error: raw_policy = [ast.literal_eval(line) for line in raw_policy] File "/usr/lib/python2.7/ast.py", line 80, in literal_eval return _convert(node_or_string) Is there anything wrong with my understanding?

[arcelien]

Sorry for the late reply, I just pushed a change that should fix the issue you're seeing. If it doesn't, could you upload the policy.txt file you're using?

[snownus]

@arcelien , yes, it works now. Thanks for your reply!

So I need to convert the generated pbt_policy_15.txt to the format of schedules/rsvhn_16_wrn.txt, and then use it into the train.py to evaluate, right? If it is, I suggest adding a script to convert the generated policy file.

[arcelien]

No, you should be able to use your generated policy file with no conversion.

In the new version, the entire set of hyper parameters is saved for flexibility (instead of just the augmentation ones), but the code will work fine with either.

[snownus]

@arcelien ， yes, it works now.

Thanks a lot!