Api.py: TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

[PhMueller]

Hey,

first, thanks for your awesome work.

Describe the bug

While trying NAS-Bench-201, I stumbled across a bug, which occurs calling the get_more_info()-function with an old benchmark file. I am using the benchmark file NAS-Bench-201-v1_0-e61699.pth from here.

To Reproduce

from nas_201_api import NASBench201API as API

api = API('./NAS-Bench-201-v1_0-e61699.pth')
result_dict = api.get_more_info(index=0,
                                dataset='cifar10-valid',
                                iepoch=11,
                                use_12epochs_result=False,
                                is_random=False)

Output: Traceback (most recent call last): File "", line 10, in File "/home/philipp/anaconda3/envs/HPOlib3/lib/python3.6/site-packages/nas_201_api/api.py", line 293, in get_more_info 'train-per-time': train_info['all_time'] / total, TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'

The error is caused by a missing None-value check here: https://github.com/D-X-Y/AutoDL-Projects/blob/91ee265bd2468e98fd30cf423f1a5d1c196d6000/lib/nas_201_api/api.py#L291-L294

Used versions Benchmark file: [2020.02.25] APIv1.0/FILEv1.0: NAS-Bench-201-v1_0-e61699.pth nas-bench-201: 1.3 python: 3.6

Expected behavior This benchmark does not contain values for the fields all_time in the infos dict for train, test and valid. I guess a simple check if the time value is None should be sufficient to fix this. The "old" code had this None-value check.

Thanks in advance

[D-X-Y]

Thanks for pointing out this issue!

1, I would recommend you to use our newest benchmark file (NAS-Bench-201-v1_1-096897.pth): https://drive.google.com/file/d/16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_/view?usp=sharing , where all all-time are avaliable.

2, I will make a major change to the API codes soon, where this issue will be fixed. I will post the news here once I push the new codes.

Best regards,

[D-X-Y]

@PhMueller I just pushed the new codes, you can have a try. With this new code, you can use the following to query:

from nas_201_api import NASBench201API as API

api = API('./NAS-Bench-201-v1_0-e61699.pth')
result_dict = api.get_more_info(index=0,
                                dataset='cifar10-valid',
                                iepoch=11,
                                hp='200',
                                is_random=False)

where hp='200' indicates using the hyper-parameters of 200 epoch training.

[PhMueller]

Thanks a lot!

[D-X-Y]

You are welcome~