Problem:
When running the corruptions benchmark on either of the CIFAR datasets I encounter the following error:
Traceback (most recent call last):
File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
return self._engine.get_loc(casted_key)
File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 191, in pandas._libs.index.IndexEngine.get_loc
File "index.pyx", line 234, in pandas._libs.index.IndexEngine._get_loc_duplicates
File "index.pyx", line 242, in pandas._libs.index.IndexEngine._maybe_get_bool_indexer
File "index.pyx", line 134, in pandas._libs.index._unpack_bool_indexer
KeyError: 'AlexNet'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/labor/Schreibtisch/robustbench/claim.py", line 18, in <module>
clean_acc, robust_acc = benchmark(
File "/home/labor/Schreibtisch/robustbench/robustbench/eval.py", line 130, in benchmark
adv_accuracy, adv_mce = corruptions_evaluation(
File "/home/labor/Schreibtisch/robustbench/robustbench/eval.py", line 211, in corruptions_evaluation
alexnet_accs_corruption = full_results.loc['AlexNet']
File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
return self._getitem_axis(maybe_callable, axis=axis)
File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
return self._get_label(key, axis=axis)
File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexing.py", line 1381, in _get_label
return self.obj.xs(label, axis=axis)
File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/generic.py", line 4301, in xs
loc = index.get_loc(key)
File "/home/labor/Schreibtisch/robustbench/.venv/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
raise KeyError(key) from err
KeyError: 'AlexNet'
This is a result of this commit where adv_mce was introduced for ImageNet in eval.py. Due to the missing results for AlexNet in the unaggregated_results.csv of CIFAR-10 and CIFAR-100 corruptions benchmark a KeyError is raised.
Possible Solution:
As suggested by Hendrycks et al. themselves "any baseline will do (even a baseline with 100% error rates, corresponding to an average of CEs)" for calculating the mCE. Hence, the easiest solution would be to insert a dummy entry for AlexNet in the unaggregated_results.csv of CIFAR-10 and CIFAR-100 corruptions with 0.0 for each corruption and severity. The resulting adv_mce then simply corresponds to 1 - adv_accuracy.
Problem: When running the corruptions benchmark on either of the CIFAR datasets I encounter the following error:
This is a result of this commit where
adv_mcewas introduced for ImageNet ineval.py. Due to the missing results for AlexNet in theunaggregated_results.csvof CIFAR-10 and CIFAR-100 corruptions benchmark a KeyError is raised.Possible Solution: As suggested by Hendrycks et al. themselves "any baseline will do (even a baseline with 100% error rates, corresponding to an average of CEs)" for calculating the mCE. Hence, the easiest solution would be to insert a dummy entry for AlexNet in the
unaggregated_results.csvof CIFAR-10 and CIFAR-100 corruptions with 0.0 for each corruption and severity. The resultingadv_mcethen simply corresponds to 1 -adv_accuracy.