Open galshachaf opened 5 years ago
@galshachaf Could you give me some example use case? Give me some python pseudocode so I know how you want to use this with existing classes. Note that the way I see it there are 3 (mutually exclusive) options:
How do you want to control when the experiment loads previous data or not?
Output a single dict JSON file, load it once and update throughout the experiment
evaluation takes a lot of time, should be able to load it from previous runs. when we run calc_full_rebustness_results, we get a "results" dict. We should add an "evaluation_results" attribute to ResourceManager per model, which will initialize to an empty dict. When trying to get results per layer_indices_list, we first try to look it in this dictionary. If not found, we will evaluate and update this dict. (and save it)