dev harness fails with IndexError: list index out of range when running MNIST sweep

[jlema]

After 4 minutes of running mnist on the harness I get IndexError: list index out of range. The sweep ends and the status bar (% complete) dissapears.

127.0.0.1 - - [13/Nov/2020 13:22:55] "GET /api/v1/sweep_status HTTP/1.1" 200 - Exception in thread Thread-5: Traceback (most recent call last): File "c:\users\v-julema\miniconda3\envs\dropdowns\lib\threading.py", line 916, in _bootstrap_inner self.run() File "c:\users\v-julema\miniconda3\envs\dropdowns\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "c:\users\v-julema\git\backwardcompatibilityml\backwardcompatibilityml\helpers\training.py", line 798, in compatibility_sweep device=device) File "c:\users\v-julema\git\backwardcompatibilityml\backwardcompatibilityml\helpers\training.py", line 562, in evaluate_model_performance_and_compatibility_on_dataset get_instance_metadata=get_instance_metadata, device=device) File "c:\users\v-julema\git\backwardcompatibilityml\backwardcompatibilityml\helpers\training.py", line 356, in get_all_error_instance_indices error_instances_metadata = list(map(get_instance_metadata, error_instance_ids)) File "C:\Users\v-julema\git\BackwardCompatibilityML\development\app.py", line 197, in get_instance_label label = data_loader[instance_id][2].item() IndexError: list index out of range