Reading Location files gives unhelpful error

[michaelbornholdt]

I have managed to either corrupt my location files or my index.csv in such a way that this error appears. Unfortunately it is quite unhelpful since one does not know which files are missing or where the problem lies.

Ideally, DP would check if location files exist and include the missing file or values in the error message.

None
{'BRD-K68867920-051-11-5': 0, 'BRD-K74363950-004-01-0': 1, 'BRD-K75958547-238-01-0': 2, 'DMSO': 3}
Reading single-cell locations
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "deepprofiler/__main__.py", line 197, in <module>
    cli(obj={})
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "deepprofiler/__main__.py", line 162, in train
    dset = deepprofiler.dataset.image_dataset.read_dataset(context.obj["config"], mode='train')
  File "/DeepProfiler/deepprofiler/dataset/image_dataset.py", line 243, in read_dataset
    dset.prepare_training_locations()
  File "/DeepProfiler/deepprofiler/dataset/image_dataset.py", line 74, in prepare_training_locations
    locations = pd.concat(locations)
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/reshape/concat.py", line 284, in concat
    sort=sort,
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/reshape/concat.py", line 331, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate

[michaelbornholdt]

@Arkkienkeli @jccaicedo

[michaelbornholdt]

So this error occurs when your config is not in sync with the index file. More precisely: My Split column in the index file had values of 'Test' and 'Training'

My config however had the line: training_values": ["Train"]

This leads to an empty dataframe being handed to this line: https://github.com/cytomining/DeepProfiler/blob/master/deepprofiler/dataset/image_dataset.py#L73

This specific error would need to be changed to an assert error which warns the user of this mismatch.

[michaelbornholdt]

This is an example of bad error messages. I think we should maybe open a thread to gather these mistakes and and some point maybe build better error logs. Your call @jccaicedo