datalad / datalad-metalad

Next generation metadata handling
Other
11 stars 11 forks source link

User unfriendly error message when `metalad_studyminimeta` extractor fails #150

Open jsheunis opened 2 years ago

jsheunis commented 2 years ago

With both meta-extract and meta-conduct, if they're run with the metalad_studyminimeta extractor on a dataset that doesn't have a .studyminimeta.yaml file locally available, the error messages are not informative.

Example 1:

>> datalad meta-extract -d /Users/jsheunis/Documents/psyinf/Data/super metalad_studyminimeta
Studyminimeta metadata extraction:   0%|

Example 2:

>> datalad -f json meta-extract -d /Users/jsheunis/Documents/psyinf/Data/super metalad_studyminimeta

{"action": "meta_extract", "message": "file /Users/jsheunis/Documents/psyinf/Data/super/.studyminimeta.yaml could not be opened", "path": "/Users/jsheunis/Documents/psyinf/Data/super", "status": "error", "type": "dataset"}
Studyminimeta metadata extraction:   0%|

Example 3:

>> datalad -f json meta-conduct /Users/jsheunis/Documents/psyinf/datalad-catalog/datalad_catalog/examples/extract_dataset_metadata.json \
    traverser:/Users/jsheunis/Documents/psyinf/Data/super \
    traverser:dataset \
    extractor:Dataset \
    extractor:metalad_studyminimeta

Studyminimeta metadata extraction:   0%|                                                                                      | 0.00/37.0 [00:00<?, ? Files/s][ERROR  ] Exception None. 1 failed:
| [{'action': 'meta_extract',
|   'message': 'file '
|              '/Users/jsheunis/Documents/psyinf/Data/super/.studyminimeta.yaml '
|              'could not be opened',
|   'path': '/Users/jsheunis/Documents/psyinf/Data/super',
|   'status': 'error',
|   'type': 'dataset'}] in processor <Future at 0x7fd73e5fa910 state=finished raised IncompleteResultsError>
{"action": "meta_conduct", "message": "concurrent.futures.process._RemoteTraceback: \n\"\"\"\nTraceback (most recent call last):\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/concurrent/futures/process.py\", line 243, in _process_worker\n    r = call_item.fn(*call_item.args, **call_item.kwargs)\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/site-packages/datalad_metalad/processor/base.py\", line 26, in execute\n    return context, self.process(pipeline_element)\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/site-packages/datalad_metalad/processor/extract.py\", line 85, in process\n    for extract_result in meta_extract(**kwargs):\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/site-packages/datalad/interface/utils.py\", line 483, in eval_func\n    return return_func(generator_func)(*args, **kwargs)\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/site-packages/datalad/interface/utils.py\", line 476, in return_func\n    results = list(results)\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/site-packages/datalad/interface/utils.py\", line 461, in generator_func\n    raise IncompleteResultsError(\ndatalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:\n[{'action': 'meta_extract',\n  'message': 'file '\n             '/Users/jsheunis/Documents/psyinf/Data/super/.studyminimeta.yaml '\n             'could not be opened',\n  'path': '/Users/jsheunis/Documents/psyinf/Data/super',\n  'status': 'error',\n  'type': 'dataset'}]\n\"\"\"\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/site-packages/datalad_metalad/conduct.py\", line 281, in process_parallel\n    source_index, pipeline_element = future.result()\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/concurrent/futures/_base.py\", line 438, in result\n    return self.__get_result()\n  File \"/Users/jsheunis/opt/miniconda3/envs/catalognew/lib/python3.9/concurrent/futures/_base.py\", line 390, in __get_result\n    raise self._exception\ndatalad.support.exceptions.IncompleteResultsError: None. 1 failed:\n[{'action': 'meta_extract',\n  'message': 'file '\n             '/Users/jsheunis/Documents/psyinf/Data/super/.studyminimeta.yaml '\n             'could not be opened',\n  'path': '/Users/jsheunis/Documents/psyinf/Data/super',\n  'status': 'error',\n  'type': 'dataset'}]\n", "status": "error"}
jsheunis commented 2 years ago

Or to rephrase, the informative parts of the error messages are nested deep, and not immediately evident.