Closed Remi-Gau closed 3 months ago
Also the command too should probably be updated to something like this
DATASET_PATH="path/to/mydataset"
PIPELINE_PATH="path/to/extract_dataset_pipeline.json"
datalad meta-conduct "$PIPELINE_PATH" \
traverser.top_level_dir=$DATASET_PATH \
traverser.item_type=dataset \
traverser.traverse_sub_datasets=True \
extractor1.extractor_type=dataset \
extractor1.extractor_name=metalad_core \
extractor2.extractor_type=dataset \
extractor2.extractor_name=metalad_studyminimeta \
adder.aggregate=True
Though when running it with this I get:
[ERROR ] 'list' object is not a mapping
and when relying on one of the example from metalad I get this:
datalad meta-conduct \
extract_metadata \
traverser.top_level_dir=$DATASET_PATH \
traverser.item_type=file \
traverser.traverse_sub_datasets=True \
extractor.extractor_type=file \
extractor.extractor_name=metalad_example_file \
adder.aggregate=True
[ERROR ] A child process terminated abruptly, the process pool is not usable anymore
@Remi-Gau So sorry that I missed this! not sure how that happened...
I will try and reproduce this, and update docs to the latest correct functionality.
@Remi-Gau I think the error you got:
[ERROR ] 'list' object is not a mapping
was because the value of the arguments
keyword should be a dictionary rather than a list. So the pipeline object should rather be:
{
"provider": {
"module": "datalad_metalad.provider.datasettraverse",
"class": "DatasetTraverser",
"name": "traverser",
"arguments": {}
},
"processors": [
{
"module": "datalad_metalad.processor.extract",
"class": "MetadataExtractor",
"name": "extractor1",
"arguments": {}
},
{
"module": "datalad_metalad.processor.extract",
"class": "MetadataExtractor",
"name": "extractor2",
"arguments": {}
},
{
"name": "adder",
"module": "datalad_metalad.processor.add",
"class": "MetadataAdder",
"arguments": {}
}
]
}
This worked for me.
I updated the docs accordingly in this PR: https://github.com/datalad/datalad-catalog/pull/258
Let me know if this solves your issue.
Was trying to follow this example
http://docs.datalad.org/projects/catalog/en/latest/pipeline_description.html
Running this
with the suggested pipeline
gives this error
Checking on the structure of the package:
https://github.com/datalad/datalad-metalad/tree/master/datalad_metalad/pipeline/provider
it seems that the json should have things like
datalad_metalad.pipeline.processor.extract