datalad / datalad

Keep code, data, containers under control with git and git-annex
http://datalad.org
Other
527 stars 109 forks source link

duecredit support: prevent warnings about datasets without aggregated metadata #4562

Closed yarikoptic closed 4 years ago

yarikoptic commented 4 years ago

They come up even multiple times, flooding the screen etc:

DUECREDIT_ENABLE=1 heudiconv -o /tmp/out -f reproin --datalad --files QA/sourcedata/sub-qa/ses-20190930/anat/sub-qa_ses-20190930_acq-MPRAGE_T1w.dicom.tgz
[sudo] password for yoh: 
2020-05-21 12:08:10,622 [WARNING] Could not find fsl in module <module 'nipype.interfaces' from '/usr/lib/python3/dist-packages/nipype/interfaces/__init__.py'>: module 'nipype.interfaces' has no attribute 'fsl' (injector.py:210)
WARNING: Could not find fsl in module <module 'nipype.interfaces' from '/usr/lib/python3/dist-packages/nipype/interfaces/__init__.py'>: module 'nipype.interfaces' has no attribute 'fsl'
2020-05-21 12:08:10,622 [WARNING] Could not find spm in module <module 'nipype.interfaces' from '/usr/lib/python3/dist-packages/nipype/interfaces/__init__.py'>: module 'nipype.interfaces' has no attribute 'spm' (injector.py:210)
WARNING: Could not find spm in module <module 'nipype.interfaces' from '/usr/lib/python3/dist-packages/nipype/interfaces/__init__.py'>: module 'nipype.interfaces' has no attribute 'spm'
INFO: Running heudiconv version 0.8.0 latest 0.8.0
INFO: Analyzing 192 dicoms
INFO: Filtering out 0 dicoms based on their filename
INFO: Generated sequence info for 1 studies with 1 entries total
INFO: Processing sequence infos to deduce study/session
INFO: Considering study (43b67d9139e8c7274578b7451ab21123) specific substitutions
INFO: Study session for StudySessionInfo(locator='dbic/QA', session=None, subject='qa')
INFO: Need to process 1 study sessions
INFO: Initiating <Dataset path=/tmp/out>
[INFO   ] Creating a new git repo at /tmp/out 
INFO   : Creating a new git repo at /tmp/out
[WARNING] Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry
INFO: Initiating <Dataset path=/tmp/out/dbic>
[INFO   ] Creating a new git repo at /tmp/out/dbic 
INFO   : Creating a new git repo at /tmp/out/dbic
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry
INFO: Initiating <Dataset path=/tmp/out/dbic/QA>
[INFO   ] Creating a new annex repo at /tmp/out/dbic/QA 
INFO   : Creating a new annex repo at /tmp/out/dbic/QA
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/QA/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/QA/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic/QA)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic/QA)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic/QA>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic/QA>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry
INFO: PROCESSING STARTS: {'subject': 'qa', 'outdir': '/tmp/out/dbic/QA', 'session': None}
....
INFO: [Node] Finished "convert".
INFO: PROCESSING DONE: {'subject': 'qa', 'outdir': '/tmp/out/dbic/QA', 'session': None}
[WARNING] Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/QA/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/QA/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic/QA)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic/QA)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic/QA>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic/QA>. Will not provide duecredit entry
[INFO   ] Creating a new annex repo at /tmp/out/dbic/QA/.heudiconv 
INFO   : Creating a new annex repo at /tmp/out/dbic/QA/.heudiconv
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/QA/.heudiconv/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/QA/.heudiconv/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic/QA/.heudiconv)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic/QA/.heudiconv)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic/QA/.heudiconv>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic/QA/.heudiconv>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry
[WARNING] Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally. 
WARNING: Found no aggregated metadata info file /tmp/out/dbic/.datalad/metadata/aggregate_v1.json. You will likely need to either update the dataset from its original location or reaggregate metadata locally.
[WARNING] Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)] 
WARNING: Dataset at . contains no aggregated metadata on this path [metadata(/tmp/out/dbic)]
[WARNING] Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry 
WARNING: Failed to obtain metadata for <Dataset path=/tmp/out/dbic>. Will not provide duecredit entry

DueCredit Report:
- DataLad - Data management and distribution platform / datalad (v 0.12.6) [1]
- Flexible DICOM converter for organizing brain imaging data / heudiconv (v 0.8.0) [2]
- I/O library to access to common neuroimaging file formats / nibabel (v 3.0.1) [3]
- Scientific tools library / numpy (v 1.17.4) [4]

4 packages cited
0 modules cited
0 functions cited

References
----------

[1] Halchenko, Y.O. et al., 2019. datalad/datalad 0.12.0rc6.
[2] Halchenko, Y. et al., 2020. nipy/heudiconv v0.8.0.
[3] Brett, M. et al., 2015. Nibabel 2.0.1.
[4] Van Der Walt, S., Colbert, S.C. & Varoquaux, G., 2011. The NumPy array: a structure for efficient numerical computation. Computing in Science & Engineering, 13(2), pp.22–30.
DUECREDIT_ENABLE=1 heudiconv -o /tmp/out -f reproin --datalad --files   15.50s user 8.34s system 104% cpu 22.917 total
yarikoptic commented 4 years ago

fixed by #4568