Upcoming datalad 0.10 will not be able to set metadata

[yarikoptic]

With @mih we reached an agreement to retract ability to set metadata associated with the files, so currently heudiconv would break if ran with master of datalad:

INFO   : Creating a new annex repo at /home/yoh/picts/mris/dartmouth-phantoms/OUTPUT/Halchenko/Yarik/950_bids_test4/.heudiconv
Traceback (most recent call last):                                                                       
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/bin/heudiconv", line 11, in <module>
    load_entry_point('heudiconv==0.4', 'console_scripts', 'heudiconv')()
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/heudiconv/cli/run.py", line 120, in main
    process_args(args)
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/heudiconv/cli/run.py", line 343, in process_args
    add_to_datalad(outdir, study_outdir, msg, args.bids)
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/heudiconv/external/dlad.py", line 115, in add_to_datalad
    mark_sensitive(ds, 'sourcedata')
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/heudiconv/utils.py", line 144, in mark_sensitive
    ds.metadata(recursive=True, **sens_kwargs)
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/datalad/distribution/dataset.py", line 437, in apply_func
    return f(**kwargs)
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/datalad/interface/utils.py", line 463, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/datalad/interface/utils.py", line 451, in return_func
    results = list(results)
  File "/home/yoh/picts/mris/heudiconv/venvs/dev/local/lib/python2.7/site-packages/datalad/interface/utils.py", line 405, in generator_func
    wrapped(*_args, **_kwargs),
TypeError: __call__() got an unexpected keyword argument 'init'

so fix is needed to manipulate git-annex metada directly via AnnexRepo.set_metadata while avoiding datalad.api.metadata call

[satra]

@yarikoptic - so if we have nested git annex repos, we will need to find each one and set metadata explicitly? seems like datalad should do this, since datalad intends to be a bridge across multiple git-annex repos.

[yarikoptic]

Here specifically it is just a matter of replacing one datalad call with another one. Nothing will need to be set manually, nothing will change to the user.

Or are your asking more generic datalad question? Please give use case

[satra]

@yarikoptic - replacing a datalad call with a git-annex call right? i think in the heudiconv context for the moment it may be fine.

i'm just thinking more generally of datalad datasets, wouldn't there be an api to read and write metadata at any level?

[mih]

Cherrio!

Datalad 0.10+ will have an API to read arbitrary metadata on any level. The "write" side is not so easy. So far the metadata command pretended that file-based metadata could be set with it, but that was ultimately only true for annexed files, and only with the limitations of annex (size, and key composition). Datalad 0.10 forcefully ends this dream :(

A generic "write anything, of any size with any structure" is certainly doable, but arguably of less value than adhering to a defined metadata standard -- and I say that without having a good idea on what kind of metadata heudiconv is trying to write...