JSON and BIDS-Prov

[cmaumet]

Update proposal for BIDS Prov (BEP028)

By @yarikoptic in https://github.com/bids-standard/BEP028_BIDSprov/issues/125#issuecomment-2492133921

• dissolving "Justification for Separating Provenance from file JSON" section by allowing generatedBy to be specified in the corresponding .json file, potentially with further relaxations such as not demanding id (assume to be unique) and overall have a clear schema which we could encode in our BIDS schema and validate

[yarikoptic]

BTW, this would also be somewhat aligned with the effort of @surchs et al in https://github.com/OpenNeuroDatasets-JSONLD/ where instead of aiming at directly equipping openneuro BIDS datasets with .jsonld files "directly", they work on improving description of phenotypic data within JSON files so then there could be improved (over original BIDS) .jsonld files "derived" from the JSON files metadata. In other words: they are concentrating on the description of metadata in more concise/to the point way, relying on @context etc to be "encoded" elsewhere, to then be able to render the enriched .jsonld files. I think that is the path which BEP028 should allow for as well: to provide a way to "enter" provenance in concise, more "human accessible" form, to later be able to produce the "standard" prov.jsonld files. IMHO some additional aspects worth adding to "design" here

• *prov.jsonld across different levels (#144 , #145) would still be allowed but validator must be able to detect possible conflicts and incongruencies and warn or error
  • TODO: identify common scenarios like that (e.g. records with the same id but conflicting metadata -- different other fields etc)
• should be sufficiently expressive to accommodate current GeneratedBy (#148)
• schema for JSON records would be defined within BIDS schema (here is current for GeneratedBy) to be validatable by stock bids-validator
• develop good set of examples for already present common use-cases e.g.
  • conversion software(s) version specification -- e.g. for dcm2niix and heudiconv
  • additional manual annotation
• cool feature idea (good-for-hackathon style)
  • develop tool to automatically extract/enrich based on git history. Might already be in part addressed by https://docs.datalad.org/projects/metalad/en/latest/generated/datalad_metalad.extractors.runprov.html#module-datalad_metalad.extractors.runprov but I thought there was even more ...

edit 1: IMHO think this issue is one of the most important aspects to facilitate adoption.