SNIRF independent entries / groups in metaDataTags

[MichaelUM]

Does the snirf format support to add individual fields that are not defined in the snirf format or would this cause an incompatibility with the format? I'm thinking about more detailed descriptions of the preprocessing that may was performed. Or more experimental / participant / manufacturer / device information that are currently not part of the snirf format.

I could already store additional information in /nirs(i)/metaDataTags/ but as far as I understood it this tag doesn't support groups but only datasets if I'm not mistaken ("Each metadata record is represented as a dataset...") . Or would it be OK to also store groups within the metaDataTags group?

Thanks for the clarification.

[fangq]

@MichaelUM, this specification does not specify the format or behaviors of the fields outside of the SNIRF-related fields. Because HDF5 is general purpose, I don't see a problem if one attaches additional customized data entries along the SNIRF data structure, but one must be aware that SNIRF-aware toolboxes may not be able to recognize or process such information if they are not part of the standard.

regarding metaDataTag, the current language does exclude the ability to store a group under an entry, although I think the current data loaders/validators do not treat this strictly. I don't mind extending the language to "dataset or a group" for metaDataTag entries, the risk is that if a user "abuses" this feature, it can make the metaDataTag field bulky and less efficient to process.

on the other hand, if additional data structures/metaDataTag entries are deemed necessary for a particular type of experiment or study, we would love to hear about such need and perhaps include those into this spec in the future releases.

[dboas]

@MichaelUM , I agree with @fangq . The SNIRF file is generally meant for a single data set. I do understand the desire for that data set to carry information about the subject and the group in which it is contained. The inclusion of information about the subject and group is being addressed in the BIDS-fNIRS specification. This is still be developed. So take a look at it here and comment about it. This BIDS-fNIRS should specify all the capabilities you are looking for and becoming a standard approach for doing so should mean that future software will be able to read it.

[MichaelUM]

@fangq @dboas thanks for the feedback. We are currently thinking about adding pre-processing information to the SNIRF file to address the problem of sharing processed data. If we would just use the available fields we would loose specific information to be able to reproduce the data from raw wl. So to not blow up the metaDataTag to much we thought of adding a software specific group (if the language can be adapted to allow gorups) and include the detailed information about e.g. pre-processing in here. Therefore it would be great if there would be a clear definition where to store additional information that are not covered by the SNIRF yet and might be added in a later version to the format. For bids we would of course end up in the derivatives folders but also here a definition bout the applied processing steps would be essential. But this might be resolved using sidecar containers but thanks for the link, I'll check it regularly and try to participate.

So we didn't consider to add multiple datasets in one file. Even though it is possible to have 1 or more NIRS datasets in the Root-group or 1 or more data blocks in the specific root group in the SNIRF format. This would be more maintenance work then benefit so that we decided to store the data in separate SNIRF files.

[fangq]

@dboas, by allowing "group" in metaDataTag, @MichaelUM did not mean the study population group, but a HDF5 group (if I understood it correctly) - which is basically a struct/cell-array if mapped to MATLAB.

for example, should we allow someone to store	Metadata Key Name	Metadata value
ManufacturerName	"Company Name"
Model	"Model Name"
SubjectName/	FirstName: "firstname"
	MiddleName: ""
	LastName: "lastname"
Address/	Street: "149 13th St"
	City: "Charlestown"
	State: "Massachusetts"
	Country: "USA"
Preprocessing/	Name: "low-pass filter"
	Order: 25
	FilterType: "butterworth"
	CutOffFrequency: 20

in metaDataTags? currently, we only specified that each entry in this list must be a dataset, which does not allow to store a struct-like record like above.

[dboas]

@fangq , thanks for clarifying this for me. Should we modify the spec to allow for groups within the metaDataTags? What impact will that have on current readers and writers? Is it backward compatible?

[MichaelUM]

@dboas @fangq it would be great if this could be added, exactly as @fangq described. But it wouldn't be backward compatible in my opinion to the current version since it doesn't directly allow groups which would imply that current readers might expect different input. But since the Metadata can include arbitrary entries either way I think this wouldn't be too big of a problem. In our IO classes we basically search for specific types which would ensure comparability from IO perspective.

[dboas]

Okay, given that we are trying to close version 1.0 drafts to finalize version 1.0, I think we should save this for version 1.1 draft.

[MichaelUM]

It would be great if this could be changed in the current version already since it is only a minimal change in the specification but would have a bigger practical influence on our data specs within the sNIRF format. And since I'm not sure when 1.1 will be released it would be highly appreciated to have this change already now.