Open yarikoptic opened 2 months ago
I would vote for uniformity between .tsv and .tsv.gz
.tsv.gz
was a pragmatically useful choice for working with existing non-BIDS tools (e.g., FSL's PNM) that expected separately-entered column identification and accepted compressed data. One possibility to compromise here would be something like:
Extension | Headers | Compression | Examples |
---|---|---|---|
.tsv |
First line | None | events.tsv |
.tsv.gz |
First line | gzip |
New |
.bare.tsv |
Sidecar | None | motion.tsv -> motion.bare.tsv.gz |
.bare.tsv.gz |
Sidecar | gzip |
physio.tsv.gz -> physio.bare.tsv.gz |
.parquet |
In-file | Optional | New |
We could state that any of these is acceptable (perhaps with a preference in some use cases), assume people will use one that matches the typical use case for their dataset, and make a simple tool available to convert among them.
This issue is collating two aspects but I think it is warranted. If would be desired - we could split into two.
BIDS 1.x situation
ATM,
.tsv.gz
are a not just a compressed.tsv
like e.g. it happens with.nii.gz
and.nii
-- they are *special** as they are not to carry the header as.tsv
files do.The "specialty" extends into side-car .json files
.tsv
s we carry.json
file where each entry is a structured record describing that column BIDS 1: tabular-files ..tsv
files, as in the case with_beh.json
also carry metadata fields such asTaskName
alongside with columns descriptors -- thus possibly leading to collisions (snake_case is just a recommendation for column names, and overall making schema for such .json files a concoction of two aspects)..tsv.gz
we carry.json
file with a dedicatedColumns
field with a list of header fields and in addition again optionally descriptors per each column (same possible collision) BIDS 1: compressed-tabular-filesIf I got it right (@effigies can correct) the header was excluded from
.tsv.gz
as "not readily readable". May be some folks remember also further details? IMHO argument is weak since it is just a matter of adequate abstraction of "file opener" like e.g. is done in Python. But even if we place that aspect aside I think we would benefit from a more harmonious approach, which only might require 1 extra check for validator:BIDS 2 proposal
.tsv
and.tsv.gz
should carry a header..gz
would only signal compression..tsv.gz
and.tsv
should be supported interchangeably across usessubjects.tsv
,sessions.tsv
etc) unless prohibitive in size (e.g.subjects.tsv
for 10000 subjects with 100 columns or smth like that).json
for either case of.tsv
or.tsv.gz
MAY describe columns withinColumns
field of the.json
which would be a dict containing records conforming current set of fields we reserve for .tsv files .json's but also adding 1 OPTIONAL field (but may be RECOMMENDED for .tsv.gz) -Index
which would provide ordering information.bids-validator
could easily ensure corresponding to the order in.tsv
or.tsv.gz
.Index
, ie. if dicts are ordered like now in Python.Cons
Columns
field. But it might actually be even simplification in some cases (e.g. those_beh.json
) where now they should "subselect" what to choose for descriptors and what for metadata..tsv.gz
through file abstraction would need to "build" index based onIndex
field . But it is really not a rocket sciencePros
.tsv
and.tsv.gz
so if someone obtains a long.tsv
and decides to compress it -- it would be just a matter of exactly that -- compression, without changing content (removing header). Would allow for simpler/generic code/handling.Columns
without any ambiguity (jsonschema or linkml model would be much easier to construct) and possibility of collision.