[BUG] In BIDS, data-type may be at subject or session level

[JoeZiminski]

Currently NeuroBlueprint mandates the placement of data-type folders (behav, ephys, funcimg) in the session level, and histology in the subject level. We discussed a little on slack about moving histology into the session level for simplicity. However of course BIDS does not actually mandate the session level, and data-type folders can be at the subject or session level.

I think for BIDS compliance it makes sense to also support having this flexibility. However it does make data-discovery in things like spikewrap more difficult, and datashuttle will require some massaging + a lot more tests and checks (e.g. error if trying to put a data-type folder at the subject level that is already somewhere in the session level, and vice-versa). But I think in principle it makes sense?

[niksirbi]

Ah, I hadn't thought of that. Some projects might have no sessions at all. It could be just histology, we also want to support them. But I don't know how to exactly resolve this.

We could say that even in that case, we force people to create ses-01 and put histology under that (easier for us to implement). Or we could allow people to not use the session level altogether (compliant with general BIDS).

[adamltyson]

I think ideally users could do both, I can see microscopy only projects that could be either:

• Just one "session", i.e. a single confocal image, for which adding creating ses-01 seems onerous
• A more complex project involving tissue clearing, imaging session 1, restaining, imaging session 2, sectioning, imaging session 3. Here sessions makes much more sense.

If this can be implemented within DataShuttle and NIU tools that ingest these data it would be great, but that could be a lot of overhead. If it's too much work, I'd be happy to prescribe what users should do.

[niksirbi]

• A more complex project involving tissue clearing, imaging session 1, restaining, imaging session 2, sectioning, imaging session 3. Here sessions makes much more sense.

Actually, that makes sense. I was not familiar with such histology workflows.

How about this:

• make session level optional (up to the user)
• if no session level, put all datatypes under subject (obviously)
• if session level is used, put datatypes under a session (incl. histology)
• do not mix subject-level and session-level datatypes.

[JoeZiminski]

(NOTE - I completely misunderstood the BIDS spec, this SHOULD refers to that if there is a session level for one subject, there should be for all subjects. I could not find this for the issue we are discussing, I asked on the google group).

I think this is most clear and would avoid lots of problems with parsing. Technically BIDS does allow this however, but it is under SHOULD

In datasets where at least one subject has more than one session, this additional subdirectory later SHOULD be added for all subjects in the dataset. 

I guess this becomes a more general question, as how strictly it is feasible to align with BIDS / if we should enforce some BIDS SHOULD as MUST to make software development easiest. Some possible options:

1) Follow BIDS exactly in NeuroBlueprint but in specific tools enforce stricter behaviour (probably not good because we cannot then claim our tools work with NeuroBlueprint 2) Have NeuroBlueprint be more restrictive than BIDS in places (not good as then there is confusion between the specifications, and essentially they have diverged). 3) Allow full BIDS (possibly lead to hellish file / folder discovery problems).

I think all of these are bad, personally I'd be up for trying to maintain full alignment with BIDS unless there is a specific instance where it is just not possible to robustly perform file / folder search operations. I could try and handle this case in datashuttle and see how hard it is.

[adamltyson]

personally I'd be up for trying to maintain full alignment with BIDS unless there is a specific instance where it is just not possible to robustly perform file / folder search operations.

I think this should be the general aim. We should align with BIDS as long as it allows us to achieve our other goals. If we need to diverge, then we should. There's a reason why we created NeuroBlueprint in the first place. Turning "real" BIDS into something that supports all of our data types is currently too much work to be feasible in the time frame we have.

[adamltyson]

Ofc I hope that NeuroBlueprint and BIDS will converge eventually

[adamltyson]

Actually, that makes sense. I was not familiar with such histology workflows.

@niksirbi to be clear, these are rare, but it's something we would like to make possible at SWC with the mesospim etc.

[JoeZiminski]

I think this should be the general aim. We should align with BIDS as long as it allows us to achieve our other goals. If we need to > diverge, then we should. There's a reason why we created NeuroBlueprint in the first place. Turning "real" BIDS into something > that supports all of our data types is currently too much work to be feasible in the time frame we have.

This makes sense, if we are getting bogged down and spending a lot of time trying to maintain BIDS compatability, and making some more stringent criteria will make life much easier and the software more robust we should do it. I suggest we start with full BIDS compliance and then reduce it if issues arise with implementation.

[JoeZiminski]

updated with decision in #45