Check that all BIDS subjects are in pheno.jsonld prior to BIDS parsing

[alyssadai]

The goal is to catch + return errors to do with non-matching BIDS and pheno subject lists before slow pybids processing.

Some notes

• This is actually not easily done using pybids without instantiating a BIDSLayout object (which is the slowest step anyways) as the layout is needed for subject indexing (e.g., .get_subjects())
• Some things that do speed up the BIDSLayout step:
  • passing in a BIDSLayoutIndexer(index_metadata=False) ✅ (See: https://bids-standard.github.io/pybids/generated/bids.layout.index.BIDSLayoutIndexer.html#bids.layout.index.BIDSLayoutIndexer
  • BUT, for overall speed, we probably don't want to have to call BIDSLayout twice (once with index_metadata=False and once without)
• Using the ignore param with a regex pattern to match paths except top-level directories starting with sub- doesn't seem to work - the resulting layout always has 0 subjects
  • probably because pybids indexes subjects based on file metadata

Steps to implement:

• [x] use a simple directory traversal to match sub- directory names
• [x] output more intermediate messages to user during slow BIDS step such as
  • when initial check of subjects / validation of pheno JSONLD has successfully completed
  • before and after slow BIDS dataset initialization (e.g., "reading BIDS dataset")
  • when the Neurobagel merging process for the parsed BIDS metadata and phenotypic annotations begins

Related issues

• [ ] update CLI docs with messages to look out for when running BIDS command

[github-actions[bot]]

We want to keep our issues up to date and active. This issue hasn't seen any activity in the last 30 days. We have applied the _flag:stale label to indicate that this issue should be reviewed again. When you review, please reread the spec and then apply one of these three options:

• prioritize: apply the flag:schedule label to suggest moving this issue into the backlog now
• close: if the issue is no longer relevant, explain why (give others a chance to reply) and then close.
• archive: sometimes an issue has important information or ideas but we won't work on it soon. In this case apply the someday label to show that this won't be prioritized. The stalebot will ignore issues with this label in the future. Use sparingly!