nellh
commented
4 years ago The ignoreSymlinks option does only apply to directory recursion when running under Node.js. The aim with it is really an optimization for OpenNeuro, where calling stat on each symlink and target doubles the I/O cost of evaluating a git-annex dataset even though we know none of the symlinks are directories.
Some users have used symlinked directories to construct BIDS datasets where session data is not present in the dataset because it is kept on another (usually large and network accessible) filesystem, so allowing the validator to follow those directory symlinks by default is useful.
kousu
There is now a number of options which could be used to provide some partial remedy:
--ignoreSymlinks(its --help output talks about directories but I think it was meant about files),--remoteFiles, and--ignoreNiftiHeaders.All are good but insufficient to ensure that whatever content IS present (a regular file or was fetched using git-annex) constitute a legit BIDS dataset.
I feel that
--ignoreSymlinksshould really be--ignoreOrphanedSymlinks(I do not like "Orphan" , like "Dangling" or "Broken" more, but there isORPHANED_SYMLINKerror already)bids-validatormust first collect a list of files which are either plain files, or non-orphaned symlinks, and then perform validation ignoring any other file which might be an orphaned symlink.It could still issue a warning (not error) about ORPHANED_SYMLINK which would be informative, or it could be even a stronger more dedicated warning that it is only a PARTIAL_CHECK since some files are not (yet) available.
edit 1:
git links
git-annex has "unlocked" mode in which it is not symlinks committed but git links, which would manifest themselves as regular files containing the reference to the annexed object (to be
annex get'ed):so -- it would be nice if "is_orphaned_link" helper function would not only check if it is a orphaned symlink, but in case of small (e.g. <500B?) files it would read first bytes to confirm that it does not start with
/annex/objects/general utility
Depending on how implemented, it might be useful to expose options to point validator to only specific set of paths to be used in validation. E.g. if it is
-i|--input, then I could quickly validate a single subject (without full validation for consistency with all other subjects) with smth like-i *.json *tsv sub-000002, or even with consistency to the canonical pilot subject if I specify another folder (e.g.sub-000001) as well. It might be very beneficial while composing large bids datasets (100s or 1000s of subjects) without waiting increasingly longer times after adding each subject.