search

bids-apps / CPAC

BIDS Application for the Configurable Pipeline for the Analysis of Connectomes (C-PAC)
Apache License 2.0
14 stars 18 forks source link

ENH: added bids-validator options #19

Closed jdkent closed 6 years ago

jdkent commented 6 years ago

Same addition as in the freesurfer BIDS-App. When I'm testing cpac runs on our cluster, the bids-validator is a large time sink and this helps me get the testing done quickly with our (already BIDS organized) datasets.

Best, James

chrisgorgo commented 6 years ago

It might be also worth updating the validator - latest version is way faster.

jdkent commented 6 years ago

@chrisfilo, you're referring to version 0.24.0, right? That version is way faster, but I think the bottleneck is the i/o between the compute node of our cluster and the server we have the data collected on. when I run the command locally it takes less than 10 seconds, but on our cluster it appears to take minutes if not longer.

chrisgorgo commented 6 years ago

Yup - that's the one!

jdkent commented 6 years ago

I'm also still advocating this pull request, at least with how I structure job files when I submit them to the cluster, each subject gets their own job, which means that for each subject, the bids-validator will be run, which is unnecessary.

PRO: more computationally efficient and my jobs take less time.

CON: people have errors that their data are not in BIDS, use this option to get past those errors, and then the script may error at some other point, and then they will open an issue about that secondary error instead of BIDS formatting error.

chrisgorgo commented 6 years ago

This is consistent with https://groups.google.com/forum/#!topic/bids-apps-dev/IBRDplI0aVk and already implemented in couple of apps.