search

andersonwinkler / PALM

PALM: Permutation Analysis of Linear Models
69 stars 28 forks source link

Time estimate for/stalling during HCP data analyses #10

Closed mimburgi closed 6 years ago

mimburgi commented 6 years ago

I'm trying to analyze functional data from 225 HCP participants in PALM, but the program seems to be stalling after completing the permutations. Some of the (subcortical) analyses completed in 3-5 hours, but others seem to stall and continue to run for 20+ hours.

We're using FSL 5.0.11, MATlab R2018a, and Workbench 1.3.1. We're running on centos6 with 8GB of RAM allocated. Here's an example of what we're submitting for a cortical file:

-i wm_2-r_L.func.gii -d design_n219_mean.mat -t design_n219_mean.con -o wm_2-r_L_cort -T -tfce2D -s L_midthickness.surf.gii L_area.func.gii -logp -n 10000 -accel tail -nouncorrected -ise

This is the end of one of the .out files before we stopped the analysis, which is the point at which they all seem to be stalling:

Elapsed time with permutations: ~ 38384.6 seconds. Computing p-values. Saving p-values (uncorrected, and corrected within modality and within contrast).

No slurm or segmentation errors have been generated and the jobs appear active at this point, 20+ hours after they were submitted.

How long can we expect these analyses to take? Also, do you know of any reason why they would be stalling at this point? Thank you.

andersonwinkler commented 6 years ago

This sounds an issue with writing the output files to the disk. It should be quick, a couple of seconds per file, never take hours. Most of the time is consumed with permutations, which appear to have run fine in your case.

Having said that, CIFTI files can be slow if they are too large (e.g., several gigabytes), and if the storage is NFS or any kind of network attached system, it will indeed be quite slow.