pyushkevich
commented
1 year ago Hi Natasha,
When you run on Slurm, there should be a bunch of output files generated in the dump folder of the work directory. Please check those files for errors, perhaps there is some library missing on one of the slurm nodes, or there is an error invoking slurm in the first place.
Paul
On Tue, Aug 8, 2023 at 2:08 PM Natasha Pavlovikj @.***> wrote:
Hi,
I recently downloaded ASHS 2.0.0 (the release from March 2 2022). I ran ashs_main.sh with the test data you provide https://github.com/pyushkevich/ashs/tree/master/testing/atlas_system_test/images and the UPENN PMC Atlas (ashs_atlas_upennpmc_20170810) on our HPC Center that supports Slurm.
When I run ashs_main.sh both with the serial option and the parallel option (-P): ashs_main.sh -I sub07 -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm or ashs_main.sh -I sub07 -P -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm the test run is successful (please see the generated log here, https://gist.github.com/npavlovikj/9b089f11283ed98dbe1cfddfa6d6a6b2).
However, when I run ashs_main.sh with the Slurm option (-S) on the same dataset with: ashs_main.sh -I sub07 -S -q "--mem=60gb --time=168:00:00" -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm_parallel the run fails (please see the generated log here, https://gist.github.com/npavlovikj/4d22d26e713d7406961ee42540927515) with:
**** !! ERROR !! *** Validity check at end of stage 2 detected missing files. (multiatlas/tseg_left_train000/greedy_atlas_to_s ubj_affine.mat and 231 other files).
When I submit the job, I can see many *ashs_mul** jobs being submitted and running, but ultimately the run fails with the error from above.
Do you know why this error happens only when I use the Slurm option? Also, do you have any suggestions on how to fix it?
Please let me know if you need any additional information.
Thank you, Natasha
— Reply to this email directly, view it on GitHub https://github.com/pyushkevich/ashs/issues/9, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJPEWY2YWILVXCRUZYHJQTXUJ6CJANCNFSM6AAAAAA3I3ABEQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
npavlovikj
Hi,
I recently downloaded ASHS 2.0.0 (the release from March 2 2022). I ran ashs_main.sh with the test data you provide and the UPENN PMC Atlas (_ashs_atlas_upennpmc20170810) on our HPC Center that supports Slurm.
When I run _ashsmain.sh both with the serial option and the parallel option (-P):
ashs_main.sh -I sub07 -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurmorashs_main.sh -I sub07 -P -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurmthe test run is successful (please see the generated log here, https://gist.github.com/npavlovikj/9b089f11283ed98dbe1cfddfa6d6a6b2).However, when I run _ashsmain.sh with the Slurm option (-S) on the same dataset with:
ashs_main.sh -I sub07 -S -q "--mem=60gb --time=168:00:00" -a $ASHS_ATLAS_UPENN_PMC -g sub07_mprage.nii.gz -f sub07_tse.nii.gz -w sub07_output_slurm_parallelthe run fails (please see the generated log here, https://gist.github.com/npavlovikj/4d22d26e713d7406961ee42540927515) with:When I submit the job, I can see many **ashs_mul*** jobs being submitted and running, but ultimately the run fails with the error from above.
Do you know why this error happens only when I use the Slurm option? Also, do you have any suggestions on how to fix it?
Please let me know if you need any additional information.
Thank you, Natasha