search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
523 stars 111 forks source link

Error running Cactus with SLURM: 'bad interpreter: No such file or directory' #1227

Open pc363449207 opened 1 year ago

pc363449207 commented 1 year ago

Hello,

I am encountering an issue when running Cactus on a SLURM cluster. After executing the command:

cactus ./js ./examples/evolverMammals.txt evolverMammals.hal --batchSystem slurm --batchLogsDir batch-logs --consCores 80

I receive the following error message:

[2023-11-15T15:59:37+0800] [MainThread] [W] [toil.leader] No log file is present, despite job failing: 'progressive_workflow' kind-progressive_workflow/instance-hlg2sd3l v1) [00:32<00:00, 0.75 jobs/s] [2023-11-15T15:59:37+0800] [MainThread] [W] [toil.leader] The batch system left an empty file batch-logs/toil_1d743db1-387f-4c1f-be9a-55f8d653b0dc.24.83368.out.log [2023-11-15T15:59:37+0800] [MainThread] [W] [toil.leader] The batch system left a non-empty file batch-logs/toil_1d743db1-387f-4c1f-be9a-55f8d653b0dc.24.83368.err.log: [2023-11-15T15:59:37+0800] [MainThread] [W] [toil.leader] Log from job "kind-progressive_workflow/instance-hlg2sd3l" follows: =========> /opt/gridview/slurm/spool/slurmd/job83368/slurm_script: /public/home/pancheng/cactus/cactus_env/bin/_toil_worker: /public/home/pancheng/cactus/cactus_env/bin/python3: bad interpreter: No such file or directory <=========

However, I have verified that the path /public/home/pancheng/cactus/cactus_env/bin/python3 does indeed exist and is accessible.

Here are some key details regarding my setup: I am using a shared file system in the SLURM cluster. The virtual environment is correctly set up at the mentioned path. Python 3.8 is installed, and the virtual environment was created and activated without any issues. The issue persists despite the environment being correctly activated in the SLURM job script. I am not sure why this error is occurring since the specified interpreter path is correct and works outside of the SLURM environment. Any insights or suggestions on how to resolve this issue would be greatly appreciated.

Thank you!

glennhickey commented 1 year ago

I don't know. The 

 /public/home/pancheng/cactus/cactus_env/bin/python3: bad interpreter: No such file or directory

message makes it really look like the worker job cannot find /public/home/pancheng/cactus/cactus_env/bin/python3 for some reason.