Closed irvinng98 closed 1 year ago
This is my pbs_status.py:
#!/usr/bin/env python3
import sys
import subprocess
import xml.etree.cElementTree as ET
jobid = sys.argv[1]
try:
res = subprocess.run(
"qstat {}".format(jobid),
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True,
)
if not res.stdout:
print("failed")
else:
xmldoc = ET.ElementTree(ET.fromstring(res.stdout.decode())).getroot()
job_state = xmldoc.findall(".//job_state")
if len(job_state) == 0:
print("failed")
else:
job_state = job_state[0].text
if job_state == "C":
exit_status = xmldoc.findall(".//exit_status")[0].text
if exit_status == "0":
print("success")
else:
print("failed")
else:
print("running")
except (subprocess.CalledProcessError, IndexError, KeyboardInterrupt) as e:
print("failed")
I see there qsub -A <allocation> -N initialize_qc -l select=1:ncpus=4:mem=10000mb -l walltime=6000
Do you think the -A <allocation>
could cause problems?
I also see there are these warings:
CLUSTER: 2023-05-10 16:41:14 parameter 'mem_mb' not in keymapping! It would be better if you add the key to the file: /home/irvinng/.config/snakemake/cluster/key_mapping.yaml I try without the key! CLUSTER: 2023-05-10 16:41:14 parameter 'queue' not in keymapping! It would be better if you add the key to the file: /home/irvinng/.config/snakemake/cluster/key_mapping.yaml I try without the key!
So I suggest you to adapt the key_mapping file for these parameters.
Previoulsy I used the mem argument in GB and now the mem_mb in megabites.
Thanks for replying! I don't think it would be an issue with the allocation or queue because otherwise I wouldn't have been able to submit the job (which was an issue for me initially until I tweaked some stuff). Normally I wouldn't have to specify the queue as my cluster has a default queue (in fact, if I try to specify the queue, it would give me: "qsub: Access to queue is denied"). In terms of the mem_mb key, my cluster doesn't accept it as a separate line (eg. -l mem_mb=10000mb). Instead, it only takes it in the format of -l select=1:ncpus=4:mem=10000mb. As such, I had to tweak the key_mapping.yaml such that it looked like:
pbs:
command: "qsub"
key_mapping:
name: "-N {}"
account: "-A {}"
threads: "-l select=1:ncpus={}:mem={}mb"
time_min: "-l walltime={}00" #min= seconds x 100
and my scheduler.py:
# construct command:
for key in cluster_param:
if key not in key_mapping:
logging.warning(
f"parameter '{key}' not in keymapping! It would be better if you add the key to the file: {key_mapping_file} \n I try without the key!"
)
else:
command += " "
if key == "threads":
command += key_mapping[key].format(cluster_param[key], cluster_param["mem_mb"])
else:
command += key_mapping[key].format(cluster_param[key])
I was wondering if my error had to do with perhaps how the jobs are named which is messing up the syntax.
As mentioned, the job IDs have a format of #######.pbsha.ib.sockeye and because I'm not super familiar with python, I'm not sure if the qsub_status.py (code is in a comment above) is able to interpret the status.
Ok, you found a hack to submit the script. Great!
Ok, let's look at the job status script.
when you run qstat 4989470
what do you get back? a xml?
Does it need to be qstat 4989470.pbsha
to work?
I don't know how to parse the output correctly. I took the code from another pbs torque profile. and cannot test it as I don't have pbs system.
If you get a xml we might try to find how to adapt the code to make it work.
If you know some options on how to get a simpler text output using qstat It would ease the process.
In the meantime, I think you can also comment out the pbs_statsu.py in the cluster_profile/config.yaml .
There was no activity since some time. I hope your issue is solved in the mean time. This issue will automatically close soon if no further activity occurs.
Thank you for your contributions.
Firstly, thank you making such a wonderful pipeline.
I am trying to run atlas with
and after a lot of tweaking with the snakemake files, I finally got it to run on my PBS cluster system. However, I now get this error message:
Here is the log:
Atlas version 2.15.1
Additional context Not sure if this is relevant, but when I qstat on my cluster, it looks something like this:
and the full Job id would be #######.pbsha.ib.sockeye