search

fsimkovic / pyjob

Job execution management in Python
http://pyjob.rtfd.io
MIT License
8 stars 4 forks source link

TaskFactory doesn't wait for task completion #37

Closed FilomenoSanchez closed 4 years ago

FilomenoSanchez commented 4 years ago

System information

Problem description

TaskFactory doesn't wait for all the jobs in the task array to be completed. This behaviour has been observed on SGE versions older than 8.1.9 as well.

Example

from pyjob import TaskFactory, Script

my_script=Script(directory="/home/test", prefix="mytest", stem='', suffix='.sh')
my_script.append("sleep 100")

with TaskFactory("sge", tuple(my_script), cwd="/home/test") as task:
        task.run()

print('This part is reached before the task is completed!')

Traceback

There is no Python traceback, the taskfactory doesn't wait for all the processes to completed and exits the 'with-block ' immediately after submission to the HPC queue.

FilomenoSanchez commented 4 years ago

I have traced down this bug to @property info at pyjob.sge.The output of qstat -j PID cannot be parsed properly with the current regex and the property returns an empty dictionary, which then makes @property completed at pyjob.task evaluate True.