search

MikeDacre / fyrd

Submit functions and shell scripts to torque and slurm clusters or local machines using python.
https://fyrd.science
MIT License
22 stars 8 forks source link

Problem with array job id #59

Closed smhr closed 7 years ago

smhr commented 7 years ago

Hi In our queue system there are some array jobs with job ids like: 136374[].psk When I run for example: job = fyrd.Job('ls .', profile='short') this gives me: `~/anaconda3/lib/python3.5/site-packages/fyrd/queue.py in torque_queue_parser(user, partition) --> 680 job_id = int(xmljob.find('Job_Id').text.split('.')[0]) 681 job_owner = xmljob.find('Job_Owner').text.split('@')[0] 682 if user and job_owner != user:

ValueError: invalid literal for int() with base 10: '136374[]'`

MikeDacre commented 7 years ago

Thanks for the report @smhr.

Would it work to strip all non-integer characters from the job ID? Is your queue able to find jobs with the format 136374, or do they have to be the complete '136374[]'?

The other alternative is that I could change the code to not require integer job IDs, there actually isn't any real reason to enforce that anymore.

smhr commented 7 years ago

Would it work to strip all non-integer characters from the job ID? Is your queue able to find jobs with the format 136374, or do they have to be the complete '136374[]'?

Yes, to temporarily get around this problem, I add this line to queue.py before converting it to integer: job_id = re.sub('[^A-Za-z0-9.]+', '', job_id)

In our queue if I run qstat it gives: 136374[], but if I use qstat -t it gives 136374[1], 136374[2], etc.

MikeDacre commented 7 years ago

OK, thanks @smhr , after considering I think I will actually change job ID handling to use strings instead of integers. I originally did that to try and enforce similar job IDs between torque and slurm, but there actually isn't a good reason to do that anymore, and it neglects the job array use case.

I have a question though: when submitting jobs from fyrd, would you want to use job arrays for anything, right now I don't think that is possible.

smhr commented 7 years ago

I have a question though: when submitting jobs from fyrd, would you want to use job arrays for anything, right now I don't think that is possible.

I have not test array job with fyrd. I found this issue because someone uses array job in our torque queue system.

MikeDacre commented 7 years ago

OK, thanks, I will fix this in the next couple of days, the next beta release will work properly.

MikeDacre commented 7 years ago

I am doing more work to fully integrate array job IDs, but for now the latest commit to master means that they at least do not break queue parsing anymore. Thanks for the bug report @smhr