NotImplementedError: Executor pbs is not supported, abort

[zhouyanli11]

Hi @kirilenkobm,

I've been trying to run make_lastz_chains on my dataset and I ran the following command: ./make_chains.py homSap gorGor ./raw-genome/homSap.sm.fa ./raw-genome/gorGor.sm.fa --executor pbs --executor_queuesize 100 --project_dir homSap-gorGor

However I have encountered an error message.

************ HgStepManager: executing step 'lastz' Sat May  7 20:48:40 2022.
doLastzClusterRun ....
testCleanState current step: doLastzClusterRun, /gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz, /gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz/lastz.done     previous step: doPartition, /gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz/partition.done
# chmod a+x /gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz/doClusterRun.sh
# /gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz/doClusterRun.sh
+ cd /gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz
+ gensub2 homSap.lst gorGor.lst gsub jobList
+ parallel_executor.py lastz_homSapgorGor jobList -q day --memoryMb 10000 -e pbs --co None -p batch --eq 100
Traceback (most recent call last):
  File "/gpfs/home/liunyw/biosoft/make_lastz_chains/HL_scripts/parallel_executor.py", line 176, in <module>
    main()
  File "/gpfs/home/liunyw/biosoft/make_lastz_chains/HL_scripts/parallel_executor.py", line 155, in main
    queue=_q_arg,
  File "/gpfs/home/liunyw/soft/python3/lib/python3.6/site-packages/py_nf/py_nf.py", line 125, in __init__
    self.__check_executor()
  File "/gpfs/home/liunyw/soft/python3/lib/python3.6/site-packages/py_nf/py_nf.py", line 216, in __check_executor
    raise NotImplementedError(msg)
NotImplementedError: Executor pbs is not supported, abort

This looks like the --executor parameter is not set properly. However, what confuses me is that the Executors of cluster I am using is pbs and I have used process.executor = 'pbs' in my TOGA test data(nextflow_config_files/call_cesar_config_template.nf) and it worked.

Thank you Yawen

[zhouyanli11]

Hello,

I have solved the problems mentioned above. I maked some changes in ~/python3/lib/python3.6/site-packages/py_nf/py_nf.py. I changed the code "psb": "qsub", to "pbs": "qsub", on line 65 and it worked. However, I encountered another bug.

Launching `/gpfs/home/liunyw/howler_monkey/gene_loss/homSap-gorGor/TEMP_run.lastz/lastz_homSapgorGor/script.nf` [zen_bassi] - revision: 00ae0d4c33
[6a/a05dbb] process > execute_jobs (47) [  2%] 47 of 1922, failed: 47, retries: 47
[89/bd941f] NOTE: Error submitting process 'execute_jobs (15)' for execution -- Execution is retried (1)
[0f/7d8bc7] NOTE: Error submitting process 'execute_jobs (16)' for execution -- Execution is retried (1)
[e5/7118f0] process > execute_jobs (56) [  2%] 56 of 1931, failed: 56, retries: 56
[89/bd941f] NOTE: Error submitting process 'execute_jobs (15)' for execution -- Execution is retried (1)
[0f/7d8bc7] NOTE: Error submitting process 'execute_jobs (16)' for execution -- Execution is retried (1)

Then, I release this may be due to a configuration file projectname/TEMP_run.lastz/lastz_projectname/config.nf, which is shown below

process {
    executor = 'pbs'
    queue = 'batch'
    memory = '10.GB'
    time = '24.h'
    cpus = '1'

    withLabel: retry_increase_mem {
        memory = {10.GB * task.attempt}
        errorStrategy = 'retry'
    }

    withLabel: retry_increase_time {
        time = {24.h * task.attempt}
        errorStrategy = 'retry'
    }
}
executor.queueSize = 100 

The cluster server I'm using should actually be set to queue = 'core28' rather than queue = 'batch'. However, it doesnt work. Could you help me?

Thank you very much Yawen

[kirilenkobm]

Hello @zhouyanli11,

Sorry for the long silence, and thank you for your patience. Thanks for noticing the typo, I fixed it in the py_nf library. About queue = 'core28' pls add --executor_partition core28 to the make_chains.py call, this parameter should adjust it.

Best, Bogdan