search

roryk / ipython-cluster-helper

Tool to easily start up an IPython cluster on different schedulers.
148 stars 23 forks source link

Can bcbio_nextgen run via ipython parallel on an SGE cluster where the qsub command can't be executed from the compute nodes? #62

Closed amizeranschi closed 5 years ago

amizeranschi commented 5 years ago

Hello,

Is there any chance to run bcbio_nextgen run via ipython parallel on an SGE cluster where the qsub command can't be executed from the compute nodes? I posted more information about this on the bcbio_nextgen issue tracker: https://github.com/bcbio/bcbio-nextgen/issues/2701.

roryk commented 5 years ago

Hi @amizeranschi, for this case, you'll have to run the main bcbio_nextgen.py job from the login node. The bcbio_nextgen.py job is the one that does the submission to the queue, so all of those jobs should get submitted via the login node if you run it there.

amizeranschi commented 5 years ago

Thanks for your reply. I'm having some trouble also when running bcbio_nextgen.py from the login node. The sge_controller and sge_engine are starting up, but don't seem to be communicating properly. I posted more info on the bcbio_nextgen issue tracker.

amizeranschi commented 5 years ago

Just to clarify, when you mean run the main bcbio_nextgen.py job from the login node, do you mean it has to be run via qsub (via the SGE queue), but it should then get executed on the login node instead of a compute node? Or do you mean that the script which invokes bcbio_nextgen.py should be run directly via Bash, on the login node?

roryk commented 5 years ago

Hi @amizeranschi,

Sorry for not being clear. The latter is right, just calling it from bash.