kevinVervier
commented
4 years ago Hi Ben, Building a Kraken database requires a lot of memory, which is why in the current version it only allows hugemem or teramem queue. In farm3/4, requesting more than 200G memory on other queues ended up with jobs pending forever. Maybe it is different now on farm5.
So either you wait for farm5 to get a hugemem queue (which should exist at some point), or we change the script to allow the krakenbuild step on any queue, knowing it will likely not work on other queues.
To verify that, could you submit the same job but using the queue you'd prefer:
bsub -q YOUR_QUEUE_NAME -E 'test -e /nfs/users/nfs_b/bb11' -n 12 -R "span[hosts=1] select[mem>300000] rusage[mem=300000]" -M300000 -o Krakenbuild.o -e Krakenbuild.e -J Krakenbuild kraken2-build --build --db ./test1/Kraken
If it runs correctly on another queue, I am happy to edit the code later.
Hi Kevin,
I am trying to build my KrakenDB using metagm. The script successfully generates the kraken library, but when it comes to submitting the kraken2-build job it still gets submitted to hugemem with 300 GB, even though I have specified otherwise in my command. Farm5 obviously doesn't have a hugemem queue, so was wondering how to get around this?
Best wishes, Ben
Job: metagm_build.py -q long -m 100 --KrakenDB --BrackenDB ../metagm_build/metagm_build_batchfile.txt ./test1
Error message: hugemem: No such queue. Job not submitted. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/farmpy-0.42.1-py3.6.egg/farmpy/lsf.py", line 127, in run File "/usr/lib/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command 'bsub -q hugemem -E 'test -e /nfs/users/nfs_b/bb11' -n 12 -R "span[hosts=1] select[mem>300000] rusage[mem=300000]" -M300000 -o Krakenbuild.o -e Krakenbuild.e -J Krakenbuild kraken2-build --build --db ./test1/Kraken ' returned non-zero exit status 255.