search

wustl-oncology / analysis-wdls

Scalable genomic analysis pipelines, written in WDL
MIT License
5 stars 11 forks source link

pvacseq/pvacfuse hangs and fails to complete #131

Open rkageyama opened 1 year ago

rkageyama commented 1 year ago

The tasks pvacseq and pvacfuse hang and eventually time out and to to a retry. This happened with a number of different samples. I was able to fix the hanging by changing back to the docker image used before the last PR.

Changing griffithlab/pvactools:4.0.3 to susannakiwala/pvactools:4.0.1rc1 to run the task in both analysis-wdls/definitions/tools/pvacseq.wdl and analysis-wdls/definitions/tools/pvacfuse.wdl fixed the problem and the run went to completion.

susannasiebert commented 1 year ago

Thank you for reporting this issue. It seems like the main difference between these two docker containers is that one (4.0.1rc1) is based on Python 3.11 and the other (4.0.3) is based on Python 3.7. We reverted our containers back to Python 3.7 any container based on Python 3.11 (or 3.8, 3.9, and 3.10) is causing failures on our compute cluster. We are considering releasing several images for different python versions to support multiple compute environments but we need to do more testing on our end to see if there is a way forward with a single image.

susannasiebert commented 11 months ago

@rkageyama I just released version 4.0.5, which we believe will fix this issue (https://github.com/wustl-oncology/analysis-wdls/pull/132). Please give it a try and let me know if the problem persists for you.