search

bgruening / docker-galaxy

:whale::bar_chart::books: Docker Images tracking the stable Galaxy releases.
http://bgruening.github.io/docker-galaxy
MIT License
226 stars 134 forks source link

Bowtie2 Unable to finish job #489

Closed lavrovSA closed 3 days ago

lavrovSA commented 5 years ago

Hello, I've tried to install Bowtie2 from ToolShed into the docker-galaxy-stable (18.09). It installs Bowtie2 and Samtools, but an attempt to run any mapping produces an error "Unable to complete job". The Bowtie2 itself works since there are a mapping stats and SAM file in the output, so it appears that something wrong with samtools sort -o command after Bowtie2 run. Then I've started the container galaxy-deeptools (now it 19.01) with just docker run -d -p 8080:80 bgruening/galaxy-deeptools command, uploaded one short fasta file and tried to run Bowtie2 already installed into the container (against dm3 genome build). The result was the same, "Unable to complete job". Is it possible to run Bowtie2 somehow? Regards, Sergey Lavrov

bgruening commented 5 years ago

@lavrovSA I could verify this issue. I think there are some metadata problems involved on the Galaxy side. So not related to the Docker image. We will try to fix that. In the meantime can you use BWA?

lavrovSA commented 5 years ago

Thanks for the response!

I've done some testing and found that "Unable to finish job" happens with Bowtie2 wrappers versions from 23 to 19 (they all based on Bowtie2 2.3.4.x). Versions from 18 to 13 (Bowtie2 2.3.3.x) works, but the final output, marked as BAM, is really a SAM (one can see text content clicking on an eye). BWA do the same. Versions of Bowtie2 wrapper below 13 (based on Bowtie2 2.x) are able to produce compressed BAM (zlib library in dependences helps?), but appears to not accept Bowtie2 indexes produced by Bowtie2 index builder based on Bowtie2 2.3.0). It seems not container-related, but I have no locally installed Galaxy to test...

Regards, Sergey

bgruening commented 5 years ago

@lavrovSA Galaxy can display BAM files :) So they are likely BAM and not SAM even when you can see them.

bgruening commented 5 years ago

If you like please test this all on usegalaxy.eu.

FelixErnst commented 5 years ago

I have the same problem. However, I could not solve it by using other bowtie2 versions.

However, on usegalaxy.eu the job runs without problem using the latest bowtie2 version.

FelixErnst commented 5 years ago

ok. I could solve the issue by going back to version 18. I forgot to restart galaxy. So this seems to be specific to the docker installation, since the usegalaxy.eu works fine.

mpaya commented 5 years ago

Hi. I'm preparing a Galaxy instance for publication with same tools I used for data analysis and I'm also finding this error, which breaks the running workflows. When I installed Bowtie2 2.3.4.2 in September on quay.io/bgruening/galaxy latest, 18.05, it had no issue, but using a new image of the same galaxy it has this error. The tool gets installed and works, the outputs are shown on 'View data', but instead to change status to Done, it changes to Fail. I cannot find differences on the installed tool, and the docker image should be the same, so no clue. @FelixErnst did you get this working? Or is there a way to change the status of a job via API? Thanks

FelixErnst commented 5 years ago

I installed version 18 of the tool. I don't remember into which Bowtie2 version that actually translates, but its 2.3.x. Don't forget to restart galaxy after you installed the tool version, because some information get read only during the start up.

bgruening commented 5 years ago

@mpaya do you have more information about the error, e.g. form the log files that are displayed if you start the container with -i -t

mpaya commented 5 years ago

@FelixErnst, I'm using version 23, and restarting docker did not help. I know the latest one, the 24, works, but it does not serve my purpose of reproducibility. Thanks anyway :) @bgruening looking at the logs, the error comes from Job wrapper finish method failed (handler.log). On the previous docker, no such error appears and job ends normally with job 4395 ended (finish() executed in (226.079 ms)).

mvdbeek commented 5 years ago

Oh, interesting. That particular case would be fixed by https://github.com/galaxyproject/galaxy/pull/7889

So what exact docker image are you using now ?

mpaya commented 5 years ago

Like indicated before, I pulled quay.io/bgruening/galaxy latest which contained Galaxy 18.05. No current image in the repo matches this sha256.

docker images --digest
quay.io/bgruening/galaxy   latest   sha256:f477cdb5c9d5aae871dbe8e72b83f9606900c5cdbeea8a7a8436e02172087042

On my tests I also tried specifying the tag for 18.05 and 19.01, seeing the same issue on all three images. Same issue on galaxy-stable.

mvdbeek commented 5 years ago

This will be fixed in 19.05, for which the docker image is not yet released.

bgruening commented 3 days ago

Thanks to @jyotipm29 we are back on track :smile: The new 24.1 image contains a lot of changes and reflects the latest developments in Galaxy. I would like to close this PR, but please feel free to reopen and rebase against the latest version.

Please give it a try:

docker run -p 8080:80 -p 8021:21 -p 4002:4002 --privileged=true -e "GALAXY_DESTINATIONS_DEFAULT=slurm_cluster_docker"   -v /tmp/galaxy-data/:/export/ quay.io/bgruening/galaxy:24.1

... or any other combination. The readme has been updated. Please add any useful tip to it.

For a list of changes, see the Changelog.