Closed tamuanand closed 1 year ago
I think it has to do with https://pypi.org/project/galaxy-tool-util/ and the mulled-search command that it provides, but I can't actually find any documentation or get it to output anything.
I have been able to find combinations by pulling this repository, then:
$ cd combinations
$ grep
Hi all, there is some documentation here: https://galaxy-lib.readthedocs.io/en/latest/topics/mulled.html#search-for-containers
A quick example is:
pip install galaxy-tool-util[mulled]
bag@bag:~$ mulled-search -d quay -s bwa samtools
The query returned the following result(s).
LOCATION NAME VERSION COMMAND
quay bwa 0.7.3a--hed695b0_5 docker pull quay.io/biocontainers/bwa:0.7.3a--hed695b0_5
quay bwa 0.7.8--hed695b0_4 docker pull quay.io/biocontainers/bwa:0.7.8--hed695b0_4
quay bwa 0.7.8--hed695b0_5 docker pull quay.io/biocontainers/bwa:0.7.8--hed695b0_5
quay bwa 0.7.4--h5bf99c6_4 docker pull quay.io/biocontainers/bwa:0.7.4--h5bf99c6_4
quay bwa 0.7.17--ha92aebf_3 docker pull quay.io/biocontainers/bwa:0.7.17--ha92aebf_3
quay bwa 0.7.17--hed695b0_6 docker pull quay.io/biocontainers/bwa:0.7.17--hed695b0_6
quay bwa 0.7.17--hed695b0_7 docker pull quay.io/biocontainers/bwa:0.7.17--hed695b0_7
quay bwa 0.7.8--h84994c4_3 docker pull quay.io/biocontainers/bwa:0.7.8--h84994c4_3
quay bwa 0.7.8--1 docker pull quay.io/biocontainers/bwa:0.7.8--1
quay bwa 0.7.4--ha92aebf_0 docker pull quay.io/biocontainers/bwa:0.7.4--ha92aebf_0
quay bwa 0.7.3a--h5bf99c6_6 docker pull quay.io/biocontainers/bwa:0.7.3a--h5bf99c6_6
quay bwa 0.7.17--h5bf99c6_8 docker pull quay.io/biocontainers/bwa:0.7.17--h5bf99c6_8
quay bwa 0.6.2--ha92aebf_2 docker pull quay.io/biocontainers/bwa:0.6.2--ha92aebf_2
quay bwa 0.7.8--0 docker pull quay.io/biocontainers/bwa:0.7.8--0
quay bwa 0.7.4--hed695b0_2 docker pull quay.io/biocontainers/bwa:0.7.4--hed695b0_2
quay bwa 0.7.4--hed695b0_3 docker pull quay.io/biocontainers/bwa:0.7.4--hed695b0_3
quay bwa 0.5.9--0 docker pull quay.io/biocontainers/bwa:0.5.9--0
quay bwa 0.5.9--1 docker pull quay.io/biocontainers/bwa:0.5.9--1
quay bwa 0.7.8--h5bf99c6_6 docker pull quay.io/biocontainers/bwa:0.7.8--h5bf99c6_6
quay bwa 0.7.3a--ha92aebf_2 docker pull quay.io/biocontainers/bwa:0.7.3a--ha92aebf_2
quay bwa 0.7.8--ha92aebf_2 docker pull quay.io/biocontainers/bwa:0.7.8--ha92aebf_2
quay bwa 0.7.13--1 docker pull quay.io/biocontainers/bwa:0.7.13--1
quay bwa 0.7.15--0 docker pull quay.io/biocontainers/bwa:0.7.15--0
quay bwa 0.7.17--h84994c4_5 docker pull quay.io/biocontainers/bwa:0.7.17--h84994c4_5
quay bwa 0.7.17--h84994c4_4 docker pull quay.io/biocontainers/bwa:0.7.17--h84994c4_4
quay bwa 0.7.15--1 docker pull quay.io/biocontainers/bwa:0.7.15--1
quay bwa 0.7.4--h84994c4_1 docker pull quay.io/biocontainers/bwa:0.7.4--h84994c4_1
quay bwa 0.7.17--pl5.22.0_2 docker pull quay.io/biocontainers/bwa:0.7.17--pl5.22.0_2
quay bwa 0.7.17--pl5.22.0_1 docker pull quay.io/biocontainers/bwa:0.7.17--pl5.22.0_1
quay bwa 0.7.17--pl5.22.0_0 docker pull quay.io/biocontainers/bwa:0.7.17--pl5.22.0_0
quay bwa 0.7.3a--0 docker pull quay.io/biocontainers/bwa:0.7.3a--0
quay bwa 0.7.3a--1 docker pull quay.io/biocontainers/bwa:0.7.3a--1
quay bwa 0.7.16--pl5.22.0_0 docker pull quay.io/biocontainers/bwa:0.7.16--pl5.22.0_0
quay bwa 0.7.3a--h84994c4_3 docker pull quay.io/biocontainers/bwa:0.7.3a--h84994c4_3
quay bwa 0.6.2--1 docker pull quay.io/biocontainers/bwa:0.6.2--1
quay bwa 0.5.9--ha92aebf_2 docker pull quay.io/biocontainers/bwa:0.5.9--ha92aebf_2
quay bwa 0.7.3a--h84994c4_4 docker pull quay.io/biocontainers/bwa:0.7.3a--h84994c4_4
quay bwa-mem2 2.2.1--he513fc3_0 docker pull quay.io/biocontainers/bwa-mem2:2.2.1--he513fc3_0
quay bwa-mem2 2.2.1--h9a82719_1 docker pull quay.io/biocontainers/bwa-mem2:2.2.1--h9a82719_1
quay bwa-mem2 2.0--he513fc3_0 docker pull quay.io/biocontainers/bwa-mem2:2.0--he513fc3_0
quay bwa-mem2 2.0--he513fc3_1 docker pull quay.io/biocontainers/bwa-mem2:2.0--he513fc3_1
quay bwa-mem2 2.2--he513fc3_0 docker pull quay.io/biocontainers/bwa-mem2:2.2--he513fc3_0
quay bwa-mem2 2.1--he513fc3_0 docker pull quay.io/biocontainers/bwa-mem2:2.1--he513fc3_0
quay samtools 1.3.1--2 docker pull quay.io/biocontainers/samtools:1.3.1--2
quay samtools 0.1.19--2 docker pull quay.io/biocontainers/samtools:0.1.19--2
quay samtools 0.1.19--3 docker pull quay.io/biocontainers/samtools:0.1.19--3
quay samtools 1.2 docker pull quay.io/biocontainers/samtools:1.2
quay samtools 1.9--h57cc563_7 docker pull quay.io/biocontainers/samtools:1.9--h57cc563_7
quay samtools 1.0--hdd8ed8b_2 docker pull quay.io/biocontainers/samtools:1.0--hdd8ed8b_2
quay samtools 1.3.1--h1b8c3c0_8 docker pull quay.io/biocontainers/samtools:1.3.1--h1b8c3c0_8
quay samtools 1.7--2 docker pull quay.io/biocontainers/samtools:1.7--2
quay samtools 1.6--he673b24_3 docker pull quay.io/biocontainers/samtools:1.6--he673b24_3
quay samtools 1.0--0 docker pull quay.io/biocontainers/samtools:1.0--0
quay samtools 1.1--h270b39a_6 docker pull quay.io/biocontainers/samtools:1.1--h270b39a_6
quay samtools 0.1.12--0 docker pull quay.io/biocontainers/samtools:0.1.12--0
quay samtools 1.6--h244ad75_4 docker pull quay.io/biocontainers/samtools:1.6--h244ad75_4
quay samtools 1.2-0 docker pull quay.io/biocontainers/samtools:1.2-0
quay samtools 1.5--0 docker pull quay.io/biocontainers/samtools:1.5--0
quay samtools 0.1.13--0 docker pull quay.io/biocontainers/samtools:0.1.13--0
quay samtools 0.1.19--hf89b575_7 docker pull quay.io/biocontainers/samtools:0.1.19--hf89b575_7
quay samtools 0.1.18--h270b39a_11 docker pull quay.io/biocontainers/samtools:0.1.18--h270b39a_11
quay samtools 1.3--1 docker pull quay.io/biocontainers/samtools:1.3--1
quay samtools 1.3--2 docker pull quay.io/biocontainers/samtools:1.3--2
quay samtools 1.9--h8571acd_11 docker pull quay.io/biocontainers/samtools:1.9--h8571acd_11
quay samtools 0.1.19--h94a8ba4_5 docker pull quay.io/biocontainers/samtools:0.1.19--h94a8ba4_5
quay samtools 0.1.18--hfb9b9cc_10 docker pull quay.io/biocontainers/samtools:0.1.18--hfb9b9cc_10
quay samtools 0.1.19--h94a8ba4_6 docker pull quay.io/biocontainers/samtools:0.1.19--h94a8ba4_6
quay samtools 1.9--h57cc563_6 docker pull quay.io/biocontainers/samtools:1.9--h57cc563_6
quay samtools 1.9--h91753b0_4 docker pull quay.io/biocontainers/samtools:1.9--h91753b0_4
quay samtools 0.1.19--hfb9b9cc_8 docker pull quay.io/biocontainers/samtools:0.1.19--hfb9b9cc_8
quay samtools 1.8--h46bd0b3_5 docker pull quay.io/biocontainers/samtools:1.8--h46bd0b3_5
quay samtools 1.7--0 docker pull quay.io/biocontainers/samtools:1.7--0
quay samtools 0.1.12--2 docker pull quay.io/biocontainers/samtools:0.1.12--2
quay samtools 0.1.15--0 docker pull quay.io/biocontainers/samtools:0.1.15--0
quay samtools 1.8--3 docker pull quay.io/biocontainers/samtools:1.8--3
quay samtools 1.9--h8ee4bcc_1 docker pull quay.io/biocontainers/samtools:1.9--h8ee4bcc_1
quay samtools 1.12--hd5e65b6_0 docker pull quay.io/biocontainers/samtools:1.12--hd5e65b6_0
quay samtools 1.8--2 docker pull quay.io/biocontainers/samtools:1.8--2
quay samtools 1.8--4 docker pull quay.io/biocontainers/samtools:1.8--4
quay samtools 1.7--1 docker pull quay.io/biocontainers/samtools:1.7--1
quay samtools 1.0--1 docker pull quay.io/biocontainers/samtools:1.0--1
quay samtools 1.3--h1b8c3c0_5 docker pull quay.io/biocontainers/samtools:1.3--h1b8c3c0_5
quay samtools 1.6--h9dace67_6 docker pull quay.io/biocontainers/samtools:1.6--h9dace67_6
quay samtools 1.1--h94a8ba4_3 docker pull quay.io/biocontainers/samtools:1.1--h94a8ba4_3
quay samtools 1.10--h9402c20_2 docker pull quay.io/biocontainers/samtools:1.10--h9402c20_2
quay samtools 1.10--h9402c20_1 docker pull quay.io/biocontainers/samtools:1.10--h9402c20_1
quay samtools 1.10--h9402c20_0 docker pull quay.io/biocontainers/samtools:1.10--h9402c20_0
quay samtools 1.3--h80b0bb3_4 docker pull quay.io/biocontainers/samtools:1.3--h80b0bb3_4
quay samtools 1.3.1--5 docker pull quay.io/biocontainers/samtools:1.3.1--5
quay samtools 1.6--0 docker pull quay.io/biocontainers/samtools:1.6--0
quay samtools 1.3.1--3 docker pull quay.io/biocontainers/samtools:1.3.1--3
quay samtools 1.9--h43f6869_9 docker pull quay.io/biocontainers/samtools:1.9--h43f6869_9
quay samtools 1.5--2 docker pull quay.io/biocontainers/samtools:1.5--2
quay samtools 1.5--1 docker pull quay.io/biocontainers/samtools:1.5--1
quay samtools 1.12--h9aed4be_1 docker pull quay.io/biocontainers/samtools:1.12--h9aed4be_1
quay samtools 1.9--h91753b0_2 docker pull quay.io/biocontainers/samtools:1.9--h91753b0_2
quay samtools 1.6--hb116620_7 docker pull quay.io/biocontainers/samtools:1.6--hb116620_7
quay samtools 1.10--h2e538c0_3 docker pull quay.io/biocontainers/samtools:1.10--h2e538c0_3
quay samtools 1.11--h6270b1f_0 docker pull quay.io/biocontainers/samtools:1.11--h6270b1f_0
quay samtools 1.6--h244ad75_5 docker pull quay.io/biocontainers/samtools:1.6--h244ad75_5
quay samtools 1.4.1--0 docker pull quay.io/biocontainers/samtools:1.4.1--0
quay samtools 0.1.13--1 docker pull quay.io/biocontainers/samtools:0.1.13--1
quay samtools 1.1--hf89b575_4 docker pull quay.io/biocontainers/samtools:1.1--hf89b575_4
quay samtools 1.3.1--4 docker pull quay.io/biocontainers/samtools:1.3.1--4
quay samtools 1.2--0 docker pull quay.io/biocontainers/samtools:1.2--0
quay samtools 1.2.rglab--0 docker pull quay.io/biocontainers/samtools:1.2.rglab--0
quay samtools 1.9--h91753b0_5 docker pull quay.io/biocontainers/samtools:1.9--h91753b0_5
quay samtools 1.3--h0592bc0_3 docker pull quay.io/biocontainers/samtools:1.3--h0592bc0_3
quay samtools 1.1--hfb9b9cc_5 docker pull quay.io/biocontainers/samtools:1.1--hfb9b9cc_5
quay samtools 0.1.16 docker pull quay.io/biocontainers/samtools:0.1.16
quay samtools 0.1.14--1 docker pull quay.io/biocontainers/samtools:0.1.14--1
quay samtools 0.1.14--0 docker pull quay.io/biocontainers/samtools:0.1.14--0
quay samtools 1.13--h8c37831_0 docker pull quay.io/biocontainers/samtools:1.13--h8c37831_0
quay samtools 1.3.1--h0592bc0_6 docker pull quay.io/biocontainers/samtools:1.3.1--h0592bc0_6
quay samtools 1.9--h91753b0_8 docker pull quay.io/biocontainers/samtools:1.9--h91753b0_8
quay samtools 1.14--hb421002_0 docker pull quay.io/biocontainers/samtools:1.14--hb421002_0
quay samtools 1.9--h8571acd_10 docker pull quay.io/biocontainers/samtools:1.9--h8571acd_10
quay samtools 1.6--h4da6232_1 docker pull quay.io/biocontainers/samtools:1.6--h4da6232_1
quay samtools 0.1.12--1 docker pull quay.io/biocontainers/samtools:0.1.12--1
quay samtools 1.9--h10a08f8_12 docker pull quay.io/biocontainers/samtools:1.9--h10a08f8_12
quay samtools 1.4--0 docker pull quay.io/biocontainers/samtools:1.4--0
quay samtools 1.9--h91753b0_3 docker pull quay.io/biocontainers/samtools:1.9--h91753b0_3
quay samtools 1.3.1--h80b0bb3_7 docker pull quay.io/biocontainers/samtools:1.3.1--h80b0bb3_7
quay samtools 1.9--h46bd0b3_0 docker pull quay.io/biocontainers/samtools:1.9--h46bd0b3_0
quay samtools 0.1.19--h270b39a_9 docker pull quay.io/biocontainers/samtools:0.1.19--h270b39a_9
quay samtools 1.6--h02bfda8_2 docker pull quay.io/biocontainers/samtools:1.6--h02bfda8_2
quay perl-bio-samtools 1.43--pl5321h5bf99c6_2 docker pull quay.io/biocontainers/perl-bio-samtools:1.43--pl5321h5bf99c6_2
quay perl-bio-samtools 1.43--0 docker pull quay.io/biocontainers/perl-bio-samtools:1.43--0
quay perl-bio-samtools 1.43--pl526h1341992_1 docker pull quay.io/biocontainers/perl-bio-samtools:1.43--pl526h1341992_1
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 1476e745a911a5a2ac22207311b275c51e745ba9-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:1476e745a911a5a2ac22207311b275c51e745ba9-0
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 adf6dbcd664c38605c6653b8ffcc270d54faceb9-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:adf6dbcd664c38605c6653b8ffcc270d54faceb9-0
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 23592e4ad15ca2acfca18facab87a1ce22c49da1-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:23592e4ad15ca2acfca18facab87a1ce22c49da1-0
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 66ed1b38d280722529bb8a0167b0cf02f8a0b488-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:66ed1b38d280722529bb8a0167b0cf02f8a0b488-0
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 fd8d2f8772eb08c49919582b0b6d4fd1ae79281b-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:fd8d2f8772eb08c49919582b0b6d4fd1ae79281b-0
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 03dc1d2818d9de56938078b8b78b82d967c1f820-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:03dc1d2818d9de56938078b8b78b82d967c1f820-0
quay mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40 eabfac3657eda5818bae4090db989e3d41b01542-0 docker pull quay.io/biocontainers/mulled-v2-fe8faa35dbf6dc65a0f7f5d4ea12e31a79f73e40:eabfac3657eda5818bae4090db989e3d41b01542-0
Anyone interested to write this up for the readme file?
For me, I tried to find seaborn+pandas using "mulled-search -d quay -s seaborn pandas", but it returned nothing. However, I know these is at least one mulled package for seaborn+pandas. Does anyone have an idea about this? Thanks
try seaboarn+pandas+python. I think this is what you specified :)
Thanks @bgruening. I tried seaborn+pandas+python, but it still returned nothing.
(mulled) [~]$ mulled-search -d quay -s seaborn+pandas+python
No results found for that query.
A correct command looks like that mulled-search -s seaborn pandas python
. But you are correct we need to improve the UX here at some point.
Can you try this one please https://quay.io/repository/biocontainers/mulled-v2-d14219255233ee6cacc427e28a7caf8ee42e8c91?tab=tags&tag=latest
Thanks @bgruening. The image you sent me works well!
Hi @bgruening
When using galaxy-tool-util==21.9.2
and the search command in your comment, I was unable to produce output listing mulled multi-package containers:
mulled-search -d quay -s bwa samtools
outputWhen trying to use the same API for quay.io as the mulled-search
tools (https://github.com/galaxyproject/galaxy/blob/a972d234c290e13bdbc616e426ad1b3be49ac644/lib/galaxy/tool_util/deps/mulled/mulled_search.py#L35), I notice that I am only getting 100 results from that API URL with a link to the next page, however, I am not seeing any handling of this in the tool code, which appears to be assuming that the full list of repositories is being returned by the API.
{
"repositories": [{"100": "repos only"},
"next_page": "gAAAAABilO179Su1xqwOJQ2r9YxmexUmt3c-RHjrDCC2ih-YhrGACYvQF46avBstLp2W4V5idiT360iaAjsjn1H1jGmlli1kKQCyyAsE4NXtmp7uCkYOI0RLbrTtPYg8NzDwEfX7MZS3p7weIFJJ0mQRpyc9bOMSGA=="
}
It looks like there is a next_page
HTTP param for the https://quay.io/api/v1/repository
endpoint (https://docs.quay.io/api/swagger/#!/repository/listRepos)
On a related note, I am unable to find info for single packages with mulled-search
that should exist on quay.io such as Mappy (https://quay.io/repository/biocontainers/mappy?tab=tags&tag=latest):
$ mulled-search -d quay -s mappy
No results found for that query.
Is there an alternative way to search for which tool combos map to which hashes/multi-package containers? e.g something like the hash.tsv file with an extra column?
Thanks!
Follow-up to my previous comment
I hacked together a basic script using code from galaxyproject/galaxy to transform one or more entries from the [hash.tsv]() file (or an arbitrary list of packages) into the "mulled" image name with a link to Quay.io and a guess at what the Docker pull command would be:
$ python multi_container_mulled_hash.py "python=3.9,pandas=1.3.0,seaborn=0.11.0,rich=10.6.0,typer=0.3.2,bcbio-gff=0.6.6,dna_features_viewer=3.0.3"
Packages : python=3.9,pandas=1.3.0,seaborn=0.11.0,rich=10.6.0,typer=0.3.2,bcbio-gff=0.6.6,dna_features_viewer=3.0.3
# packages : 7
Mulled name : mulled-v2-596f42d854e849eb773ecd1b48f2b698c2d09c9f:400d0a2593841aa0bfa3402fe85debd55a29cf37
Image Name : mulled-v2-596f42d854e849eb773ecd1b48f2b698c2d09c9f
Version/Tag Name : 400d0a2593841aa0bfa3402fe85debd55a29cf37
Quay.io URL : https://quay.io/repository/biocontainers/mulled-v2-596f42d854e849eb773ecd1b48f2b698c2d09c9f
Docker pull command (quess): quay.io/biocontainers/mulled-v2-596f42d854e849eb773ecd1b48f2b698c2d09c9f:400d0a2593841aa0bfa3402fe85debd55a29cf37-0
The script doesn't actually check if the multi-package container exists and only uses the same hashing functions as Galaxy to produce the "Image Name". It seems to do what I want it to do so far.
Script available here: https://gist.github.com/peterk87/c28167ceffbb36dcbcf5b5b225c4043d
$ cat hash.tsv | awk -F'\t' 'NR>1 {print $1}' | parallel python multi_container_mulled_hash.py > hash-tsv-mulled-image-names.txt
)Hope the script and the info above is useful for others that are having trouble figuring out what the image name is for their multi-containers!
I added some help text to the main readme, hope that helps. https://github.com/BioContainers/multi-package-containers/blob/master/README.md#searching-for-containers
@peterk87 there is already a tool for this. Its called mulled-hash
.
> mulled-hash changeo=1.2.0,igblast=1.17.1,wget=1.20.1
mulled-v2-7d8e418eb73acc6a80daea8e111c94cf19a4ecfd:e7f88c6f7da46a5407f261ca406c050d5bd12dea
Please feel free to enhance this tool and get this upstreamed.
I'm closing this one now. Please feel free to open it again if anything is unclear.
Dear BioContainers team,
Is there a way to know if a given combo is already built?
Say I want to know if bwa+samtools exists already, how do I find that if it is already available on BioContainers
Thanks in advance