Closed NavjothM closed 1 year ago
mmcguffi
commented
2 years ago @NavjothM Ive only used docker, so I don't have any personal experience with singularity. Looking at that error though, my guess is that the streamlit caching may not be working or perhaps infernal might be not working.
What is the output of: conda list ?
Also, how did you build the container?
NavjothM
commented
2 years ago Thank you so much for the quick response!
conda list output inside singularity container
countainer was made from a pre-existing docker image docker save [image_id] -o plannotate.tar singularity build plannotate.sif docker-archive://plannotate.tar
Thanks
mmcguffi
commented
2 years ago @NavjothM ah, this looks like your base conda environment? Inside your singularity container, what does conda list output look like once you've activated the conda plannotate environment? (conda activate plannotate)
NavjothM
commented
2 years ago Below is the output for conda list
(plannotate) Singularity> conda list
# packages in environment at /opt/conda/envs/plannotate:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge
altair 4.2.0 pyhd8ed1ab_1 conda-forge
argh 0.26.2 pyh9f0ad1d_1002 conda-forge
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py37h5e8e339_1 conda-forge
arrow-cpp 6.0.1 py37hbd77c41_5_cpu conda-forge
astor 0.8.1 pyh9f0ad1d_0 conda-forge
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
aws-c-auth 0.6.8 hadad3cd_1 conda-forge
aws-c-cal 0.5.12 h70efedd_7 conda-forge
aws-c-common 0.6.17 h7f98852_0 conda-forge
aws-c-compression 0.2.14 h7c7754b_7 conda-forge
aws-c-event-stream 0.2.7 hd2be095_32 conda-forge
aws-c-http 0.6.10 h416565a_3 conda-forge
aws-c-io 0.10.14 he836878_0 conda-forge
aws-c-mqtt 0.7.10 h885097b_0 conda-forge
aws-c-s3 0.1.29 h8d70ed6_0 conda-forge
aws-c-sdkutils 0.1.1 h7c7754b_4 conda-forge
aws-checksums 0.1.12 h7c7754b_6 conda-forge
aws-crt-cpp 0.17.10 h6ab17b9_5 conda-forge
aws-sdk-cpp 1.9.160 h36ff4c5_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
backports.zoneinfo 0.2.1 py37h5e8e339_4 conda-forge
base58 2.1.1 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.10.0 pyha770c72_0 conda-forge
biopython 1.78 py37h5e8e339_2 conda-forge
blast 2.10.1 pl526he19e7b1_3 bioconda
bleach 4.1.0 pyhd8ed1ab_0 conda-forge
blinker 1.4 py_1 conda-forge
bokeh 2.4.1 py37h89c1867_2 conda-forge
boost-cpp 1.74.0 h6cacc03_7 conda-forge
boto3 1.21.18 pyhd8ed1ab_0 conda-forge
botocore 1.24.18 pyhd8ed1ab_0 conda-forge
brotlipy 0.7.0 py37h5e8e339_1003 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cachetools 5.0.0 pyhd8ed1ab_0 conda-forge
certifi 2021.10.8 py37h89c1867_1 conda-forge
cffi 1.15.0 py37h036bc23_0 conda-forge
charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cryptography 36.0.1 py37hf1a17b8_0 conda-forge
curl 7.82.0 h7bff187_0 conda-forge
debugpy 1.5.1 py37hcd2ae1e_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
diamond 2.0.13 hdcc8f71_0 bioconda
entrez-direct 16.2 he881be0_0 bioconda
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
flit-core 3.7.1 pyhd8ed1ab_0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
gitdb 4.0.9 pyhd8ed1ab_0 conda-forge
gitpython 3.1.27 pyhd8ed1ab_0 conda-forge
glog 0.5.0 h48cff8f_0 conda-forge
grpc-cpp 1.42.0 ha1441d3_1 conda-forge
icu 69.1 h9c3ff4c_0 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.11.3 py37h89c1867_0 conda-forge
importlib_resources 5.4.0 pyhd8ed1ab_0 conda-forge
infernal 1.1.4 h779adbc_0 bioconda
ipykernel 6.9.1 py37h6531663_0 conda-forge
ipython 7.32.0 py37h89c1867_0 conda-forge
ipython-genutils 0.2.0 pypi_0 pypi
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.5 pyhd8ed1ab_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.18.1 py37h89c1867_0 conda-forge
jinja2 3.0.3 pyhd8ed1ab_0 conda-forge
jmespath 0.10.0 pyh9f0ad1d_0 conda-forge
jpeg 9e h7f98852_0 conda-forge
jsonschema 4.4.0 pyhd8ed1ab_0 conda-forge
jupyter_client 7.1.2 pyhd8ed1ab_0 conda-forge
jupyter_core 4.9.2 py37h89c1867_0 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_widgets 1.0.2 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.19.2 h3790be6_4 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 13_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h7f98852_6 conda-forge
libbrotlidec 1.0.9 h7f98852_6 conda-forge
libbrotlienc 1.0.9 h7f98852_6 conda-forge
libcblas 3.9.0 13_linux64_openblas conda-forge
libcurl 7.82.0 h7bff187_0 conda-forge
libdeflate 1.10 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 h9b69904_4 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 11.2.0 h1d223b6_13 conda-forge
libgfortran-ng 11.2.0 h69a702a_13 conda-forge
libgfortran5 11.2.0 h5c6108e_13 conda-forge
libgomp 11.2.0 h1d223b6_13 conda-forge
liblapack 3.9.0 13_linux64_openblas conda-forge
libnghttp2 1.47.0 h727a467_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libprotobuf 3.19.4 h780b84a_0 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_13 conda-forge
libthrift 0.15.0 he6d91bd_1 conda-forge
libtiff 4.3.0 h542a066_3 conda-forge
libutf8proc 2.7.0 h7f98852_0 conda-forge
libwebp 1.2.2 h3452ae3_0 conda-forge
libwebp-base 1.2.2 h7f98852_1 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markupsafe 2.1.0 py37h540881e_1 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
mistune 0.8.4 py37h5e8e339_1005 conda-forge
nbclient 0.5.13 pyhd8ed1ab_0 conda-forge
nbconvert 6.4.4 py37h89c1867_0 conda-forge
nbformat 5.2.0 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h9c3ff4c_0 conda-forge
nest-asyncio 1.5.4 pyhd8ed1ab_0 conda-forge
notebook 6.4.8 pyha770c72_0 conda-forge
numpy 1.21.5 py37hf2998dd_0 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1l h7f98852_0 conda-forge
orc 1.7.1 h1be678f_1 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py37he8f5f7f_0 conda-forge
pandoc 2.17.1.1 ha770c72_0 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
perl 5.26.2 h36c2ea0_1008 conda-forge
perl-archive-tar 2.32 pl526_0 bioconda
perl-carp 1.38 pl526_3 bioconda
perl-common-sense 3.74 pl526_2 bioconda
perl-compress-raw-bzip2 2.087 pl526he1b5a44_0 bioconda
perl-compress-raw-zlib 2.087 pl526hc9558a2_0 bioconda
perl-exporter 5.72 pl526_1 bioconda
perl-exporter-tiny 1.002001 pl526_0 bioconda
perl-extutils-makemaker 7.36 pl526_1 bioconda
perl-io-compress 2.087 pl526he1b5a44_0 bioconda
perl-io-zlib 1.10 pl526_2 bioconda
perl-json 4.02 pl526_0 bioconda
perl-json-xs 2.34 pl526h6bb024c_3 bioconda
perl-list-moreutils 0.428 pl526_1 bioconda
perl-list-moreutils-xs 0.428 pl526_0 bioconda
perl-pathtools 3.75 pl526h14c3975_1 bioconda
perl-scalar-list-utils 1.52 pl526h516909a_0 bioconda
perl-types-serialiser 1.0 pl526_2 bioconda
perl-xsloader 0.24 pl526_0 bioconda
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.0.1 py37h44f0d7a_2 conda-forge
pip 22.0.4 pyhd8ed1ab_0 conda-forge
plannotate 1.2.0 pypi_0 pypi
prometheus_client 0.13.1 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.27 pyha770c72_0 conda-forge
protobuf 3.19.4 py37hcd2ae1e_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pyarrow 6.0.1 py37h20dbb2a_5_cpu conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydeck 0.7.1 pyh6c4a22f_0 conda-forge
pygments 2.11.2 pyhd8ed1ab_0 conda-forge
pympler 0.9 pyh9f0ad1d_0 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.7 pyhd8ed1ab_0 conda-forge
pyrsistent 0.18.1 py37h5e8e339_0 conda-forge
pysocks 1.7.1 py37h89c1867_4 conda-forge
python 3.7.12 hb7a2778_100_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-tzdata 2021.5 pyhd8ed1ab_0 conda-forge
python_abi 3.7 2_cp37m conda-forge
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pytz-deprecation-shim 0.1.0.post0 py37h89c1867_1 conda-forge
pyyaml 6.0 py37h5e8e339_3 conda-forge
pyzmq 22.3.0 py37h336d617_1 conda-forge
re2 2021.11.01 h9c3ff4c_0 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.27.1 pyhd8ed1ab_0 conda-forge
ripgrep 13.0.0 h2f28480_2 conda-forge
s2n 1.3.0 h9b69904_0 conda-forge
s3transfer 0.5.2 pyhd8ed1ab_0 conda-forge
send2trash 1.8.0 pyhd8ed1ab_0 conda-forge
setuptools 59.8.0 py37h89c1867_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 3.0.5 pyh44b312d_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
soupsieve 2.3.1 pyhd8ed1ab_0 conda-forge
sqlite 3.37.0 h9cd32fc_0 conda-forge
streamlit 1.2.0 pyhd8ed1ab_1 conda-forge
tabulate 0.8.9 pyhd8ed1ab_0 conda-forge
terminado 0.13.3 py37h89c1867_0 conda-forge
testpath 0.6.0 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py37h5e8e339_2 conda-forge
traitlets 5.1.1 pyhd8ed1ab_0 conda-forge
trnascan-se 2.0.7 pl5262h779adbc_1 bioconda
typing_extensions 4.1.1 pyha770c72_0 conda-forge
tzdata 2021e he74cb21_0 conda-forge
tzlocal 4.1 py37h89c1867_1 conda-forge
urllib3 1.26.8 pyhd8ed1ab_1 conda-forge
validators 0.18.2 pyhd3deb0d_0 conda-forge
watchdog 2.1.6 py37h89c1867_1 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
widgetsnbextension 3.5.2 py37h89c1867_1 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zipp 3.7.0 pyhd8ed1ab_1 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.2 ha95c52a_0 conda-forgeThanks
mmcguffi
commented
2 years ago @NavjothM unless I am misunderstanding something, I think the plannotate installation itself is the issue:
plannotate 1.2.0 pypi_0 pypi
I haven't put plannotate on PyPI because of the relatively complex 3rd party tools required. Im not quite sure how this docker image was constructed (or why it works in docker), but that is my best guess for why this isn't working. Im not super knowledgeable on docker though, and I know very little about singularity
I guess I would try reconstructing the docker image after installing plannotate via the recommended method:
conda create -n plannotate -c conda-forge -c bioconda plannotate
Let me know if that works!
NavjothM
commented
2 years ago Hi, apologies for the delay. So the docker image was constructed by git clone and installed from source.
FROM continuumio/miniconda3
The method recommended conda create -n plannotate -c conda-forge -c bioconda plannotate showed an issue in conda-forge and was not working
Thanks
NavjothM
commented
2 years ago Is streamlit required for the CLI implementation? I do not need the web interface actually.
mmcguffi
commented
2 years ago @NavjothM streamlit is not needed for the CLI implementation. However, Im not quite sure how I would be able to create separate versions for the CLI / GUI besides explicitly creating and maintaining 2 versions of plannotate (which I do not plan to do)
Also, I'll also take a closer look at your docker/singularity issues, though it will probably take me a few weeks to get to
NavjothM
commented
2 years ago Okay thanks a lot! So I think one of the issues it that streamlit (and maybe even conda or mamba) requires some root permissions. In docker container we are the root user but on singularity build, the image runs with the permissions of the user running it.
Lets see if there's a workaround for this. I'll try what is suggested in similar threads.
Thanks for the help.
mmcguffi
commented
2 years ago @NavjothM I played around with this a little bit -- my "recommended method" also did not work for me because it's running into issues with conda finding some packages (such as BLAST) using the continuumio/miniconda3 base image
I also directly copied your dockerfile and tried to build that, but that method also cannot solve the environment because it still cannot find BLAST, etc. My guess is that it might be because these don't have a noarch tag/build within their channels?
If you have any tips or insight here on how you got your dockerfile to build, I would appreciate it :)
NavjothM
commented
2 years ago @mmcguffi There's a long known issue of implementing conda within docker :( I used something similar to this https://kevalnagda.github.io/conda-docker-tutorial and https://pythonspeed.com/articles/activate-conda-dockerfile/
Let me know if you are able to build the image.
mmcguffi
commented
2 years ago @NavjothM I did play around with it a little bit -- my specific issue turned out to be a problem with my computer (M1 Mac). Anyway, I generally followed this, I was able to get this Dockerfile to work:
FROM condaforge/mambaforge AS build
SHELL ["/bin/bash", "--login", "-c"]
RUN mamba create -n plannotate -c conda-forge -c bioconda plannotate
# Install conda-pack:
RUN conda install -c conda-forge conda-pack
# Use conda-pack to create a standalone enviornment
# in /venv:
RUN conda-pack -n plannotate -o /tmp/env.tar && \
mkdir /venv && cd /venv && tar xf /tmp/env.tar && \
rm /tmp/env.tar
# We've put venv in same path it'll be in final image,
# so now fix up paths:
RUN /venv/bin/conda-unpack
# The runtime-stage image; we can use Debian as the
# base image since the Conda env also includes Python
# for us.
FROM debian:buster AS runtime
# Copy /venv from the previous stage:
COPY --from=build /venv /venv
# When image is run, run the code with the environment
# activated:
SHELL ["/bin/bash", "-c"]
EXPOSE 8501
RUN echo "source /venv/bin/activate" >> ~/.bashrcTo build:
docker build --platform linux/x86_64 . --tag mmcguffi/plannotate_slim To run:
docker run -p 8501:8501 -it mmcguffi/plannotate_slim However, there is an issue with Streamlit and click, so either in the Dockerfile or inside the container, you have to run pip install streamlit --upgrade (there are other ways to fix this, but this seemed to be the easiest).
I want to fix that click/streamlit issue before I put this image up on Dockerhub, as well as tidy up a few errors/warnings.
I still haven't tried Singularity at all, but let me know if this helps!
NavjothM
commented
2 years ago @mmcguffi Hi, Thanks for providing the Dockerfile!
Thought I'll come back to this after a break. Tried building and using the Docker image
The tool works fine when I go inside the container and then run the commands
At terminal
docker run -p 8501:8501 -v `pwd`:/data -it mmcguffi/plannotate_slim
Then inside container
plannotate batch -i /data/old_dockerfiles/1.fasta -o /data/ -h -c
But I cant seem to run it from outside. Still getting the same error regarding Streamlit :( I think I should also note that I am using an Azure VM in which I'm testing these docker images (I am root in the VM) Did you face any issue in running the docker image from outside?
Command used : docker run -p 8501:8501 -v `pwd`:/data mmcguffi/plannotate_slim /venv/bin/plannotate batch -i /data/old_dockerfiles/1.fasta -o /data/
Will try the Singularity implementation in a bit. Apologies for the delay. Let me know if you have any inputs on this. Thanks
NavjothM
commented
1 year ago @mmcguffi Update on this, Found some singularity images here. https://depot.galaxyproject.org/singularity/
Not sure how they were able to build the image but working for me. Closing this for now.
Hi! Thanks for the great tool.
I am trying to run plannotate batch in a singularity container and I'm facing the below error. My input file is a fasta file and the command is as below plannotate batch -h -i reference.fasta
This error is not seen when I run in docker container. Hope you can help me with this issue. Thanks