search

anaconda / docker-images

Repository of Docker images created by Anaconda
https://hub.docker.com/u/continuumio/
819 stars 282 forks source link

Anaconda3 Docker image does not have pip #122

Closed jennirinker closed 5 years ago

jennirinker commented 5 years ago

This is likely related to #120, but I'm not familiar enough with the linux build to explain.

My CI/CD pipelines that rely on the anaconda docker image have broken because pip is no longer recognized. I've tried it on two separate repos that both rely on the image and they both fail now.

Example pipeline config block

test_pyconturb:  # name the job what we like
  image: continuumio/anaconda3:latest
  stage:  # build, test, deploy defined by default [2]
    test
  script:  # install dev options, then code, then run tests and do coverage report
  - pip install pytest pytest-cov
  - pip install -e .
  - python -m pytest -m "not long" --cov-report term-missing:skip-covered --cov=pyconturb --cov-config .coveragerc
  tags:  # only runners with this tag can do the job [3]
  - python

Example of failed pipeline output

Running with gitlab-runner 10.0.2 (a9a76a50)
  on synnefo-docker-iamge (761634a5)
Using Docker executor with image continuumio/anaconda3:latest ...
Using docker image sha256:8932848158968b014b012a13b2bb172aba7d0f769277cfbdd858942194f2d205 for predefined container...
Pulling docker image continuumio/anaconda3:latest ...
Using docker image continuumio/anaconda3:latest ID=sha256:9e186aada2eeee49531aa19229cf727fbab5a0d31aa15470cdf7e733647411d6 for build container...
Running on runner-761634a5-project-570-concurrent-0 via snf-713...
Fetching changes...
HEAD is now at 967a398 fixing tests
From https://gitlab.windenergy.dtu.dk/rink/pyconturb
   967a398..3f28676  restruct   -> origin/restruct
Checking out 3f286762 as restruct...
Skipping Git submodules setup
/bin/sh: eval: line 66: pip: not found
$ pip install pytest pytest-cov
ERROR: Job failed: exit code 127
soapy1 commented 5 years ago

I'll need more info about what your ci/cd is doing to provide a better answer but pip seems to work for me. For example:

Startup docker container

$ docker run -it  continuumio/anaconda3:latest

Double check I have the conda I think I have - yep

(base) 36454543f5e3:/$ conda info

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/anaconda/.condarc
 populated config files :
          conda version : 4.6.11
    conda-build version : 3.17.8
         python version : 3.7.3.final.0
       base environment : /opt/conda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/anaconda/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/anaconda/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.11 requests/2.21.0 CPython/3.7.3 Linux/4.9.125-linuxkit alpine/3.9.2 glibc/2.28
                UID:GID : 10151:101
             netrc file : None
           offline mode : False

Check if I have pip - yep

(base) 36454543f5e3:/$ which -a pip
/opt/conda/bin/pip

Check if I can install stuff with pip - yep

(base) 5fe104b4b70e:/$ pip install pytest pytest-cov
Requirement already satisfied: pytest in /opt/conda/lib/python3.7/site-packages (4.3.1)
Collecting pytest-cov
  Downloading https://files.pythonhosted.org/packages/7d/b5/92f32674ab954f80499ac73347bfeb815545ea295439c12b0ef3ac8f0975/pytest_cov-2.6.1-py2.py3-none-any.whl
Requirement already satisfied: py>=1.5.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (1.8.0)
Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (1.12.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from pytest) (40.8.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (19.1.0)
Requirement already satisfied: atomicwrites>=1.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (1.3.0)
Requirement already satisfied: pluggy>=0.7 in /opt/conda/lib/python3.7/site-packages (from pytest) (0.9.0)
Requirement already satisfied: more-itertools>=4.0.0 in /opt/conda/lib/python3.7/site-packages (from pytest) (6.0.0)
Collecting coverage>=4.4 (from pytest-cov)
  Downloading https://files.pythonhosted.org/packages/b4/6a/8e2918c56682739cf177a7358929fed427d3982cc649e859b7a2d5c7daea/coverage-4.5.3-cp37-cp37m-manylinux1_x86_64.whl (204kB)
    100% |████████████████████████████████| 215kB 184kB/s
Installing collected packages: coverage, pytest-cov
Successfully installed coverage-4.5.3 pytest-cov-2.6.1

Maybe pip is returning a non-zero exit code????? - nope

(base) 5fe104b4b70e:/$ echo $?
0

Pip seems to be working fine

soapy1 commented 5 years ago

so, I poked around a little bit more, and if you run the docker image overriding the entrypoint, then you can get into a sticky situation

$ docker run -it  continuumio/anaconda3:latest sh
/ $ which -a pip
/ $

so, my guess is that your ci/cd is overwriting the entrypoint somehow? I can't really test this since I don't have gitlab steup but maybe this is a good jumping off point for your investigation. I suspect that not setting an entrypoint, or setting it to sh --login -i should solve your issue.

eg.

$ docker run -it  continuumio/anaconda3:latest sh --login -i
(base) 3b38517696e5:/$ which -a pip
/opt/conda/bin/pip
leycec commented 5 years ago

@jennirinker: The exact same failure has begun afflicting our GitLab-CI pipelines for the past several days. I also note a suspiciously similar GitLab issue submitted yesterday, entitled: Kubernetes executor ignores Docker ENTRYPOINT. (Insert squinty eyes meme here.)

@soapy1 almost certainly has the right of it. This does appear to be a GitLab-CI regression introduced by the most recent stable release of GitLab. It would be super helpful, @jennirinker, if you wouldn't mind submitting a new GitLab-CI issue on this topic or commenting in the aforementioned GitLab-CI issue.

Thanks for the generous assistance, @soapy1. While the recent Alpine switch hasn't been without its issues, this doesn't seem to be one of them. Glorious pineapples for all! :pineapple: :pineapple: :pineapple:

jennirinker commented 5 years ago

Got it, thanks! I'll submit the issue when I get to work. Thanks @soapy1 for digging into it and @leycec for the further info. :)

I'll close this issue since it's likely GitLab and not the docker image.

leycec commented 5 years ago

@soapy1: Wait. This might be an Anaconda-side issue after all.

By inspection, the latest Anaconda3 Dockerfile doesn't actually define an ENTRYPOINT at all. It does define a CMD, but that's absolutely not the same thing. While the two are often understandably confused, best practices in the Dockerfile community suggest that most Anaconda Dockerfiles should probably define at least an ENTRYPOINT (and possibly but not necessarily also a CMD). Specifically:

Alpine-based Anaconda Dockerfiles appear to require usage of at least sh and likely also --login -i to ensure sane startup (e.g., environment variable initialization). It thus follows that:

# Replacing this line...
CMD [ "sh", "--login", "-i" ]

# ...with this line...
ENTRYPOINT [ "sh", "--login", "-i" ]

# ...or possibly these two lines...
ENTRYPOINT [ "sh" ]
CMD [ "--login", "-i" ]

...is likely to yield end user happiness.

jennirinker commented 5 years ago

It would make sense that this is something on Anaconda's side, since I've done some further testing today and other images don't have this problem. I'll reopen it. cc @soapy1

EDIT: Hmm, I just recreated the issue using python's 3.6-slim image, which is based on debian (dockerfile). That file does also not have an entrypoint defined, but it ends with a cmd command.

soapy1 commented 5 years ago

@jennirinker could you share what other images you used that did not have this problem?

pierre-nicolas-nelio commented 5 years ago

Just in case there should be a link, we encountered a problem while using a custom entrypoint as well.

Dockerfile:

FROM continuumio/anaconda3:latest
ADD environment.yml /home/root/
# The RUN statement below did not work without tweaking the PATH :(
ENV PATH /opt/conda/bin/:$PATH
RUN conda env create -n myenv -f /home/root/environment.yml
WORKDIR /home/root
# add env activation to .bashrc, sourced in entrypoint.sh
RUN echo "source activate myenv" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
ADD ./entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh:

#!/bin/bash
source ~/.bashrc
exec "$@"

It fails on run, I haven't got the error right now unfortunately.

(Ignore this message if unrelated, we found workarounds reverting to the old anaconda container.)

soapy1 commented 5 years ago

I'm pushing a new docker image continuumio/anaconda3:test with some changes from https://github.com/ContinuumIO/docker-images/pull/124

This should also resolve https://github.com/ContinuumIO/docker-images/issues/119

@jennirinker when you have a sec, please try this out

soapy1 commented 5 years ago

Ok so, we've made some additional changes. The latest and 2019.03 images have been rebuilt to use debian (like the old images). You should be able to update safely now. We added a new tag -alpine. This has the alpine images in case anyone still wants to use them. Hopefully this will clear up issues, while still providing access to the new anaconda.

That being said, testing to see if the docker image tagged as test would still be greatly appreciated.

jennirinker commented 5 years ago

Running off latest, it almost works...except make is not found so I can't build my sphinx docs. If I reinstall it before running my build scripts, it works. pip and git are fine.

For test, git is not found.

Incidentally, I had similar problems with the python 3.6 slim image: neither make nor git were found and I had to reinstall them in the before_script block.

soapy1 commented 5 years ago

Thanks @jennirinker! I think it's acceptable for make and git to not be in those images, they aren't really conda specific things.