search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
456 stars 101 forks source link

Mixed conda / private pip lock file does not install inside docker container #623

Open metgrahamr opened 2 months ago

metgrahamr commented 2 months ago

Checklist

What happened?

I created a lock file using a mix of conda-forge packages and private pip packages in an AWS CodeArtifact repo in a similar way to here but as I am using conda-lock version 2.5.6 I don't have the issue with the value of the CodeArtifact password being passed through. With the CodeArtifact password set as an environment variable, I can successfully use this lock file to create a new conda environment. However, if I try to do this from within a docker container then I get the following error

conda-lock install -p /opt/gauge /locks/conda-lock.yml
INFO:root:Downloading and Extracting Packages: ...working... done
INFO:root:
INFO:root:Downloading and Extracting Packages: ...working... done
INFO:root:Preparing transaction: ...working... done
INFO:root:Verifying transaction: ...working... done
INFO:root:Executing transaction: ...working... done
ERROR:root:ERROR: Exception:
ERROR:root:Traceback (most recent call last):
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
ERROR:root:    status = run_func(*args)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
ERROR:root:    return func(self, options, args)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
ERROR:root:    requirement_set = resolver.resolve(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
ERROR:root:    collected = self.factory.collect_root_requirements(root_reqs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 534, in collect_root_requirements
ERROR:root:    reqs = list(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
ERROR:root:    cand = self._make_base_candidate_from_link(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
ERROR:root:    self._link_candidate_cache[link] = LinkCandidate(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
ERROR:root:    super().__init__(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
ERROR:root:    self.dist = self._prepare()
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
ERROR:root:    dist = self._prepare_distribution()
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
ERROR:root:    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
ERROR:root:    return self._prepare_linked_requirement(req, parallel_builds)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 596, in _prepare_linked_requirement
ERROR:root:    local_file = unpack_url(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 168, in unpack_url
ERROR:root:    file = get_http_url(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 109, in get_http_url
ERROR:root:    from_path, content_type = download(link, temp_dir.path)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/download.py", line 134, in __call__
ERROR:root:    resp = _http_get_download(self._session, link)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/download.py", line 117, in _http_get_download
ERROR:root:    resp = session.get(target_url, headers=HEADERS, stream=True)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 602, in get
ERROR:root:    return self.request("GET", url, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/session.py", line 520, in request
ERROR:root:    return super().request(method, url, *args, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 589, in request
ERROR:root:    resp = self.send(prep, **send_kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 710, in send
ERROR:root:    r = dispatch_hook("response", hooks, r, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/hooks.py", line 30, in dispatch_hook
ERROR:root:    _hook_data = hook(hook_data, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 500, in handle_401
ERROR:root:    username, password, save = self._prompt_for_password(parsed.netloc)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 455, in _prompt_for_password
ERROR:root:    username = ask_input(f"User for {netloc}: ") if self.prompting else None
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/utils/misc.py", line 251, in ask_input
ERROR:root:    return input(message)
ERROR:root:EOFError: EOF when reading a line
ERROR:root:
ERROR:root:ERROR conda.cli.main_run:execute(124): `conda run pip install --no-deps -r /tmp/tmp7ssjgkkc` failed. (See above for error)
INFO:root:Collecting arnparse@ https://aws:****@sample-111222333444.d.codeartifact.us-west-2.amazonaws.com/pypi/repo-name/simple/arnparse/0.0.2/arnparse-0.0.2-py2.py3-none-any.whl#sha256=b0906734e4b8f19e39b1e32944c6cd6274b6da90c066a83882ac7a11d27553e0 (from -r /tmp/tmp7ssjgkkc (line 1))
INFO:root:User for sample-111222333444.d.codeartifact.us-west-2.amazonaws.com:
Traceback (most recent call last):
  File "/opt/conda/bin/conda-lock", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1497, in click_install
    install(
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1551, in install
    install_func(file=lockfile)
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 240, in do_conda_install
    _conda(["run"], ["pip", "install", "--no-deps", "-r", str(requirements_path)])
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/invoke_conda.py", line 143, in _invoke_conda
    raise subprocess.CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['/opt/conda/bin/conda', 'run', '--prefix', '/opt/gauge', 'pip', 'install', '--no-deps', '-r', '/tmp/tmp7ssjgkkc']' returned non-zero exit status 2.

I wondered whether the error was anything to do with this and tried adding --no-capture-output to the conda run command that gets called. This just caused the install to hang at the first package install. Running the pip install command manually with the requirements file in the /tmp directory prompted for a user and then password. If I entered aws and the CodeArtifact password at each prompt then pip was able to install each package.

It looks like when running inside a docker container the install command is not getting the password from the environment and is trying to prompt for the information. How do I get this lock file to work inside a docker container in the same way that it works outside?

Conda Info

active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : 
          conda version : 24.3.0
    conda-build version : not installed
         python version : 3.11.7.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.3.0=0
                          __glibc=2.31=0
                          __linux=5.15.146.1=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/24.3.0 requests/2.31.0 CPython/3.11.7 Linux/5.15.146.1-microsoft-standard-WSL2 debian/11.9 glibc/2.31 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.6
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Conda Config

No response

Conda list

No response

Additional Context

I am using the continuumio/miniconda3:latest docker image and install conda-lock into the base environment using pip.

maresb commented 2 months ago

Hi @metgrahamr! This seems like a tricky one. Thanks a lot for experimenting with --no-capture-output, that's pretty interesting.

I think we'll need to drill down quite a bit more before we can isolate the cause of this issue.

  1. Could you clarify exactly what you mean by "within a docker container"? Is this from a bash prompt following a docker run command, or is this occurring in a Dockerfile via docker build?
  2. How are you creating the environment variables with the credentials? Are they coming from the -e argument to docker run?
  3. How exactly are you referencing the environment variables within your original environment definition? And how are those environment variables being translated within your lockfile? There is some quirkiness where sometimes you must use ${ENV_VAR} instead of $ENV_VAR when referencing the environment variables.

It would help a lot with the diagnosis if you could share relevant snippets (or better the whole thing minus personally identifiable info if not confidential) from your original environment definition, the generated lockfile, and the particular docker commands you're executing. Thanks!

metgrahamr commented 2 months ago 
1. Could you clarify exactly what you mean by "within a docker container"? Is this from a `bash` prompt following a `docker run` command, or is this occurring in a `Dockerfile` via `docker build`?

I initially tried this in a Dockerfile as I want to use conda-lock to install all the dependencies for an AWS Lambda function. However, when it didn't work I also fired up a container with docker run and tried replicating what I was doing in the Dockerfile but got the same results.

2. How are you creating the environment variables with the credentials? Are they coming from the `-e` argument to `docker run`?

For the Dockerfile I used ARG and supplied the CodeArtifact token as a build argument. When using docker run I created an environment variable once inside the container. I checked that it was available in the environment, e.g. echo $CODEARTIFACT_AUTH_TOKEN and even tried CODEARTIFACT_AUTH_TOKEN=$(echo $CODEARTIFACT_AUTH_TOKEN) conda-lock install .... When I tried running the requirements file directly I was prompted for the user and password and this worked (although obviously not via the environment variable).

3. How exactly are you referencing the environment variables within your original environment definition? And how are those environment variables being translated within your lockfile? There is some quirkiness where sometimes you must use `${ENV_VAR}` instead of `$ENV_VAR` when referencing the environment variables.
ARG FUNCTION_DIR="/opt/gauge/"
ARG GENERATOR="gauge"

FROM continuumio/miniconda3:latest as BUILDER

ARG FUNCTION_DIR
ARG GENERATOR
ARG CODEARTIFACT_AUTH_TOKEN

RUN pip install conda-lock
COPY conda-lock.yml /locks/conda-lock.yml
RUN CODEARTIFACT_AUTH_TOKEN=${CODEARTIFACT_AUTH_TOKEN} conda-lock install -p ${FUNCTION_DIR} /locks/conda-lock.yml

I then pass the credentials in like this

docker build . -t gauge_lock --build-arg CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain sample --domain-owner 111222333444 --region us-west-2 --query authorizationToken --output text`

The env file looks like this

name: gauge
channels:
  - conda-forge
pip-repositories:
  - https://aws:$CODEARTIFACT_AUTH_TOKEN@sample-111222333444.d.codeartifact.us-west-2.amazonaws.com/pypi/repo/simple/
dependencies:
  - python=3.9
  - pyyaml
  - pip
  - pip:
    - aws-lambda-powertools
    - awslambdaric
    - fr-helpers  # a private package in the CodeArtifact repo

platforms:
  - linux-64

conda-lock.yml

Thanks for looking into this (and the project as a whole) it's much appreciated.

maresb commented 2 months ago

Hey @metgrahamr, thanks a lot for bearing with my questions. This seems pretty challenging to debug, and unfortunately I wasn't able to get to it over the weekend. Just wanted you to know that I haven't forgotten. Please feel free to ping me again later or continue debugging yourself.

metgrahamr commented 2 months ago

@maresb Thanks for the update and for keeping this under consideration.