Password supplied via environment variable in `environment.yaml` is expanded in `conda-lock.yml`

wholtz commented 5 months ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

I have an environment.yaml:

name: base
platforms:
  - linux-64
channels:
  - conda-forge
pip-repositories:
  - https://aws:$CODEARTIFACT_PASSWORD@sanitized-example.d.codeartifact.us-west-2.amazonaws.com/pypi/sanitized/simple/
dependencies:
  - python=3.9
  - pip
  - pip:
      - privatepackage==1.2.3

and I run:

conda-lock lock -f environment.yaml

The resulting conda-lock.yml contains the expanded value of the $CODEARTIFACT_PASSWORD environmental variable. I expected this environment variable would not be expanded in the conda-lock.yml/

Conda Info

I don't have conda or mamba installed. Only micromamba.
micromamba info 

       libmamba version : 1.5.3
     micromamba version : 1.5.3
           curl version : libcurl/8.4.0 SecureTransport (OpenSSL/3.1.4) zlib/1.2.13 zstd/1.5.5 libssh2/1.11.0 nghttp2/1.58.0
     libarchive version : libarchive 3.7.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.5
       envs directories : /Users/willholtz/micromamba/envs
          package cache : /Users/willholtz/micromamba/pkgs
                          /Users/willholtz/.mamba/pkgs
            environment : None (not found)
           env location : -
      user config files : /Users/willholtz/.mambarc
 populated config files :
       virtual packages : __unix=0=0
                          __osx=14.2.1=0
                          __archspec=1=arm64
               channels :
       base environment : /Users/willholtz/micromamba
               platform : osx-arm64

Conda Config

micromamba  config sources
Configuration files (by precedence order):

Conda list

micromamba list
List of packages in environment: ""

Additional Context

I also tried supplying the the environmental variable inside {}:

  - https://aws:${CODEARTIFACT_PASSWORD}@sanitized-example.d.codeartifact.us-west-2.amazonaws.com/pypi/sanitized/simple/

but there was no change in observed behavior.

maresb commented 5 months ago

Oh, that's bad. Thanks for the report. I will try to figure out what's going on.

maresb commented 5 months ago

I'd like a reproducer that doesn't strictly rely on a private pypi repo. I tried this but failed to reproduce the problem:

name: base
platforms:
  - linux-64
channels:
  - conda-forge
pip-repositories:
  - https://$USER:$PASS@pypi.org/simple/
dependencies:
  - python=3.9
  - pip
  - pip:
      - cowsay

maresb commented 5 months ago

Reference: https://github.com/conda/conda-lock/pull/529

Noteworthy is the mock private pypi.

maresb commented 5 months ago

Still trying to figure out what's going on. This problem should have been caught by this test:

https://github.com/conda/conda-lock/blob/2c2999bb9805dbc4abe46e38f7d70d038728b31c/tests/test_pip_repositories.py#L164-L172

@jacksmith15, do you have any insights?

wholtz commented 4 months ago

I've been executing my lock file generation in parallel to the test test_it_uses_pip_repositories_with_env_var_substitution[micromamba] to see where there is some divergence in the env var being in the URL vs the expanded value. Before line https://github.com/conda/conda-lock/blob/431c09b2cc7e1c9ab7bda6c5b37c87083a000572/conda_lock/pypi_solver.py#L361 they appear to be similar, but the value of link diverges.

From the test:

<Link http://private-pypi.org/files/fake-private-package-1.0.0.tar.gz (from <conda_lock._vendor.poetry.repositories.legacy_repository.Page object at 0x109aa3c50>)>

From my case:

<Link https://aws:*************@sanitized-example.d.codeartifact.us-west-2.amazonaws.com/pypi/sanatized/simple/tabulate/0.9.0/tabulate-0.9.0-py3-none-any.whl#sha256=024ca478df22e9340661486f85298cff5f6dcdba14f3813e8830015b9ed1948f (from <conda_lock._vendor.poetry.repositories.legacy_repository.Page object at 0x107a882d0>) (requires-python:>=3.7)>

Where the ************ is actually my password.

Before https://github.com/conda/conda-lock/blob/431c09b2cc7e1c9ab7bda6c5b37c87083a000572/conda_lock/pypi_solver.py#L361 op.package.source_url for both cases included the expanded password.

wholtz commented 4 months ago

I got further. Here is a stack trace to a point in execution where I believes the problem lies:

  /Users/willholtz/micromamba/envs/conda-lock-dev/lib/python3.11/site-packages/click/core.py(1078)main()
-> rv = self.invoke(ctx)
  /Users/willholtz/micromamba/envs/conda-lock-dev/lib/python3.11/site-packages/click/core.py(1688)invoke()
-> return _process_result(sub_ctx.command.invoke(sub_ctx))
  /Users/willholtz/micromamba/envs/conda-lock-dev/lib/python3.11/site-packages/click/core.py(1434)invoke()
-> return ctx.invoke(self.callback, **ctx.params)
  /Users/willholtz/micromamba/envs/conda-lock-dev/lib/python3.11/site-packages/click/core.py(783)invoke()
-> return __callback(*args, **kwargs)
  /Users/willholtz/micromamba/envs/conda-lock-dev/lib/python3.11/site-packages/click/decorators.py(33)new_func()
-> return f(get_current_context(), *args, **kwargs)
  /Users/willholtz/repos/conda-lock/conda_lock/conda_lock.py(1401)lock()
-> else:
  /Users/willholtz/repos/conda-lock/conda_lock/conda_lock.py(1108)run_lock()
-> conda_exe, mamba=mamba, micromamba=micromamba
  /Users/willholtz/repos/conda-lock/conda_lock/conda_lock.py(393)make_lock_files()
-> fresh_lock_content = create_lockfile_from_spec(
  /Users/willholtz/repos/conda-lock/conda_lock/conda_lock.py(835)create_lockfile_from_spec()
-> for platform in platforms or spec.platforms:
  /Users/willholtz/repos/conda-lock/conda_lock/conda_lock.py(759)_solve_for_arch()
-> raise ValueError("Got pip specs without Python")
  /Users/willholtz/repos/conda-lock/conda_lock/pypi_solver.py(488)solve_pypi()
-> result = s.solve(use_latest=to_update)
  /Users/willholtz/repos/conda-lock/conda_lock/pypi_solver.py(361)get_requirements()
-> link = chooser.choose_for(op.package)
  /Users/willholtz/repos/conda-lock/conda_lock/_vendor/poetry/installation/chooser.py(60)choose_for()
-> for link in self._get_links(package):
  /Users/willholtz/repos/conda-lock/conda_lock/_vendor/poetry/installation/chooser.py(94)_get_links()
-> links = repository.find_links_for_package(package)
  /Users/willholtz/repos/conda-lock/conda_lock/_vendor/poetry/repositories/legacy_repository.py(331)find_links_for_package()
-> page = self._get("/{}/".format(package.name.replace(".", "-")))
> /Users/willholtz/repos/conda-lock/conda_lock/_vendor/poetry/repositories/legacy_repository.py(410)_get()
-> return Page(response.url, response.content, response.headers)

When I'm running my use case, within that final call to _get(), response.url contains the (expanded) username and password. When the unit test test_it_uses_pip_repositories_with_env_var_substitution[micromamba] executes, response.url does not contain the username and password. I have no idea if having the username and password in the response is standard or not, but this is coming from AWS CodeArtifact, so I'm sure I won't be the only person to hit this.

I haven't given any though to a solution yet, but it seems to me that the test for my case is to change https://github.com/conda/conda-lock/blob/2c2999bb9805dbc4abe46e38f7d70d038728b31c/tests/test_pip_repositories.py#L87 to

response.url = request.url

conda / conda-lock