Please add support for credentials via environment variables

[venthur]

• Pip version: all
• Python version: all
• Operating system: all

Description:

We're using pip in a CI/CD pipeline to install packages from a private repository protected by username/password. Currently there are two options to pass those credentials to pip, either encode it directly in the URL or create a pip.conf file. Both options are not very attractive. The first option would entail to have those credentials hard coded in the source code, the second one would mean we'd have to generate this config file during the build process.

Most CI/CD build pipelines support some kind of "secret variables", which is a fancy word for environment variables that you can set in the CI/CD and that will be enabled in the build pipeline. This is usually the way to pass secrets.

It would be very helpful if pip would also support some mechanism to read secrets from environment variables.

See also: https://www.jfrog.com/confluence/display/RTF/PyPI+Repositories#PyPIRepositories-UsingCredentials for a realistic use case.

[wwuck]

Ok, so can anyone help on my keyring questions? The pip docs don't mention how to specify the username for the keyring credentials. Where is this username specified?

And what would happen if I enter multiple username credentials into keyring for a single pip index url? Will it pick one at random? Will it try them all until it gets a successful auth attempt?

keyring set https://private-pypi-index.example.com myusername1
keyring set https://private-pypi-index.example.com myusername2

[rehevkor5]

it is already possible to specify auth in PIP_INDEX_URL and the like

I don't see any info about that in the docs... am I missing something? https://pip.pypa.io/en/stable/search/?q=PIP_INDEX_URL&check_keywords=yes&area=default

Edit: ah, they're named based on command line options, ok https://pip.pypa.io/en/stable/topics/configuration/#environment-variables

[YevheniiPokhvalii]

Are there any updates on this? I'll explain where it could be useful. For example, Tekton pipelines in Kubernetes.

I have requirements.txt with this private repo link:

http://nexus:8081/repository/edp-python-releases/packages/some_python_package.whl

(Let's assume, it is possible to use variables PIP_USERNAME and PIP_PASSWORD). So I can attach a secret to python pod this way:

spec:
  params:
    - name: BASE_IMAGE
      type: string
      default: "python:3.8-alpine3.16"
    - name: PIP_EXTRA_INDEX_URL
      type: string
      default: "http://nexus:8081/"
    - name: PIP_TRUSTED_HOST
      type: string
      default: "nexus"
    - name: ci-secret
      type: string
      default: ci.user
  steps:
    - name: python
      image: $(params.BASE_IMAGE)
      workingDir: $(workspaces.source.path)
      env:
        - name: PIP_USERNAME
          valueFrom:
            secretKeyRef:
              name: $(params.ci-secret)
              key: username
        - name: PIP_PASSWORD
          valueFrom:
            secretKeyRef:
              name: $(params.ci-secret)
              key: password
        - name: PIP_EXTRA_INDEX_URL
          value: "$(params.PIP_EXTRA_INDEX_URL)"
        - name: PIP_TRUSTED_HOST
          value: "$(params.PIP_TRUSTED_HOST)"
      script: |
        pip install -r requirements.txt

And the above example should work without hassle.

But what I have to do without variables PIP_USERNAME and PIP_PASSWORD is to change requirements.txt to this:

http://${NEXUS_USERNAME}:${NEXUS_PASSWORD}@nexus:8081/repository/edp-python-releases/packages/some_python_package.whl

Or another workaround for multiple pipeline steps is to create pip.conf:

      env:
        - name: HOME
          value: $(workspaces.source.path)
        - name: CI_USERNAME
          valueFrom:
            secretKeyRef:
              name: $(params.ci-secret)
              key: username
        - name: CI_PASSWORD
          valueFrom:
            secretKeyRef:
              name: $(params.ci-secret)
              key: password
      script: |
        pipdir="$HOME/.pip"
        if [ ! -d "${pipdir}" ]; then
          mkdir -p "${pipdir}"
          cat <<-EOF > "${pipdir}"/pip.conf
        [global]
        trusted-host = nexus
        extra-index-url = http://${CI_USERNAME}:${CI_PASSWORD}@nexus:8081/
        EOF
        fi
        pip install -r requirements.txt

It does not look nice. PIP should have the way to pass a user and password via environment variables like Twine has.

[rehevkor5]

It's worth noting that for Docker image builds, ~/.pip/pip.conf or similar is almost certainly a better choice than using environment variables, even if values are provided via ARG. By using pip.conf, you can provide it to the Docker build in a secure way via the RUN --mount=type=secret approach documented here https://docs.docker.com/engine/reference/builder/#run---mounttypesecret assuming that you've enabled BuildKit via export DOCKER_BUILDKIT=1 or similar.

[wwuck]

https://github.com/pypi/warehouse/issues/10030 has finally been fixed, so the keyring plugin is finally uploaded to pypi.

https://pypi.org/project/keyrings.envvars/

[vyadh]

I think I have worked out a way to use pip in a secure way when building container images and environment variables, which might be helpful for others using Docker in a CI/CD context. It's a lot of code, but I think it's at least secure.

My specific requirements are:

• Avoiding the chicken-and-egg scenario of using the keyring functionality (alternative would be to use curl to get the keyring wheel from somewhere and install it first, but that would be even more complicated!)
• Not using the index-url property as it would leak credentials into our various HTTP logs
• Using Docker's buildx secure build secrets
• Not leaking credentials into any container layers
• Not leaking credentials onto physical storage even temporarily in case they are not correctly cleaned up by a particular usage

The following assumes the environment variables INDEX_USER and INDEX_PASS have been set as environment variables by the CI/CD system (or running locally when testing).

Firstly, to build the below image, the command would be:

docker build --secret id=INDEX_USER --secret id=INDEX_PASS .

The Dockerfile below is not particularly readable, but hopefully it makes some sense. Very briefly, this basically uses Docker build secrets, the netrc functionality and a tmpfs mount to install packages from our internal authenticated feed within our build systems. All without the credentials being leaking in HTTP logs or hitting physical storage.

FROM [...]

ARG INDEX_SERVER=[...]
ARG INDEX_URL=https://$INDEX_SERVER/path

RUN pip config --global set global.index-url $INDEX_URL && \
    # Make pip use standard certificate store (on at least Ubuntu, Debian & Alpine)
    pip config --global set global.cert /etc/ssl/certs/ca-certificates.crt && \
    # Avoid not found errors given no Internet access within our CI/CD system
    pip config --global set global.disable-pip-version-check true

# Copy build files into a working folder
WORKDIR /build
COPY requirements.txt ./

RUN --mount=type=secret,id=INDEX_USER \
    --mount=type=secret,id=INDEX_PASS \
    # Create a tmpfs (memory-based) mount for the netrc authentication file valid for this layer only
    --mount=type=tmpfs,target=/run/auth \
    # The mounted secrets are held in tmpfs (memory-based) files that we need to pull out
    INDEX_USER=$(cat /run/secrets/INDEX_USER) \
    INDEX_PASS=$(cat /run/secrets/INDEX_PASS) \
    # Set the authentication credentials securely (using index-url is insecure)
    NETRC_CONTENT="machine $INDEX_SERVER login $INDEX_USER password $INDEX_PASS" \
    sh -c "echo \$NETRC_CONTENT" > /run/auth/.netrc && \
    # Variable to tell pip (actually Python's netrc module) to pick up the memory-based .netrc file location
    NETRC=/run/auth/.netrc \
    # Install required packages
    pip install --no-warn-script-location --no-cache-dir -r requirements.txt && \
    pip check

# Remainder of build
[...]

I'll add that this common use of environment variables by Docker seems a good case for supporting environment variables more generally to simplify above, but at least there is a workable alternative for Linux containers.