🔧 Add direct dependencies file: requirements.in

[ebouchut]

Here is a proposal to make it easier to create and add dependencies, and to be alerted to security issues in the packages we use.

Why?

The goal is:

• To be automatically notified of security vulnerabilities in the project's Python packages. The GitHub Dependabot (dependencies bot) can automatically search and notify us of security vulnerabilities in the packages used directly and indirectly by our project (provided they are all listed in requirements.txt).
• A separate file for direct dependencies
• To make adding a top-level (i.e. direct) dependency easier.
• To distinguish between:
  • direct dependencies in a new file requirements.in.
  • all the dependencies (including indirect dependencies aka. sub-dependencies) in requirements.txt

Today, when using a single file with all dependencies (requirements.txt), I find it difficult to spot the direct dependencies (sphinx, sphinx-copybutton, sphinx-rtd-theme, myst-parser) in an ocean of indirect dependencies. So, I suggest using 2 files.

How?

• split the current dependencies file requirements.txt to have 2 files:
  • requirements.in containing only the direct dependencies

    sphinx==5.3.0
    sphinx-copybutton==0.5.2
    sphinx-rtd-theme==1.2.0
    myst-parser==0.18.1

  • requirements.txt containing the exhaustive list of all dependencies (direct and indirect). Today it does not contain the latter.

    #
    # This file is autogenerated by pip-compile with Python 3.12
    # by the following command:
    #
    #    pip-compile
    #
    alabaster==0.7.16
    # via sphinx
    babel==2.16.0
    # via sphinx
    certifi==2024.8.30
    # via requests
    charset-normalizer==3.3.2
    # via requests
    docutils==0.18.1
    # via
    #   myst-parser
    #   sphinx
    #   sphinx-rtd-theme
    idna==3.10
    # via requests
    imagesize==1.4.1
    # via sphinx
    jinja2==3.1.4
    # via
    #   myst-parser
    #   sphinx
    markdown-it-py==2.2.0
    # via
    #   mdit-py-plugins
    #   myst-parser
    markupsafe==2.1.5
    # via jinja2
    mdit-py-plugins==0.3.5
    # via myst-parser
    mdurl==0.1.2
    # via markdown-it-py
    myst-parser==0.18.1
    # via -r requirements.in
    packaging==24.1
    # via sphinx
    pygments==2.18.0
    # via sphinx
    pyyaml==6.0.2
    # via myst-parser
    requests==2.32.3
    # via sphinx
    snowballstemmer==2.2.0
    # via sphinx
    sphinx==5.3.0
    # via
    #   -r requirements.in
    #   myst-parser
    #   sphinx-copybutton
    #   sphinx-rtd-theme
    #   sphinxcontrib-jquery
    sphinx-copybutton==0.5.2
    # via -r requirements.in
    sphinx-rtd-theme==1.2.0
    # via -r requirements.in
    sphinxcontrib-applehelp==2.0.0
    # via sphinx
    sphinxcontrib-devhelp==2.0.0
    # via sphinx
    sphinxcontrib-htmlhelp==2.1.0
    # via sphinx
    sphinxcontrib-jquery==4.1
    # via sphinx-rtd-theme
    sphinxcontrib-jsmath==1.0.1
    # via sphinx
    sphinxcontrib-qthelp==2.0.0
    # via sphinx
    sphinxcontrib-serializinghtml==2.0.0
    # via sphinx
    typing-extensions==4.12.2
    # via myst-parser
    urllib3==2.2.3
    # via requests

    pip-compile lists below each dependencies its sub-dependencies.

• Use pip-compile (from the pip-tools package) to generate requirements.txt from requirements.in.

pip-compile reads a source file requirements.in to generate a requirements.txt with all the dependencies. It resolves and pins the dependencies to the ad-hoc version, ensuring reproducibility. It only focuses on dependencies declared in the source file.

pip freeze uses all packages installed in the virtual environment.

Install

We need to install pip-tools first to use pip-compile:

python -m pip install pip-tools

Usage

Using pip-compile:

pip-compile
python -m pip install -r requirements.txt

pip-compile only looks at the source file (requirements.in), whereas pip freeze looks at what is currently installed in the virtual environment.

Previously, with pip:

# Assuming the `venv` virtual environment has already been created and activated
# python -m venv venv
# source venv/bin/activate

# Remove all installed dependencies/packages
python -m pip freeze --exclude-editable | xargs python -m pip  uninstall -y

# Install the project's packages listed in `requirements.in`
python -m pip install -r requirements.in

# List the pinned project packages (name + version) in `requirements.txt`
python -m pip freeze > requirements.txt

# Install dependencies
python -m pip install -r requirements.txt

[ebouchut]

EDIT: moved to PR description.

[tmhastings]

@MikePlante1 I'd rather have your review approval for this one.

[ebouchut]

@marionbarker I merged upstream/dev which dismissed your approval.

[ebouchut]

@MikePlante1 ✅ I merged the dev branch and added instructions for using pip-compile to create requirements.txt.