Add support for Rye Python Package Manager

[secustor]

Discussed in https://github.com/renovatebot/renovate/discussions/25225

^{Originally posted by **dixneuf19** October 16, 2023} ### Tell us more. [Rye](https://github.com/mitsuhiko/rye) is a somewhat _trendy_ solution for package management, still experimental and from [the author's admission probably a bad idea ](https://github.com/mitsuhiko/rye/discussions/6) since it adds another tool among the mess of Python package managers. However I (and the community it seems) find its very useful and I start to try using it for [a personal project](https://github.com/dixneuf19/whats-on-fip). However, I quickly realized that Renovate (a must have tool for me btw) does not support `rye` totally. Indeed, `rye` has the following methods to save dependencies: - for the main dependencies, it follows the [PEP 440](https://www.python.org/dev/peps/pep-0440/) specifications, saving versions in `pyproject.toml`. For that matter, Renovate got it covered ✅ ```toml dependencies = [ "fastapi>=0.103.2", "uvicorn>=0.23.2", ``` - For dev dependencies, that is already a bit different, it is saved in a `rye` setting in the same `pyproject.toml` ```toml [tool.rye] managed = true dev-dependencies = [ "pytest>=7.4.2", "black>=23.9.1", ... ] ``` - And lastly for _lockfiles_, it saves the exact pinned version in `requirements.lock` and `requirements-dev.lock`. Theses files looks almost like standard `requirements.txt` obtained by `pip freeze`, with the addition of a [`-e file:.` used for python packaging](https://github.com/mitsuhiko/rye/discussions/301#discussioncomment-6103811). The main reason for this is that there is no official consensus in Python community for how to manage a lock file since [PEP 665](https://peps.python.org/pep-0665/) has been rejected. ```txt # generated by rye # use `rye lock` or `rye sync` to update this lockfile # # last locked with the following flags: # pre: false # features: [] # all-features: false -e file:. annotated-types==0.6.0 anyio==3.7.1 ``` - To keep its lockfile up-to-date, you simply need to run `rye lock` and it update the `*.lock` files. Since this project is still experimental I am not sure that Renovate wants to support it. However, since it already supports various Python packaging solutions (Poetry, Pipenv, pdm) with their own custom lock files, the work may be relatively easy to do. Is this something the project would be interested with ? I would be very interested to contribute to Renovate about this issue, since I love the tool and want to better know how it works internally.

[secustor]

New package manager questionnaire

Did you read our documentation on adding a package manager?

• [x] I've read the [adding a package manager](https://github.com/renovatebot/renovate/discussions/adding-a-package-manager.md) documentation.

Basics

Name of package manager

rye, but might be supported by existing pep621 manager, see discussion

What language does this support?

Python

How popular is this package manager?

7,3k stars in a few month, it had a huge community response.

Does this language have other (competing?) package managers?

• [x] Yes (give names)
  • Poetry
  • pdm
  • pipenv
• [ ] No

---

Package File Detection

What type of package files and names does it use?

• pyproject.toml for dependencies
• requirements.lock and requirements-dev.lock

What [fileMatch](typora://app/usage/configuration-options.md#filematch) pattern(s) should be used?

Something like ['pyproject.toml', 'requirements(-dev)?.lock']

Is it likely that many users would need to extend this pattern for custom file names?

• [ ] Yes

• [x] No

  The tool is pretty opiniated, for now the lock file pathes are hardcoded

Is the fileMatch pattern likely to get many "false hits" for files that have nothing to do with package management?

I don't think so

---

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

No

Is there a reason why package files need to be parsed together (in serial) instead of independently?

No

What format/syntax is the package file in?

• [ ] JSON

• [x] TOML

• [ ] YAML

• [x] Custom (explain below)

  lock file are custom text files, close to requirements.txt generated by pip

How do you suggest parsing the file?

• [x] Off the shelf parser
  • existing pep621 manager for pyproject.toml
  • existingrequirements.txt manager for renovate
• [ ] Using regex
• [ ] Custom-parsed line by line
• [ ] Other

Does the package file structure distinguish between different "types" of dependencies? e.g. production dependencies, development dependencies, etc?

• [x] Yes, production and development dependencies
• [ ] No, all dependencies are treated the same

List all the sources/syntaxes of dependencies that can be extracted

It follows [PEP 440](https://peps.python.org/pep-0440/)

• For prod dependencies, in pyproject.toml. Exemple

  dependencies = [
    "fastapi>=0.103.2",
    "uvicorn>=0.23.2",
    "loguru>=0.7.2",
    "requests>=2.31.0",
    "python-engineio>=4.8.0",
    "pydantic>=2.4.2",
  ]

• For dev dependencies in pyproject.toml

  [tool.rye]
  managed = true
  dev-dependencies = [
    "pytest>=7.4.2",
    "black>=23.9.1",
    "ruff>=0.0.292",
    "isort>=5.12.0",
    "pytest-asyncio>=0.21.1",
    "pytest-cov>=4.1.0",
    "pytest-mock>=3.11.1",
    "pyright>=1.1.331",
    "httpx>=0.25.0",
  ]
  

• For lock-files, in requirements.lock and requirements-dev.lock

  # generated by rye
  # use `rye lock` or `rye sync` to update this lockfile
  #
  # last locked with the following flags:
  #   pre: false
  #   features: []
  #   all-features: false
  
  -e file:.
  annotated-types==0.6.0
  anyio==3.7.1
  certifi==2023.7.22
  

Describe which types of dependencies above are supported and which will be implemented in future

---

Versioning

What versioning scheme does the package file(s) use?

[pep440](https://peps.python.org/pep-0440/)

Does this versioning scheme support range constraints, e.g. ^1.0.0 or 1.x?

• [x] Supports range constraints (e.g ^1.0.0 or 1.x)
• [ ] No

---

Lookup

Is a new datasource required? Provide details

• [ ] Yes, provide details.

• [x] No.

  Using existing pip solution

Will users need the capability to specify a custom host/registry to look up? Can it be found within the package files, or within other files inside the repository, or would it require Renovate configuration?

Yes, see [documentation](https://rye-up.com/guide/sources/)

It could be located directly in the repo in the pyproject.toml as such:

[[tool.rye.sources]]
name = "company-internal"
url = "https://company.internal/simple/"

[[tool.rye.sources]]
name = "company-internal"
url = "https://company.internal/simple/"
type = "index"  # this is implied

Or in a global config file ~/.rye/config.toml , which can simply be configured using exiting pep621 datasource

There is also a support for environment variables in this global file, but it should be configured trough Renovate configuration I think

[[sources]]
name = "company-internal"
url = "https://${INDEX_USERNAME}:${INDEX_PASSWORD}@company.internal/simple/"

Do the package files have any "constraints" on the parent language (e.g. supports only v3.x of Python) or platform (Linux, Windows, etc.) that should be used in the lookup procedure?

Don't think so, it is builded in rust and seems to support major platforms.

Will users need the ability to configure language or other constraints using Renovate config?

Eventually the version of the tool, since it is pretty much in beta

---

Artifacts

Are lock files or checksum files used? Are they mandatory?

Yes they are used

Not mandatory but strongly encourage, rye will recreate one if necessary.

If so, what tool and exact commands should be used if updating one or more package versions in a dependency file?

rye lock --update <package-name> You can only update one at a time with this commande

You can update all with rye lock --update-all

If applicable, describe how the tool maintains a cache and if it can be controlled via CLI or environment variables? Do you recommend the cache be kept or disabled/ignored?

While the [maintainer has opinions](https://rye-up.com/philosophy/#metadata-caches) about it, there is no cache for now [it seems](https://github.com/mitsuhiko/rye/issues/456).

If applicable, what command should be used to generate a lock file from scratch if you already have a package file? This will be used for "lock file maintenance"

rye lock

Other

Is there anything else to know about this package manager?

N/A

[secustor]

The implementation for should be pretty similar to PDM and needs code changes here at Renovate and ContainerBase:

• https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/pep621/processors/pdm.ts
• https://github.com/containerbase/base/pull/973

[HonkingGoose]

Pinging @dixneuf19 so I can assign them to the issue.

[viceice]

addind rye to containerbase should be easy, see pdm sample

https://github.com/containerbase/base/blob/e2921ce8f1aadde0f212687da7cb3029b1dbd53d/src/usr/local/containerbase/tools/v2/pdm.sh#L1-L10

I'll soon add a install-pip command, like install-npm and install-gem, so we don't need to add new pip based tools to containerbase when required.