Open johnthagen opened 2 years ago
May be a duplicate of
@thedrow You mentioned at https://github.com/python-poetry/poetry/issues/1856#issuecomment-1003727777 wanting to use Poetry dependency groups with nox-poetry
. I'm curious, is what I describe close to what you were thinking, or were you imagining something else?
Yup. Exactly.
This would be useful for my projects. I can start a PR if there's interest.
@edgarrmondragon Go ahead :)
poetry export
supports --only
poetry export --help
Description:
Exports the lock file to alternative formats.
Usage:
export [options]
...
--only=ONLY The only dependency groups to include. (multiple values allowed)
So I think this would be a relatively easy way to get a requirements.txt
file out for a specific group.
Thinking about this some more, perhaps a better API for this that would allow users to mix the current functionality with groups in a single command could use a name-only argument:
def test(session: nox_poetry.Session):
session.install(".", "pytest", poetry_groups=["lint", "type_check"])
That way a single command can used locked dependencies from any source available.
If you have dependency groups, why not use plain Nox with a small helper like this?
I'd be curious what value you see in nox-poetry given that you can now easily install locked dependency groups using just Nox and Poetry >=1.2?
@cjolowicz I'm probably missing something from your example but how do you ensure that poetry install
targets the Nox session's virtual environment?
Poetry honors the VIRTUAL_ENV
environment variable, which is set by Nox when it runs processes in the session.
(At least, it did when I wrote this code 😅 )
The first time I tried this (on Windows with latest Poetry/Nox) using the install()
function in your comment, the invoked Poetry started messing with the outer Poetry environment:
# noxfile.py
...
@session
def fmt(s: Session) -> None:
install(s, groups=["fmt"], root=False)
s.run("isort", ".")
s.run("black", ".")
It started to remove a bunch of packages in the outer environment due to --sync
:
> nox -s fmt
nox > Running session fmt
nox > Re-using existing virtual environment at .nox\fmt.
nox > poetry install --no-root --sync --only=fmt
Installing dependencies from lock file
Package operations: 0 installs, 0 updates, 58 removals
• Removing argcomplete (2.0.0)
• Removing attrs (22.2.0)
• Removing beautifulsoup4 (4.11.2)
• Removing certifi (2022.12.7)
I think this demonstrates that relying on VIRTUAL_ENV
being set/passed correctly to a bare poetry install
command can be at least brittle/unreliable on some platforms.
If you have dependency groups, why not use plain Nox with a small helper like this?
I'd be curious what value you see in nox-poetry given that you can now easily install locked dependency groups using just Nox and Poetry >=1.2?
I like nox-poetry's workflow of exporting requirements and then installing them with pip, I don't want to install with poetry directly into my nox sessions. For now I'm just copy/pasting a basic helper function to handle installs:
def install_poetry_groups(session, *groups: str) -> None:
"""Install dependencies from poetry groups."""
with tempfile.NamedTemporaryFile() as requirements:
session.run(
"poetry",
"export",
*[f"--only={group}" for group in groups],
"--format=requirements.txt",
"--without-hashes",
f"--output={requirements.name}",
external=True,
)
session.install("-r", requirements.name)
But it would be nice to have that logic handled in a common package, and it seems like nox-poetry is a good candidate to house it. My copy/paste function doesn't have nox-poetry's test coverage or nice "Warning:" message filter, so for me there's value to using it. But to your point, my current workflow is to just use plain nox with a helper function.
seems like nox-poetry is a good candidate to house it
That was my thought as well. We have dozens of Poetry/Nox projects and having a shared package where we know others are also collaborating/testing/etc. has been a great way to avoid code duplication and engage with others using a similar workflow. For that nox-poetry
has been great ❤️ .
The first time I tried this (on Windows with latest Poetry/Nox) using the
install()
function in your comment, the invoked Poetry started messing with the outer Poetry environment:
This looks like a Poetry bug. Can you provide instructions to reproduce this?
To my knowledge, poetry install
respects $VIRTUAL_ENV
. For example, try the following inside docker run -ti python bash
:
pip install pipx
pipx install poetry
export PATH=$HOME/.local/bin:$PATH
poetry new project
cd project
poetry add -G test pytest
poetry add -G docs sphinx
python -m venv /tmp/venv
. /tmp/venv/bin/activate
poetry install --sync --only docs
deactivate
poetry install
You should see Poetry install the docs group into the virtual environment. Outside the virtual environment, the last poetry install
command reports: No dependencies to install or update.
Can you provide instructions to reproduce this?
> poetry show --tree
black 23.1.0 The uncompromising code formatter.
├── click >=8.0.0
│ └── colorama *
├── mypy-extensions >=0.4.3
├── packaging >=22.0
├── pathspec >=0.9.0
├── platformdirs >=2
└── tomli >=1.1.0
isort 5.12.0 A Python utility / library to sort Python imports.
nox 2022.11.21 Flexible test automation.
├── argcomplete >=1.9.4,<3.0
├── colorlog >=2.6.1,<7.0.0
│ └── colorama *
├── packaging >=20.9
└── virtualenv >=14
├── distlib >=0.3.6,<1
├── filelock >=3.4.1,<4
└── platformdirs >=2.4,<4
pydantic 1.10.5 Data validation and settings management using python type hints
└── typing-extensions >=4.2.0
[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"
packages = [{include = "poetry_test"}]
[tool.poetry.dependencies]
python = "^3.10"
pydantic = "*"
[tool.poetry.group.nox.dependencies]
nox = "*"
[tool.poetry.group.fmt.dependencies]
black = "*"
isort = "*"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
from typing import Iterable
import nox
def install(session: nox.Session, *, groups: Iterable[str], root: bool = True) -> None:
"""Install the dependency groups using Poetry.
This function installs the given dependency groups into the session's
virtual environment. When ``root`` is true (the default), the function
also installs the root package and its default dependencies.
To avoid an editable install, the root package is not installed using
``poetry install``. Instead, the function invokes ``pip install .``
to perform a PEP 517 build.
Args:
session: The Session object.
groups: The dependency groups to install.
root: Install the root package.
"""
session.run_always(
"poetry",
"install",
"--no-root",
"--sync",
"--{}={}".format("only" if not root else "with", ",".join(groups)),
external=True,
)
if root:
session.install(".")
@nox.session
def fmt(s: nox.Session) -> None:
install(s, groups=["fmt"], root=False)
s.run("isort", ".")
s.run("black", ".")
> poetry install --sync
> poetry shell
(poetry-test-py3.10) > nox -s fmt
nox > Running session fmt
nox > Creating virtual environment (virtualenv) using python.exe in .nox\fmt
nox > poetry install --no-root --sync --only=fmt
Installing dependencies from lock file
Package operations: 0 installs, 0 updates, 8 removals
• Removing argcomplete (2.0.0)
• Removing colorlog (6.7.0)
• Removing distlib (0.3.6)
• Removing filelock (3.9.0)
• Removing nox (2022.11.21)
...
PermissionError: [WinError 5] Access is denied:
The run_always("poetry install")
is uninstalling packages from the parent Poetry venv (in fact even tries to uninstall nox
itself 😅).
So note that nox
is already running from a poetry shell
and now the poetry install
run_always()
is trying to work within the parent Poetry environment, not the venv that nox
just made.
nox-poetry
avoids any of these potential issues because it invokes the correct venv
's pip
directly (as I understand it).
@johnthagen Does this reproduce when you don't have Nox as a development dependency?
I recommend installing both Nox and Poetry using pipx, side by side. You're running Nox from a Poetry environment, and then Poetry from a Nox environment. Theoretically, it should work, but it's unneeded complexity IMO. I'd also avoid constraining the dependency tree of your application with tools that can be part of the global dev environment.
FWIW, I tried your example in a Windows runner on GHA, but it didn't reproduce in that environment. Here's an example run: https://github.com/cjolowicz/poetry-test/actions/runs/4801627813/jobs/8544094456
Theoretically, it should work, but it's unneeded complexity IMO.
In our case, we want to lock the exact version of Nox (and nox-poetry) because it is part of our CI process and thus we want it to be reproducible.
In my experience it also simplifies the setup for each dev if when they run poetry install
it brings everything they need, including nox, nox-poetry, etc. That way, the only thing they need to have installed is Poetry itself, to bootstrap everything else.
I'd also avoid constraining the dependency tree of your application with tools that can be part of the global dev environment.
I don't disagree in principle (perhaps one day Hatch or something like it can replace the need for Poetry and put everything in its own environment), but for now to make the workflow simple using Poetry (which chooses to lock everything together) and reproducible we are already constraining our environment with a host of other tools we need to lock and have reproducible versions (mypy, pytest, black, isort, ruff, etc.).
A concrete example in recent past of why it's important to pin Nox is when Nox changed its behaviour in the presence of the CI
environment variable:
Without Nox being pinned, this would have broken CI builds that depended on the previous behaviour. An example fix:
So our teams insist on locking as much of the development tools as possible and keeping both developer tools and CI in sync because things like this are bound to come up as FOSS projects evolve.
Just wanted to share a workaround i'm using to parse the toml for the groups
from typing import List
from pathlib import Path
import toml
def install_dependency_groups(session, groups: List[str] = []):
"""Manually parse the pyproject file to find group(s) of dependencies, then install."""
pyproject_path = Path("pyproject.toml")
data = toml.load(pyproject_path)
all_dependencies = []
for group in groups:
dependencies = data.get("tool", {}).get("poetry", {}).get("group", {}).get(group, {}).get("dependencies", {})
all_dependencies += list(dependencies.keys())
all_dependencies = list(set(all_dependencies))
session.install(*all_dependencies)
...
install_dependency_groups(session, groups=["lint", "tests"])
Just wanted to share a workaround i'm using to parse the toml for the groups
from typing import List from pathlib import Path import toml def install_dependency_groups(session, groups: List[str] = []): """Manually parse the pyproject file to find group(s) of dependencies, then install.""" pyproject_path = Path("pyproject.toml") data = toml.load(pyproject_path) all_dependencies = [] for group in groups: dependencies = data.get("tool", {}).get("poetry", {}).get("group", {}).get(group, {}).get("dependencies", {}) all_dependencies += list(dependencies.keys()) all_dependencies = list(set(all_dependencies)) session.install(*all_dependencies) ... install_dependency_groups(session, groups=["lint", "tests"])
Nox 2024.04.15
also has nox.project.load_toml
:
data = nox.project.load_toml(pyproject_path)
https://nox.thea.codes/en/stable/tutorial.html#loading-dependencies-from-pyproject-toml-or-scripts
So, one fewer import 😅
Poetry 1.2
will be addingadded support for dependency groups. It would be ergonomic if you could install a dependency group (as a whole) into anox
session.Consider a Poetry group:
The cooresponding
nox
session could be simplified to:This avoids the DRY problem of listing all of the lint dependencies twice, once in a Poetry group and again in a
nox
session.Related to
873
977