empty `requirements.txt` when importing from `pyproject.toml`

[BBBmau]

When attempting to import from an existing pyproject.toml file I get an empty generated requirements.txt file, for context I am currently attempting to import from using poetry to using uv which could be what's causing it. Is their currently a workaround for this?

uv pip compile pyproject.toml -o requirements.txt # Read a pyproject.toml file.

uv version: uv 0.2.2 (e52ae0e2b 2024-05-22)

[konstin]

We don't support poetry's format currently, only the standardized (PEP 621) [project] table.

[charliermarsh]

This is still supposed to work though. If it’s a Poetry file, we use PEP 517, I think? Though maybe it depends on certain fields being present in the file.

Can you share a pyproject.toml file that reproduces this behavior?

[charliermarsh]

It works as expected for me with:

[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Charlie Marsh <charlie.r.marsh@gmail.com>"]
packages = [{ include = "poetry_test" }]

[tool.poetry.dependencies]
python = "^3.10.0"
anyio = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[BBBmau]

It works as expected for me with:

[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Charlie Marsh <charlie.r.marsh@gmail.com>"]
packages = [{ include = "poetry_test" }]

[tool.poetry.dependencies]
python = "^3.10.0"
anyio = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I have these included with this along with it:

[tool.poetry.group.dev.dependencies]

[charliermarsh]

Do you mind sharing the whole file? (We don't include dev dependencies though -- that's a proprietary Poetry feature, so they won't be included if you lock with uv or pip-compile or any other tool IIRC.)

[BBBmau]

sure thing @charliermarsh

[project]
name = "openpilot"
requires-python = ">= 3.11"
readme = "README.md"
license = {text = "MIT License"}

[project.urls]
Homepage = "https://comma.ai"

[tool.pytest.ini_options]
minversion = "6.0"
addopts = "--ignore=openpilot/ --ignore=cereal/ --ignore=opendbc/ --ignore=panda/ --ignore=rednose_repo/ --ignore=tinygrad_repo/ --ignore=teleoprtc_repo/ -Werror --strict-config --strict-markers --durations=10 -n auto --dist=loadgroup"
cpp_files = "test_*"
cpp_harness = "selfdrive/test/cpp_harness.py"
python_files = "test_*.py"
#timeout = "30"  # you get this long by default
markers = [
  "slow: tests that take awhile to run and can be skipped with -m 'not slow'",
  "tici: tests that are only meant to run on the C3/C3X",
]
testpaths = [
  "common",
  "selfdrive/boardd",
  "selfdrive/car",
  "selfdrive/controls",
  "selfdrive/locationd",
  "selfdrive/monitoring",
  "selfdrive/navd/tests",
  "selfdrive/test/longitudinal_maneuvers",
  "selfdrive/test/process_replay/test_fuzzy.py",
  "selfdrive/updated",
  "system/thermald",
  "system/athena",
  "system/camerad",
  "system/hardware/tici",
  "system/loggerd",
  "system/proclogd",
  "system/tests",
  "system/ubloxd",
  "system/webrtc",
  "tools/lib/tests",
  "tools/replay",
  "tools/cabana"
]

[tool.mypy]
python_version = "3.11"
plugins = [
  "numpy.typing.mypy_plugin",
]
exclude = [
  "body/",
  "cereal/",
  "opendbc/",
  "panda/",
  "rednose/",
  "rednose_repo/",
  "tinygrad/",
  "tinygrad_repo/",
  "teleoprtc/",
  "teleoprtc_repo/",
  "third_party/",
]

# third-party packages
ignore_missing_imports=true

# helpful warnings
warn_redundant_casts=true
warn_unreachable=true
warn_unused_ignores=true

# restrict dynamic typing
warn_return_any=true

# allow implicit optionals for default args
implicit_optional = true

[tool.poetry]
name = "openpilot"
version = "0.1.0"
description = "an open source driver assistance system"
authors = ["Vehicle Researcher <user@comma.ai>"]
license = "MIT"
readme = "README.md"
repository = "https://github.com/commaai/openpilot"
documentation = "https://docs.comma.ai"

[tool.poetry.dependencies]
python = "~3.11"
aiohttp = "*"
aiortc = "*"
cffi = "*"
crcmod = "*"
Cython = "*"
json-rpc = "*"
libusb1 = "*"
numpy = "*"
onnx = ">=1.14.0"
onnxruntime = { version = ">=1.16.3", platform = "linux", markers = "platform_machine == 'aarch64'" }
onnxruntime-gpu = { version = ">=1.16.3", platform = "linux", markers = "platform_machine == 'x86_64'" }
psutil = "*"
pyaudio = "*"
pycapnp = "*"
pycryptodome = "*"
PyJWT = "*"
pyserial = "*"
pyzmq = "*"
rerun-sdk = "*"
requests = "*"
scons = "*"
sentry-sdk = "*"
smbus2 = "*"
sounddevice = "*"
spidev = { version = "*", platform = "linux" }
sympy = "*"
websocket_client = "*"

# acados deps
casadi = "*"
future-fstrings = "*"

# these should be removed
markdown-it-py = "*"
timezonefinder = "*"
setproctitle = "*"

[tool.poetry.group.dev.dependencies]
av = "*"
azure-identity = "*"
azure-storage-blob = "*"
breathe = "*"
control = "*"
coverage = "*"
dictdiffer = "*"
flaky = "*"
hypothesis = "~6.47"
inputs = "*"
Jinja2 = "*"
lru-dict = "*"
matplotlib = "*"
metadrive-simulator = { version = "0.4.2.3", markers = "platform_machine != 'aarch64'" } # no linux/aarch64 wheels for certain dependencies
mpld3 = "*"
mypy = "*"
myst-parser = "*"
natsort = "*"
opencv-python-headless = "*"
parameterized = "^0.8"
pprofile = "*"
polyline = "*"
pre-commit = "*"
pyautogui = "*"
pyopencl = "==2023.1.4"  # 2024.1 is broken on arm64
pygame = "*"
pywinctl = "*"
pyprof2calltree = "*"
pytest = "*"
pytest-cov = "*"
pytest-cpp = "*"
pytest-subtests = "*"
pytest-xdist = "*"
pytest-timeout = "*"
pytest-randomly = "*"
pytest-asyncio = "*"
pytest-mock = "*"
ruff = "*"
sphinx = "*"
sphinx-rtd-theme = "*"
sphinx-sitemap = "*"
tabulate = "*"
types-requests = "*"
types-tabulate = "*"
tqdm = "*"

# this is only pinned since 5.15.11 is broken
pyqt5 = { version = "==5.15.2", markers = "platform_machine == 'x86_64'" } # no aarch64 wheels for macOS/linux

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# https://beta.ruff.rs/docs/configuration/#using-pyprojecttoml
[tool.ruff]
indent-width = 2
lint.select = [
  "E", "F", "W", "PIE", "C4", "ISC", "A", "B",
  "NPY", # numpy
  "UP",  # pyupgrade
  "TRY302", "TRY400", "TRY401", # try/excepts
  "RUF008", "RUF100",
  "TID251"
]
lint.ignore = [
  "E741",
  "E402",
  "C408",
  "ISC003",
  "B027",
  "B024",
  "NPY002",  # new numpy random syntax is worse
  "UP038",   # (x, y) -> x|y for isinstance
]
line-length = 160
target-version="py311"
exclude = [
  "body",
  "cereal",
  "panda",
  "opendbc",
  "rednose_repo",
  "tinygrad_repo",
  "teleoprtc",
  "teleoprtc_repo",
  "third_party",
]
lint.flake8-implicit-str-concat.allow-multiline=false
[tool.ruff.lint.flake8-tidy-imports.banned-api]
"selfdrive".msg = "Use openpilot.selfdrive"
"common".msg = "Use openpilot.common"
"system".msg = "Use openpilot.system"
"third_party".msg = "Use openpilot.third_party"
"tools".msg = "Use openpilot.tools"
"pytest.main".msg = "pytest.main requires special handling that is easy to mess up!"
"unittest".msg = "Use pytest"

[tool.coverage.run]
concurrency = ["multiprocessing", "thread"]
[tool.ruff.format]
quote-style = "preserve"

[tool.uv.pip]
index-url = "https://test.pypi.org/simple"

[charliermarsh]

@BBBmau - so technically, per the spec... if you have a [project] section, and dependencies is omitted, it means dependencies = [].

You can fix this by adding dynamic = ["dependencies"], like:

[project]
name = "openpilot"
requires-python = ">= 3.11"
readme = "README.md"
license = {text = "MIT License"}
dynamic = ["dependencies"]

[mihirsamdarshi]

Charlie, If I understand your last comment correctly, when [project] and [tool.poetry] sections are included, [tool.poetry.dependencies] is ignored, unless dynamic = ["dependencies"] is included under [project]?

Therefore a correct prproject.toml with both [project] and [tool.poetry] would look like this?

[project]
name = "poetry-test"
requires-python = ">= 3.11"
readme = "README.md"
license = {text = "MIT License"}
dynamic = ["dependencies"]

[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Urmom"]
packages = [{ include = "poetry_test" }]

[tool.poetry.dependencies]
python = "^3.10.0"
anyio = "*"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I was running into the same issue, that uv pip compile pyproject.toml was giving me a blank requirements.txt file. I was able to trace it back to this behavior.

Additionally, I know that isn't probably something you thought would happen, but if you don't have a [tool.poetry] section (your [build-system] is not poetry), but you do have [tool.poetry.dependencies] section (and also have dynamic = ["dependencies"]), the result of uv pip compile pyproject.toml is also empty. I was hoping to incrementally migrate from poetry to uv/rye by removing the packaging system before dependency management. For some reason, the rye add command (listing my dep requirements in the TOML's project.dependencies key) doesn't work, but it does when using uv pip compile pyproject.toml and defining the requirements in tool.poetry.dependencies.

[charliermarsh]

@mihirsamdarshi - Yeah this is technically correct. If you have a non-empty project section, and nothing listed as dynamic, then non-Poetry tools are allowed to assume that the metadata in project is "exactly correct". And if dependencies is omitted, it's equivalent to setting dependencies = []. So above, non-Poetry tools will think that the package has no dependencies.

If you add dynamic = ["dependencies"] as you did above, it should work with uv and any other non-Poetry tools (like pip-compile).

[charliermarsh]

(Happy to answer more questions, let me know if unclear.)

[mhemeryck]

Is there a way in which we can also include poetry's dev dependencies in the lock file? Essentially everything under tool.poetry.group.dev.dependencies?

Looking into a way in which I could steadily migrate our poetry-based setup into uv; would be easy if the 2 could just co-exist for some time.

[charliermarsh]

Unfortunately there isn't really any way for us to parse and access Poetry's development dependencies without supporting Poetry's entire schema. Poetry use their own schema for defining dependencies, so from uv's perspective, tool.poetry.group.dev.dependencies is effectively opaque.

[mhemeryck]

Unfortunately there isn't really any way for us to parse and access Poetry's development dependencies without supporting Poetry's entire schema. Poetry use their own schema for defining dependencies, so from uv's perspective, tool.poetry.group.dev.dependencies is effectively opaque.

OK, makes sense. In the projects we've been developing, poetry has become so much part of the standard way of managing python projects that the pyproject.toml file had become almost exclusively associated with poetry. It's only now that I have been exploring uv that I see other kinds of workflows, like pip-tools (pip-compile and pip-sync).

The quickest work-around for us will probably be to just not split the dependencies into 2 separate groups, because I see it actually works quite OK in that case.

Poetry does not support editable installs though, so that part is then again conflicting; before, we used to run poetry run <tool> and it the current local folder would be part of the python part, but that's not how the virtualenv seems to behave ...

... but I'm rambling on, dependency management in python always has been a bit of an issue so it's great to see all the effort you are putting in this, very curious to see how it will evolve! The thing poetry does right in my view is that is a single tool to manage and resolve dependencies and virtualenvs (the resolving part just tends to be really slow for larger projects).