Open LoganWalls opened 2 years ago
jax
depends on etils[epath]
. In etils
pyproject.toml
you can see that the extra epath
depends on etils[epy]
. This is where recursion happens. You've shown this in your lock file.
~Do you get an infinite recursion error?~ Now I see the title of the issue...
jax
depends onetils[epath]
. Inetils
pyproject.toml
you can see that the extraepath
depends onetils[epy]
. This is where recursion happens. You've shown this in your lock file.~Do you get an infinite recursion error?~ Now I see the title of the issue...
Yes. If I manually delete that line in poetry.lock
it fixes the problem. But I'd like to help fix this so that others don't encounter the issue. (Also, it feels pretty hacky to manually edit the lockfile).
Can / should this be fixed in poetry2nix, or does this issue need to be reported somewhere else? (I'm happy to help implement the fix in poetry2nix if needed).
Had the same problem, is there potentially a way to change the pyproject.toml to fix it similar to #273?
This is still an issue. Not sure what to do about it.
Typically with Python packaging you always install in a single environment. Tricky thing is that builds with extras can affect the resulting package. That's OK when you're in the same environment. ~When you want to build each package separately like we do, it means you cannot just keep extras out of a build, because it might result in a different build. Hence, flattening might resolve the recursion issue, but it might result in incorrect builds.~
Thinking about this further, these are just runtime dependencies. When building a wheel these dependencies are not yet in your environment (technically they are in our nix-build
but that's just how we do the build), so actually what I said was wrong, we should always be able to flatten.
Describe the issue
The latest version of
jax
depends onetils
, which has a bit of an odd packaging structure. Poetry seems to assign etils as its own dependency, which is the source of the problem:Manually deleting the offending line (highlighted above) solves the issue. I am opening the issue to ask whether this can/should be fixed in
poetry2nix
(in which case I'm happy to help implement the fix), or whether this should be reported elsewhere.Note: this problem only seems to happen when
jax
is specified as a dependency. If I make apyproject.toml
with onlyetils
, this problem does not occur.Thanks!
Additional context
flake.nix
pyproject.toml
[tool.poetry.dependencies] python = "^3.9" jax = "^0.3.14"
[tool.poetry.dev-dependencies]
[build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api"
[[package]] name = "absl-py" version = "1.1.0" description = "Abseil Python Common Libraries, see https://github.com/abseil/abseil-py." category = "main" optional = false python-versions = ">=3.6"
[[package]] name = "etils" version = "0.6.0" description = "Collection of common python utils" category = "main" optional = false python-versions = ">=3.7"
[package.dependencies] etils = {version = "", extras = ["epy"], optional = true, markers = "extra == \"epath\""} importlib_resources = {version = "", optional = true, markers = "extra == \"epath\""} typing_extensions = {version = "", optional = true, markers = "extra == \"epy\""} zipp = {version = "", optional = true, markers = "extra == \"epath\""}
[package.extras] all = ["etils", "etils", "etils", "etils", "etils", "etils", "etils", "etils", "etils", "etils", "etils"] array-types = ["numpy"] dev = ["pytest", "pytest-subtests", "pytest-xdist", "pylint (>=2.6.0)", "yapf", "chex"] ecolab = ["jupyter", "numpy", "mediapy", "etils"] edc = ["etils"] enp = ["numpy", "etils"] epath = ["importlib-resources", "zipp", "etils"] epath-no-tf = ["etils"] epy = ["typing-extensions"] etqdm = ["absl-py", "tqdm", "etils"] etree = ["etils", "etils", "etils", "etils"] etree-dm = ["dm-tree", "etils"] etree-jax = ["jax", "etils"] etree-tf = ["tf-nightly", "etils"]
[[package]] name = "importlib-resources" version = "5.8.0" description = "Read resources from Python packages" category = "main" optional = false python-versions = ">=3.7"
[package.dependencies] zipp = {version = ">=3.1.0", markers = "python_version < \"3.10\""}
[package.extras] docs = ["sphinx", "jaraco.packaging (>=9)", "rst.linker (>=1.9)"] testing = ["pytest (>=6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "pytest-black (>=0.3.7)", "pytest-mypy (>=0.9.1)"]
[[package]] name = "jax" version = "0.3.14" description = "Differentiate, compile, and transform Numpy code." category = "main" optional = false python-versions = ">=3.7"
[package.dependencies] absl-py = "" etils = {version = "", extras = ["epath"]} numpy = ">=1.19" opt_einsum = "" scipy = ">=1.5" typing_extensions = ""
[package.extras] ci = ["jaxlib (==0.3.10)"] cpu = ["jaxlib (==0.3.14)"] cuda = ["jaxlib (==0.3.14+cuda11.cudnn82)"] cuda11_cudnn805 = ["jaxlib (==0.3.14+cuda11.cudnn805)"] cuda11_cudnn82 = ["jaxlib (==0.3.14+cuda11.cudnn82)"] minimum-jaxlib = ["jaxlib (==0.3.7)"] tpu = ["jaxlib (==0.3.14)", "libtpu-nightly (==0.1.dev20220627)", "requests"]
[[package]] name = "numpy" version = "1.23.0" description = "NumPy is the fundamental package for array computing with Python." category = "main" optional = false python-versions = ">=3.8"
[[package]] name = "opt-einsum" version = "3.3.0" description = "Optimizing numpys einsum function" category = "main" optional = false python-versions = ">=3.5"
[package.dependencies] numpy = ">=1.7"
[package.extras] docs = ["sphinx (==1.2.3)", "sphinxcontrib-napoleon", "sphinx-rtd-theme", "numpydoc"] tests = ["pytest", "pytest-cov", "pytest-pep8"]
[[package]] name = "scipy" version = "1.6.1" description = "SciPy: Scientific Library for Python" category = "main" optional = false python-versions = ">=3.7"
[package.dependencies] numpy = ">=1.16.5"
[[package]] name = "typing-extensions" version = "4.2.0" description = "Backported and Experimental Type Hints for Python 3.7+" category = "main" optional = false python-versions = ">=3.7"
[[package]] name = "zipp" version = "3.8.0" description = "Backport of pathlib-compatible object wrapper for zip files" category = "main" optional = false python-versions = ">=3.7"
[package.extras] docs = ["sphinx", "jaraco.packaging (>=9)", "rst.linker (>=1.9)"] testing = ["pytest (>=6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "jaraco.itertools", "func-timeout", "pytest-black (>=0.3.7)", "pytest-mypy (>=0.9.1)"]
[metadata] lock-version = "1.1" python-versions = "^3.9" content-hash = "6a355cfe7b5d62a36d1b839e60cababf6558ff20abf850859968435597c11c3d"
[metadata.files] absl-py = [ {file = "absl-py-1.1.0.tar.gz", hash = "sha256:3aa39f898329c2156ff525dfa69ce709e42d77aab18bf4917719d6f260aa6a08"}, {file = "absl_py-1.1.0-py3-none-any.whl", hash = "sha256:db97287655e30336938f8058d2c81ed2be6af1d9b6ebbcd8df1080a6c7fcd24e"}, ] etils = [ {file = "etils-0.6.0-py3-none-any.whl", hash = "sha256:e47b14402f2fc883c70a394e86b3236e4929cfd90187fba58d2f4d3eb2e07c05"}, {file = "etils-0.6.0.tar.gz", hash = "sha256:6677241051835d0db11c4947bcad938f57b51ea102290624f892f1e4e51b70e5"}, ] importlib-resources = [ {file = "importlib_resources-5.8.0-py3-none-any.whl", hash = "sha256:7952325ffd516c05a8ad0858c74dff2c3343f136fe66a6002b2623dd1d43f223"}, {file = "importlib_resources-5.8.0.tar.gz", hash = "sha256:568c9f16cb204f9decc8d6d24a572eeea27dacbb4cee9e6b03a8025736769751"}, ] jax = [ {file = "jax-0.3.14.tar.gz", hash = "sha256:d5f7695ce43b7345b4a095f4733dd46336945057dbfb63e0a28eafa74c41fe62"}, ] numpy = [ {file = "numpy-1.23.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:58bfd40eb478f54ff7a5710dd61c8097e169bc36cc68333d00a9bcd8def53b38"}, {file = "numpy-1.23.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:196cd074c3f97c4121601790955f915187736f9cf458d3ee1f1b46aff2b1ade0"}, {file = "numpy-1.23.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f1d88ef79e0a7fa631bb2c3dda1ea46b32b1fe614e10fedd611d3d5398447f2f"}, {file = "numpy-1.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d54b3b828d618a19779a84c3ad952e96e2c2311b16384e973e671aa5be1f6187"}, {file = "numpy-1.23.0-cp310-cp310-win32.whl", hash = "sha256:2b2da66582f3a69c8ce25ed7921dcd8010d05e59ac8d89d126a299be60421171"}, {file = "numpy-1.23.0-cp310-cp310-win_amd64.whl", hash = "sha256:97a76604d9b0e79f59baeca16593c711fddb44936e40310f78bfef79ee9a835f"}, {file = "numpy-1.23.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:d8cc87bed09de55477dba9da370c1679bd534df9baa171dd01accbb09687dac3"}, {file = "numpy-1.23.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:f0f18804df7370571fb65db9b98bf1378172bd4e962482b857e612d1fec0f53e"}, {file = "numpy-1.23.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ac86f407873b952679f5f9e6c0612687e51547af0e14ddea1eedfcb22466babd"}, {file = "numpy-1.23.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ae8adff4172692ce56233db04b7ce5792186f179c415c37d539c25de7298d25d"}, {file = "numpy-1.23.0-cp38-cp38-win32.whl", hash = "sha256:fe8b9683eb26d2c4d5db32cd29b38fdcf8381324ab48313b5b69088e0e355379"}, {file = "numpy-1.23.0-cp38-cp38-win_amd64.whl", hash = "sha256:5043bcd71fcc458dfb8a0fc5509bbc979da0131b9d08e3d5f50fb0bbb36f169a"}, {file = "numpy-1.23.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:1c29b44905af288b3919803aceb6ec7fec77406d8b08aaa2e8b9e63d0fe2f160"}, {file = "numpy-1.23.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:98e8e0d8d69ff4d3fa63e6c61e8cfe2d03c29b16b58dbef1f9baa175bbed7860"}, {file = "numpy-1.23.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:79a506cacf2be3a74ead5467aee97b81fca00c9c4c8b3ba16dbab488cd99ba10"}, {file = "numpy-1.23.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:092f5e6025813e64ad6d1b52b519165d08c730d099c114a9247c9bb635a2a450"}, {file = "numpy-1.23.0-cp39-cp39-win32.whl", hash = "sha256:d6ca8dabe696c2785d0c8c9b0d8a9b6e5fdbe4f922bde70d57fa1a2848134f95"}, {file = "numpy-1.23.0-cp39-cp39-win_amd64.whl", hash = "sha256:fc431493df245f3c627c0c05c2bd134535e7929dbe2e602b80e42bf52ff760bc"}, {file = "numpy-1.23.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:f9c3fc2adf67762c9fe1849c859942d23f8d3e0bee7b5ed3d4a9c3eeb50a2f07"}, {file = "numpy-1.23.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d0d2094e8f4d760500394d77b383a1b06d3663e8892cdf5df3c592f55f3bff66"}, {file = "numpy-1.23.0-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:94b170b4fa0168cd6be4becf37cb5b127bd12a795123984385b8cd4aca9857e5"}, {file = "numpy-1.23.0.tar.gz", hash = "sha256:bd3fa4fe2e38533d5336e1272fc4e765cabbbde144309ccee8675509d5cd7b05"}, ] opt-einsum = [ {file = "opt_einsum-3.3.0-py3-none-any.whl", hash = "sha256:2455e59e3947d3c275477df7f5205b30635e266fe6dc300e3d9f9646bfcea147"}, {file = "opt_einsum-3.3.0.tar.gz", hash = "sha256:59f6475f77bbc37dcf7cd748519c0ec60722e91e63ca114e68821c0c54a46549"}, ] scipy = [ {file = "scipy-1.6.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:a15a1f3fc0abff33e792d6049161b7795909b40b97c6cc2934ed54384017ab76"}, {file = "scipy-1.6.1-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:e79570979ccdc3d165456dd62041d9556fb9733b86b4b6d818af7a0afc15f092"}, {file = "scipy-1.6.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:a423533c55fec61456dedee7b6ee7dce0bb6bfa395424ea374d25afa262be261"}, {file = "scipy-1.6.1-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:33d6b7df40d197bdd3049d64e8e680227151673465e5d85723b3b8f6b15a6ced"}, {file = "scipy-1.6.1-cp37-cp37m-win32.whl", hash = "sha256:6725e3fbb47da428794f243864f2297462e9ee448297c93ed1dcbc44335feb78"}, {file = "scipy-1.6.1-cp37-cp37m-win_amd64.whl", hash = "sha256:5fa9c6530b1661f1370bcd332a1e62ca7881785cc0f80c0d559b636567fab63c"}, {file = "scipy-1.6.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:bd50daf727f7c195e26f27467c85ce653d41df4358a25b32434a50d8870fc519"}, {file = "scipy-1.6.1-cp38-cp38-manylinux1_i686.whl", hash = "sha256:f46dd15335e8a320b0fb4685f58b7471702234cba8bb3442b69a3e1dc329c345"}, {file = "scipy-1.6.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:0e5b0ccf63155d90da576edd2768b66fb276446c371b73841e3503be1d63fb5d"}, {file = "scipy-1.6.1-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:2481efbb3740977e3c831edfd0bd9867be26387cacf24eb5e366a6a374d3d00d"}, {file = "scipy-1.6.1-cp38-cp38-win32.whl", hash = "sha256:68cb4c424112cd4be886b4d979c5497fba190714085f46b8ae67a5e4416c32b4"}, {file = "scipy-1.6.1-cp38-cp38-win_amd64.whl", hash = "sha256:5f331eeed0297232d2e6eea51b54e8278ed8bb10b099f69c44e2558c090d06bf"}, {file = "scipy-1.6.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:0c8a51d33556bf70367452d4d601d1742c0e806cd0194785914daf19775f0e67"}, {file = "scipy-1.6.1-cp39-cp39-manylinux1_i686.whl", hash = "sha256:83bf7c16245c15bc58ee76c5418e46ea1811edcc2e2b03041b804e46084ab627"}, {file = "scipy-1.6.1-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:794e768cc5f779736593046c9714e0f3a5940bc6dcc1dba885ad64cbfb28e9f0"}, {file = "scipy-1.6.1-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:5da5471aed911fe7e52b86bf9ea32fb55ae93e2f0fac66c32e58897cfb02fa07"}, {file = "scipy-1.6.1-cp39-cp39-win32.whl", hash = "sha256:8e403a337749ed40af60e537cc4d4c03febddcc56cd26e774c9b1b600a70d3e4"}, {file = "scipy-1.6.1-cp39-cp39-win_amd64.whl", hash = "sha256:a5193a098ae9f29af283dcf0041f762601faf2e595c0db1da929875b7570353f"}, {file = "scipy-1.6.1.tar.gz", hash = "sha256:c4fceb864890b6168e79b0e714c585dbe2fd4222768ee90bc1aa0f8218691b11"}, ] typing-extensions = [ {file = "typing_extensions-4.2.0-py3-none-any.whl", hash = "sha256:6657594ee297170d19f67d55c05852a874e7eb634f4f753dbd667855e07c1708"}, {file = "typing_extensions-4.2.0.tar.gz", hash = "sha256:f1c24655a0da0d1b67f07e17a5e6b2a105894e6824b92096378bb3668ef02376"}, ] zipp = [ {file = "zipp-3.8.0-py3-none-any.whl", hash = "sha256:c4f6e5bbf48e74f7a38e7cc5b0480ff42b0ae5178957d564d18932525d5cf099"}, {file = "zipp-3.8.0.tar.gz", hash = "sha256:56bf8aadb83c24db6c4b577e13de374ccfb67da2078beba1d037c17980bf43ad"}, ]