Open FlowMatric opened 1 month ago
dimbleby
commented
1 month ago Poetry is doing an editable install. "The same way as pip" is not a goal, but pip install -e path is a closer equivalence.
Though nb there is more than one way to do an editable install, I forget which pip is likely to do or when that behaviour changed.
dimbleby
commented
1 month ago Also the way you have set up the packages key in your pyproject.toml files looks suspect.
Poetry itself makes use of namespace packages, you likely want to do the same thing that it and poetry-core do
patrickhulce
commented
1 month ago I'm glad I'm not the only one facing this 😅
If it helps anyone else searching, removing...
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"from pyproject.toml restored the correct behavior for packages with shared namespaces. You lose PEP 517 builds but the repo works again.
dimbleby
commented
1 month ago the right fix in the example of this issue is to correct the packages key, then everything works fine
patrickhulce
commented
1 month ago @dimbleby do you have any examples you can point us to with the correct setup? All examples I have found online thusfar result in the same issue described above.
dimbleby
commented
1 month ago read up
you likely want to do the same thing that it and poetry-core do
FlowMatric
commented
1 month ago Also the way you have set up the
packageskey in your pyproject.toml files looks suspect.Poetry itself makes use of namespace packages, you likely want to do the same thing that it and poetry-core do
@dimbleby , thanks for your input so far. I am a bit at a loss now. You are right that it works perfectly fine with the configuration you proposed (see branch package_setup_2). I don't get it though because this is the setup that I started out with in a real-world project. I'll have to go back and better understand what is going on.
Maybe it's OK to keep the issue open until I can confirm that there is NO issue?
patrickhulce
commented
1 month ago Thanks for the direct link @dimbleby! That wasn't the part I had an issue with but was able to figure out a stray __init__.py in my linked venv was blowing things up.
@FlowMatric I found my difference by diffing the result of this script below with my existing env and found a lingering link in the poetry-managed venv in site-packages from before the multi-repo transition that contained an extra __init__.py at the namespace level. This extra __init__.py at the namespace level clobbers all previous subpackages in that namespace because the entire namespace gets treated as being sourced from that package instead of using the NamespaceLoader. In case it helps these were the various things that worked that seemed different from other examples I had tried (besides cheating by removing the poetry-core build backend).
src/namespace/pkg, NO OTHER FILES/DIRS in src/ or namespace/{ include = "namespace/pkg", from "src" } in pyproject.tomlpoetry env remove $(poetry env list))Repro Script
#!/bin/bash
set -euxo pipefail
TEMP_DIR=$(mktemp -d)
mkdir -p $TEMP_DIR/package_a/src/namespace/a
mkdir -p $TEMP_DIR/package_b/src/namespace/b
mkdir -p $TEMP_DIR/package_c/src/namespace/c
touch $TEMP_DIR/package_a/src/namespace/a/__init__.py
touch $TEMP_DIR/package_b/src/namespace/b/__init__.py
touch $TEMP_DIR/package_c/src/namespace/c/__init__.py
touch $TEMP_DIR/package_c/src/namespace/__init__.py # <-- this was the problem
cat >> $TEMP_DIR/pyproject.toml <<EOF
[tool.poetry]
name = "PACKAGE_NAME_HERE"
version = "0.1.0"
description = "Test"
authors = ["Patrick Hulce <patrick.hulce@gmail.com>"]
packages = [
{ include = "INCLUDE_PATH", from = "src" },
]
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.dependencies]
python = "==3.11.*"
EOF
cp $TEMP_DIR/pyproject.toml $TEMP_DIR/package_a/pyproject.toml
echo "VALUE_A = 1" >> $TEMP_DIR/package_a/src/namespace/a/__init__.py
sed -i '' 's/PACKAGE_NAME_HERE/namespace-a/g' $TEMP_DIR/package_a/pyproject.toml
sed -i '' 's#INCLUDE_PATH#namespace/a#g' $TEMP_DIR/package_a/pyproject.toml
cp $TEMP_DIR/pyproject.toml $TEMP_DIR/package_b/pyproject.toml
echo "import namespace.a" >> $TEMP_DIR/package_b/src/namespace/b/__init__.py
echo "VALUE_B = 2" >> $TEMP_DIR/package_b/src/namespace/b/__init__.py
sed -i '' 's/PACKAGE_NAME_HERE/namespace-b/g' $TEMP_DIR/package_b/pyproject.toml
sed -i '' 's#INCLUDE_PATH#namespace/b#g' $TEMP_DIR/package_b/pyproject.toml
echo '"namespace.a" = { path = "../package_a", develop = true }' >> $TEMP_DIR/package_b/pyproject.toml
cp $TEMP_DIR/pyproject.toml $TEMP_DIR/package_c/pyproject.toml
touch $TEMP_DIR/package_c/src/namespace/c/__init__.py
echo "import namespace.a" >> $TEMP_DIR/package_c/src/namespace/c/run.py
echo "import namespace.b" >> $TEMP_DIR/package_c/src/namespace/c/run.py
echo "print(namespace.a.VALUE_A)" >> $TEMP_DIR/package_c/src/namespace/c/run.py
echo "print(namespace.b.VALUE_B)" >> $TEMP_DIR/package_c/src/namespace/c/run.py
sed -i '' 's/PACKAGE_NAME_HERE/namespace-c/g' $TEMP_DIR/package_c/pyproject.toml
sed -i '' 's#INCLUDE_PATH#namespace/c#g' $TEMP_DIR/package_c/pyproject.toml
echo '"namespace.a" = { path = "../package_a", develop = true }' >> $TEMP_DIR/package_c/pyproject.toml
echo '"namespace.b" = { path = "../package_b", develop = true }' >> $TEMP_DIR/package_c/pyproject.toml
set +e
cd $TEMP_DIR/package_b
poetry install
poetry run python -c 'import namespace.a; print(namespace.a.VALUE_A)'
poetry run python -c 'import namespace.b; print(namespace.b.VALUE_B)'
cd $TEMP_DIR/package_c
poetry install
poetry run python -c 'import namespace.a; print(namespace.a.VALUE_A)'
poetry run python -c 'import namespace.b; print(namespace.b.VALUE_B)'
poetry run python -m namespace.c.run
poetry run python src/namespace/c/run.pyWith Accidental Extra init.py
# possible namespace for /project/packages/pkg1/src/namespace
# possible namespace for /project/packages/pkg2/src/namespace
# code object from /project/packages/pkg3/src/namespace/__init__.py
# created '/project/packages/pkg3/src/namespace/__pycache__/__init__.cpython-311.pyc'
import 'namespace' # <_frozen_importlib_external.SourceFileLoader object at 0x1025ea310>Without Accidental Extra init.py
# possible namespace for /project/packages/pkg1/src/namespace
# possible namespace for /project/packages/pkg2/src/namespace
# possible namespace for /project/packages/pkg3/src/namespace
import 'namespace' # <_frozen_importlib_external.NamespaceLoader object at 0x104a61ad0>
Description
I found that
poetry installcreates a virtual environment (or handles dependencies) not the same way as an environment managed by pip. IMO, poetry should be able to handle shared high level namespaces, the same way as pip does:When a poetry-managed project uses a namespace that is also used by a dependency, the project code in sub-packages is no longer available in the project
.venv.In contrast, when one creates a virtual environment with venv, differently named sub-packages are "on the path"
Here's the example project I created for this issue: https://github.com/FlowMatric/namespace_test
Workarounds
None
Poetry Installation Method
pipx
Operating System
Ubuntu 22.04
Poetry Version
1.8.3
Poetry Configuration
Python Sysconfig
Example pyproject.toml
Poetry Runtime Logs