Closed jacobm-splunk closed 1 year ago
Hi @jacobm-splunk, thanks for the feedback.
I'll admit I'm a little confused by this, and would appreciate some help understanding what's going wrong here or what poe should perhaps do differently.
Is the issue that poetry is itself installed into the poetry managed venv, and poe is trying to use that one instead of the one from the user's environment? I guess poetry is installed as a transitive dependency of a poetry plugin that is installed as dev dependency? If that's the case then this much is to be expected since with poetry run
poe is itself running inside poetry's venv and so will see the poetry executable from that venv.
Although that doesn't fully explain why it doesn't work... I'm also a bit confused why it's trying to execute poetry via a relative path... does shutil.which("poetry")
return a relative path? Could you check if wrapping it like Path(shutil.which("poetry")).resolve()
fixes the issue?
It might be interesting to check the following:
poetry -C <pyproject_dir> run which poetry
poetry -C <pyproject_dir> run python -c "import sys; print(sys.path)"
poetry -C <pyproject_dir> run python -c "from subprocess import run; run(['which', 'poetry'])"
Also I wonder if it makes a different if you cd into the project directory first? Any reason not to do this?
Finally I'm curious why you're using poetry run
to run poe. This approach should generally work, but it might simplify things a bit to install poe into the environment with pip or pipx and then run the poe command directly.
Hi @nat-n,
Is the issue that poetry is itself installed into the poetry managed venv, and poe is trying to use that one instead of the one from the user's environment?
Poe is trying to use the one installed in the poetry venv, but I don't think that's actually the root of the issue. It's expected since, we're running it with poetry.
I guess poetry is installed as a transitive dependency of a poetry plugin that is installed as dev dependency?
Yes, sort of. It's being installed as part of a gitlab pipeline where poetry is being used to manage our dependencies.
Although that doesn't fully explain why it doesn't work... I'm also a bit confused why it's trying to execute poetry via a relative path... does shutil.which("poetry") return a relative path?
Yes, exactly. shutil.which("poetry")
is returning the relative path: connector_gen/.venv/bin/poetry
, but the current working directory passed into the POpen
call is: /builds/phantom/connector-gen/connector_gen
:
exec_cache["poetry_virtualenv"] = (
Popen(
(self._poetry_cmd(), "env", "info", "-p"),
stdout=PIPE,
cwd=self.context.config.project_dir,
env=clean_env,
)
.communicate()[0]
.decode()
.strip()
)
Basically, shutil.which("poetry")
is returning a path relative to the repo root, and the cwd is self.context.config.project_dir,
, which resolves to one directory down in
Finally I'm curious why you're using poetry run to run poe. This approach should generally work, but it might simplify things a bit to install poe into the environment with pip or pipx and then run the poe command directly.
We're mostly using it to run poe since we're using poetry to manage our dependencies. We could definitely just install poe through pip and it would probably work.
I can try to those commands if you need them, but I'm pretty sure the issue is mostly a mismatch between the current working directory and the relative path generated through shutil.which("poetry")
.
Thanks for the extra info, it makes sense now.
I think this might work as a fix: https://github.com/nat-n/poethepoet/commit/9ba7285e5d4e20e4575accbc45c7d1b8224a535d
@jacobm-splunk Could you do me a favour and try out the fix with this pre-release? https://pypi.org/project/poethepoet/0.18.2b1/
I probably won't have time in the next month or two to get this done. The patching process is fairly involve on our systems.
I can however try it if this got merged.
Poe isn't finding the poetry executable:
We are running it with the
--root
option:poetry -C <pyproject_dir> run poe --root <path/to/pyproject.toml>
This works when run locally, but not when run in CI (Gitlab)
It looks like the
which poetry
is not finding the correct poetry executable path, because I confirmed thatconnector_gen/.venv/bin/poetry
exists relative to the root of the repo.I patched executor/poetry.py and this is the poetry executable and current working directory it's finding:
poetry_cmd: connector_gen/.venv/bin/poetry cwd: /builds/phantom/connector-gen/connector_gen
Basically, the poetry command it's finding with
which poetry
is relative to the repo root, but the current working directory is one directory in where our pyproject.toml is.Pyproject.toml is on directory from the root of the repo in connector_gen/pyproject.toml