Poe not finding the poetry executable

[jacobm-splunk]

Poe isn't finding the poetry executable:

  File "/builds/phantom/connector-gen/connector_gen/.venv/bin/poe", line 8, in <module>
    sys.exit(main())
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/__init__.py", line 33, in main
    result = app(cli_args=sys.argv[1:])
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/app.py", line 81, in __call__
    return self.run_task() or 0
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/app.py", line 114, in run_task
    return self.task.run(context=context, extra_args=self.ui["task"][1:])
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/task/base.py", line 240, in run
    return self._handle_run(
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/task/cmd.py", line 44, in _handle_run
    return context.get_executor(self.invocation, env, self.options).execute(
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/executor/poetry.py", line 24, in execute
    poetry_env = self._get_poetry_virtualenv()
  File "/builds/phantom/connector-gen/connector_gen/.venv/lib/python3.10/site-packages/poethepoet/executor/poetry.py", line 72, in _get_poetry_virtualenv
    Popen(
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1847, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'connector_gen/.venv/bin/poetry'

We are running it with the --root option: poetry -C <pyproject_dir> run poe --root <path/to/pyproject.toml>

This works when run locally, but not when run in CI (Gitlab)

It looks like the which poetry is not finding the correct poetry executable path, because I confirmed that connector_gen/.venv/bin/poetry exists relative to the root of the repo.

I patched executor/poetry.py and this is the poetry executable and current working directory it's finding:

poetry_cmd: connector_gen/.venv/bin/poetry cwd: /builds/phantom/connector-gen/connector_gen

Basically, the poetry command it's finding with which poetry is relative to the repo root, but the current working directory is one directory in where our pyproject.toml is.

Pyproject.toml is on directory from the root of the repo in connector_gen/pyproject.toml

[nat-n]

Hi @jacobm-splunk, thanks for the feedback.

I'll admit I'm a little confused by this, and would appreciate some help understanding what's going wrong here or what poe should perhaps do differently.

Is the issue that poetry is itself installed into the poetry managed venv, and poe is trying to use that one instead of the one from the user's environment? I guess poetry is installed as a transitive dependency of a poetry plugin that is installed as dev dependency? If that's the case then this much is to be expected since with poetry run poe is itself running inside poetry's venv and so will see the poetry executable from that venv.

Although that doesn't fully explain why it doesn't work... I'm also a bit confused why it's trying to execute poetry via a relative path... does shutil.which("poetry") return a relative path? Could you check if wrapping it like Path(shutil.which("poetry")).resolve() fixes the issue?

It might be interesting to check the following:

poetry -C <pyproject_dir> run which poetry
poetry -C <pyproject_dir> run python -c "import sys; print(sys.path)"
poetry -C <pyproject_dir> run python -c "from subprocess import run; run(['which', 'poetry'])"

Also I wonder if it makes a different if you cd into the project directory first? Any reason not to do this?

Finally I'm curious why you're using poetry run to run poe. This approach should generally work, but it might simplify things a bit to install poe into the environment with pip or pipx and then run the poe command directly.

[jacobm-splunk]

Hi @nat-n,

Is the issue that poetry is itself installed into the poetry managed venv, and poe is trying to use that one instead of the one from the user's environment?

Poe is trying to use the one installed in the poetry venv, but I don't think that's actually the root of the issue. It's expected since, we're running it with poetry.

I guess poetry is installed as a transitive dependency of a poetry plugin that is installed as dev dependency?

Yes, sort of. It's being installed as part of a gitlab pipeline where poetry is being used to manage our dependencies.

Although that doesn't fully explain why it doesn't work... I'm also a bit confused why it's trying to execute poetry via a relative path... does shutil.which("poetry") return a relative path?

Yes, exactly. shutil.which("poetry") is returning the relative path: connector_gen/.venv/bin/poetry, but the current working directory passed into the POpen call is: /builds/phantom/connector-gen/connector_gen:

exec_cache["poetry_virtualenv"] = (
  Popen(
    (self._poetry_cmd(), "env", "info", "-p"),
    stdout=PIPE,
    cwd=self.context.config.project_dir,
    env=clean_env,
  )
  .communicate()[0]
  .decode()
  .strip()
)

Basically, shutil.which("poetry") is returning a path relative to the repo root, and the cwd is self.context.config.project_dir,, which resolves to one directory down in /connector_gen

Finally I'm curious why you're using poetry run to run poe. This approach should generally work, but it might simplify things a bit to install poe into the environment with pip or pipx and then run the poe command directly.

We're mostly using it to run poe since we're using poetry to manage our dependencies. We could definitely just install poe through pip and it would probably work.

I can try to those commands if you need them, but I'm pretty sure the issue is mostly a mismatch between the current working directory and the relative path generated through shutil.which("poetry").

[nat-n]

Thanks for the extra info, it makes sense now.

I think this might work as a fix: https://github.com/nat-n/poethepoet/commit/9ba7285e5d4e20e4575accbc45c7d1b8224a535d

@jacobm-splunk Could you do me a favour and try out the fix with this pre-release? https://pypi.org/project/poethepoet/0.18.2b1/

[jacobm-splunk]

I probably won't have time in the next month or two to get this done. The patching process is fairly involve on our systems.

I can however try it if this got merged.

[nat-n]

@jacobm-splunk The fix is shipped in 0.19.0 🙂