Sequences are slow

[pawamoy]

I noticed that poe takes much more time than the Makefile equivalent to run a sequence of shell commands .

$ time make clean

real    0m0.085s
user    0m0.027s
sys     0m0.037s

$ time poetry run make clean

real    0m0.683s
user    0m0.566s
sys     0m0.114s

$ time poe clean

real    0m1.567s
user    0m1.244s
sys     0m0.231s

Could it be because poe tries to run each item of a sequence through poetry run? Even though I'm running it with the poe='poetry run poe' alias?

[nat-n]

Indeed the extra subprocess loading poetry for each step does add some overhead as you can see, there's an extra subprocesses involved plus loading/running poetry which is itself a bit heavy.

Unfortunately poetry doesn't provide a way to detect when you're being run as a subprocess in poetry run, I played around a bit with trying to detect it or use poetry's internals directly but felt it was getting too hacky (and was advise that poetry's internals are no stable).

I'm open to suggestions for how to improve in this area, perhaps by interacting with the poetry/run/shell API in a smarter way.

[pawamoy]

Just an idea: maybe you can avoid poetry run by checking if the VIRTUAL_ENV environment variable, which presence means the venv is activated, is equal to what poetry env info is returning (using poetry's internals)?

$ poetry run bash -c 'echo $VIRTUAL_ENV'
/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-r3EdK4eL-py3.8
$ poetry env info -p
/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-r3EdK4eL-py3.8

[pawamoy]

Oh, maybe you could use sys.prefix and sys.real_prefix insead, see https://stackoverflow.com/a/1883251/3451029.

[nat-n]

@pawamoy thanks for the suggestions, I just got around to having another look into this and still don't see a perfect solution but I might have improved it enough for your use case.

The tricky part is that AFAIK the only safe way to get the poetry virtualenv is with poetry env show -p which is about as expensive as poetry run, and so only worthwhile for task sequences.

The good news is that having the path to the virtualenv allows skipping poetry run if the virtualenv is already acitve, or replacing it with the virtualenv's activate script otherwise, which is much faster (at least for posix systems, I've not looked into how to make it work in a native windows context yet).

There's still the extra cost of at least one call to poetry to get the virtualenv. Theoretically this could be done much faster without calling poetry at all, but logically it relies on some arbitrary implementation details of poetry, which probably aren't stable long term.

I'd be interested to know if this works for you. I'm going to consider this issue resolved for now.