search

astral-sh / rye

a Hassle-Free Python Experience
https://rye.astral.sh
MIT License
13.78k stars 468 forks source link

Custom location for venv #1211

Open nazq opened 3 months ago

nazq commented 3 months ago

I've been using rye for a while now and generally very happy with it over poetry. One thing I would like to see though is a general use param to specify the location of the venv. uv has some support for this, at least on venv creation. My thinking was to add an optional param to each of [sync, lock, run] (possibly a few of the others too) which would take the location of the venv, and if not specified keep using the current behaviour of .venv

With this we'd be able to make tools like nox, and tox work well. I'm also looking at building some parallel build and testing tools which would take advantage of this functionality.

Thoughts?

nazq commented 3 months ago

Taking a fork to add this feature, will see if it causes any issues. Will work on it here https://github.com/nazq/rye_venvs

nazq commented 3 months ago

I have a first cut up there which needs some work (mainly tests but a little clean up on ownership, and no docs updated yet). Is there anyone in here who can take a look? Maybe @charliermarsh ? Thanks all, and keep up the great work.

https://github.com/nazq/rye_venvs

nazq commented 3 months ago

I see a few issues here, I'll probably fix in the am. I need to instruct uv to use the right venv in all cases

nazq commented 3 months ago

Put this PR in for review. https://github.com/astral-sh/rye/pull/1222

nazq commented 3 months ago

If we can get this in next thing I plan on doing is putting in a PR for nox to support Rye as a first class tool. I found there is some support for tox but this change may make that easier too.

saucoide commented 3 months ago

Not being configurable is by design to push for standardization as far as i can tell: https://rye.astral.sh/guide/faq/#can-i-relocate-virtualenvs https://github.com/astral-sh/rye/issues/371

nazq commented 3 months ago

Not being configurable is by design to push for standardization as far as i can tell: https://rye.astral.sh/guide/faq/#can-i-relocate-virtualenvs #371

Indeed. I see that clearly in the docs. The PR I put in hues to that philosophy and does not change the default workflow in any way. What it does is give build tools which need it an escape hatch. IMO it just makes rye more powerful for users as it improves the build/test ecosystem

mitsuhiko commented 3 months ago

What is the exact need for why you need to place the venv elsewhere? That is not entirely clear from your description here. It seems to me to be motivated by other reasons than what has been discussed in #371 before.

nazq commented 3 months ago

What is the exact need for why you need to place the venv elsewhere? That is not entirely clear from your description here. It seems to me to be motivated by other reasons than what has been discussed in #371 before.

Didn't spot this response, so I'll cross post from the PR discussion.

Thanks for responding @mitsuhiko , much appreciated. I've currently moved some 90ish percent of all my projects over to rye and the experience is great. Rye + uv were the build tool change python (py/rust) has been waiting for.

Now before we dismiss this I think it's worth exploring it a little further. In general I don't even think about the venv any more when using rye but I do have some edgier cases which make the PR here appealing. One common case I have is mixed rust python builds or simply python builds which need to target multiple python versions. The single .venv makes it difficult to parallelise large build/test runs. I'm more than willing to add this functionality into other OSS tools which would then leverage rye and hopefully make it even more broadly used than today.

The PR here should have zero impact on anyone's day to day usage of rye, I was very careful to not change any the current functionality. But it then opens up wider opportunities, Rust brought speed to rye/uv. I think a change like this brings true exogenous parallelism.

There are alternatives to a --venv within rye I've played with, for example copying the project around into tmp folders, or even faking out the filesystem so rye still thinks it's using ./.venv but none seem as clean as adding this optional feature to the small number of commands which would need it.

I hope you reconsider, I think rye (and uv) have opened the universe of beaten down py developers to what could be. It would be great to expand on that without impacting any users.

Let me know your thoughts

phi-friday commented 3 months ago

What is the exact need for why you need to place the venv elsewhere? That is not entirely clear from your description here. It seems to me to be motivated by other reasons than what has been discussed in #371 before.

Since learning about rye, I've been using only rye except when it's unavoidable. But some of those unavoidable cases are problems.

For example, polars assumes that the contributor is using .venv in the parent folder(not root folder), and everything is done on that assumption. (They have all the support for this in their makefile).

Support for custom venvs would be nice, but not necessary. But I hope it's helpful for this discussion to know that it is.

phi-friday commented 3 months ago

What is the exact need for why you need to place the venv elsewhere? That is not entirely clear from your description here. It seems to me to be motivated by other reasons than what has been discussed in #371 before.

Since learning about rye, I've been using only rye except when it's unavoidable. But some of those unavoidable cases are problems.

For example, polars assumes that the contributor is using .venv in the parent folder(not root folder), and everything is done on that assumption. (They have all the support for this in their makefile).

Support for custom venvs would be nice, but not necessary. But I hope it's helpful for this discussion to know that it is.

I've just now checked out #371 . The method you pointed out, symbolic links, solves the problem with cases like polars. Thanks.

mitsuhiko commented 3 months ago

The single .venv makes it difficult to parallelise large build/test runs.

I think this is a stronger argument. That said, For parallelization moving the venv will not be sufficient I believe. The reason is that the Python build process pokes around in the build folder too and there is no locking. Checking the code out a second time in another folder is not an option for you I presume? I think I would be open to an environment variable to temporarily rebind the .venv but only if it's marked for that particular case.