main branch fails with "0: unable to create virtualenv folder" wen syncing an existing venv

[dsp]

Steps to Reproduce

1. Clone recent rye
2. Built commit dbfdeeb90110e87754498d46ac46fca65578e80a (main as of writign this bug)
3. Use an older rye to create a project and sync: rye-old init rye-test && cd rye-test && rye-old sync.
4. Sync with rye from main rye sync

Expected Result

Synching successfully

Actual Result

$ rye sync
detected outdated rye internals. Refreshing
Bootstrapping rye internals
Found a compatible python version: cpython@3.12.1
Upgrading pip
Installing internal dependencies
Reusing already existing virtualenv
Generating production lockfile: /Users/dsp/src/rye-test/requirements.lock
Creating virtualenv for pip-tools
error: could not write production lockfile for project

Caused by:
    0: unable to create virtualenv folder '/Users/dsp/.rye/pip-tools/cpython@3.12'
    1: No such file or directory (os error 2)

Version Info

rye 0.22.0 commit: 0.21.0+21 (dbfdeeb90 2024-02-07) platform: macos (aarch64) self-python: cpython@3.12 symlink support: true

Stacktrace

No response

[cnpryer]

Use an older rye to create a project and sync: rye-old init rye-test && cd rye-test && rye-old sync.

Do you mean that you've tested this on an older version of Rye? If so do you have that rye-old --version output? Or do you mean you only reproduced this by running a sync on a project managed by an older version of Rye? If it's the latter, have you tried clearing Rye's home dir?

FWIW I just hit this with

❯ rye --version
rye 0.22.0
commit: 0.21.0+17 (e72311698 2024-02-05)
platform: windows (x86_64)
self-python: cpython@3.12
symlink support: false

I can try again on my Mac later today. I'm unable to look into this right this moment, but I can try to check it out later.

[cnpryer]

If anyone has a moment to test this on d4d2a52405c2c9aa28f071749eafe6921e901ec8 and 1deecb1e483ccebd1abfd327b85a4b39eb609ea8 I'd be interested in that.

[mitsuhiko]

So I think this one here:

    0: unable to create virtualenv folder '/Users/dsp/.rye/pip-tools/cpython@3.12'

Comes from this block: https://github.com/mitsuhiko/rye/blob/4ed0f87cc134a37bf7a68020524de1491873457c/rye/src/sync.rs#L313-L315

I think this points to the fact that the pip-tools folder was removed. We do wipe the pip-tools folder entirely if the internal toolchain revision changes, but I thought we create it before. Not sure why the folder is not created, would need to trace this back. But it might make sense to change this to create_dir_all just in case for now.

Previously we used virtualenv directly to create the folder which could also create parent folders. Maybe that's the source of that regression. But I would have expected it to fail all the time then.