Rye Error: Bootstrapping rye internals during install, sync, etc in wsl

[PadenZach]

Steps to Reproduce

Running on ubuntu in WSL behind company firewall + Cisco VPN

curl -sSf https://rye.astral.sh/get | bash

rye sync --verbose

Expected Result

Rye to install, sync.

This looks like a normal 404 error, however, the weird part is that all the downloads can be curled for me, ie:

curl https://github.com/astral-sh/uv/releases/download/0.2.22/uv-x86_64-unknown-linux-gnu.tar.gz --output uv.tar.gz
curl https://github.com/astral-sh/rye/releases/latest/download/rye-x86_64-linux.gz --output rye.tar.gz

both run successfully.

Additionally, I already have an installation of uv available.

Actual Result

This script will automatically download and install rye (latest) for you.
######################################################################## 100.0%
Welcome to Rye!

This installer will install rye to /home/zpaden/.rye
This path can be changed by exporting the RYE_HOME environment variable.

Details:
  Rye Version: 0.36.0
  Platform: linux (x86_64)

✔ Continue? · yes
✔ Select the preferred package installer · uv (fast, recommended)
✔ What should running `python` or `python3` do when you are not inside a Rye managed project? · Run a Python installed and managed by Rye
✔ Which version of Python should be used as default toolchain? · cpython@3.12
Installed binary to /home/zpaden/.rye/shims/rye
Bootstrapping rye internals
error: download of https://github.com/astral-sh/uv/releases/download/0.2.22/uv-x86_64-unknown-linux-gnu.tar.gz failed

Caused by:
    [6] Couldn't resolve host name (Could not resolve host: github.com)

After self install of rye and running rye sync:

Bootstrapping rye internals
error: could not sync because bootstrap failed

Caused by:
    0: download of https://github.com/astral-sh/uv/releases/download/0.2.22/uv-x86_64-unknown-linux-gnu.tar.gz failed
    1: [6] Couldn't resolve host name (Could not resolve host: github.com)

Version Info

❯ rye --version rye 0.36.0 commit: 0.36.0 (12c024c7c 2024-07-07) platform: linux (x86_64) self-python: not bootstrapped (target: cpython@3.12) symlink support: true uv enabled: true

Stacktrace

❯ RUST_BACKTRACE=1 rye sync --verbose Bootstrapping rye internals error: could not sync because bootstrap failed

Caused by: 0: download of https://github.com/astral-sh/uv/releases/download/0.2.22/uv-x86_64-unknown-linux-gnu.tar.gz failed 1: [6] Couldn't resolve host name (Could not resolve host: github.com)

Stack backtrace: 0: ::ext_context 1: rye::bootstrap::download_url_ignore_404 2: rye::uv::UvBuilder::ensure_exists 3: rye::bootstrap::ensure_self_venv_with_toolchain 4: rye::sync::sync 5: rye::cli::sync::execute 6: rye::cli::execute 7: std::panicking::try 8: rye::utils::panic::trap_bad_pipe 9: rye::main 10: std::sys_common::backtrace::__rust_begin_short_backtrace 11: std::rt::lang_start::{{closure}} 12: std::rt::lang_start_internal 13: main

[PadenZach]

This seems to be a common problem https://github.com/astral-sh/rye/issues/412#issue-1847624389 going to try and see if it's a WSL problem.

[PadenZach]

Downloaded the windows 64 bit installer and was able to successfully bootstrap. This seems to be an issue with WSL specifically, updating the title to match.

[PadenZach]

Disabling revocation checks in WSL resolved this issue for me locally :) IE:

    // on windows we want to disable revocation checks.  The reason is that MITM proxies
    // will otherwise not work.  This is a schannel specific behavior anyways.
    // for more information see https://github.com/curl/curl/issues/264
    // TEMP: Commented out to test on wsl.
    // #[cfg(windows)]
    // {
    println!("Disabling revocation checks");
    handle.ssl_options(curl::easy::SslOpt::new().no_revoke(true))?;
    // }

rye/target/debug on  main [!] via 🐍 v3.11.9
❯ cargo build
rye/target/debug on  main [!] via 🐍 v3.11.9
❯ RUST_BACKTRACE=1 ./rye sync
Bootstrapping rye internals
Disabling revocation checks
Downloading cpython@3.12.3
Disabling revocation checks
Checking checksum
Unpacking
Downloaded cpython@3.12.3
Downloading cpython@3.11.1
Disabling revocation checks
Checking checksum
Unpacking
Downloaded cpython@3.11.1
Initializing new virtualenv in /home/zpaden/workspace/rye/.venv
Python version: cpython@3.11.1
Generating production lockfile: /home/zpaden/workspace/rye/requirements.lock
Generating dev lockfile: /home/zpaden/workspace/rye/requirements-dev.lock
Installing dependencies
Resolved 38 packages in 24ms
   Built rye-devtools @ file:///home/zpaden/workspace/rye/rye-devtools
Prepared 38 packages in 3.34s
Installed 38 packages in 227ms
 + anyio==4.4.0
 + certifi==2024.6.2
 + charset-normalizer==3.3.2
 + click==8.1.7
 + colorama==0.4.6
 + ghp-import==2.1.0
 + h11==0.14.0
 + httpcore==1.0.5
 + httpx==0.27.0
 + idna==3.7
 + iniconfig==2.0.0
 + jinja2==3.1.4
 + markdown==3.3.7
 + markupsafe==2.1.5
 + mdx-gh-links==0.4
 + mergedeep==1.3.4
 + mkdocs==1.4.3
 + mkdocs-include-markdown-plugin==4.0.4
 + mkdocs-material==9.1.20
 + mkdocs-material-extensions==1.3.1
 + mkdocs-simple-hooks==0.1.5
 + mkdocs-version-annotations==1.0.0
 + packaging==24.1
 + pluggy==1.5.0
 + pygments==2.18.0
 + pymdown-extensions==9.11
 + pytest==8.0.2
 + python-dateutil==2.9.0.post0
 + pyyaml==6.0.1
 + pyyaml-env-tag==0.1
 + regex==2024.5.15
 + requests==2.32.3
 + rye-devtools==1.0.0 (from file:///home/zpaden/workspace/rye/rye-devtools)
 + six==1.16.0
 + sniffio==1.3.1
 + socksio==1.0.0
 + urllib3==2.2.2
 + watchdog==4.0.1
Done!

Looking into creating a PR to add this as an additional config option sometime this week. I assume this would be preferable over just creating an environmental variable.

[PadenZach]

yes, I already have done this. I have a few different nameservers (Public ones first) and a few search domains for my company's intranet

On Wed, Jul 10, 2024, 5:20 PM Vincent Besançon @.***> wrote:

Hello @PadenZach https://github.com/PadenZach, could you verify the content of /etc/resolv.conf ? I think it is using something that the VPN cannot route correctly. You may prefer to tell WSL to not auto generate the file in /etc/wsl.conf:

[network] generateResolvConf = false

Then, you may delete the resolv.conf (this is a symlink) and create the same file with your company nameserver (DNS_IP):

nameserver

— Reply to this email directly, view it on GitHub https://github.com/astral-sh/rye/issues/1212#issuecomment-2221624955, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKBR3QJDXSLNIXYIAFAVMLZLWXURAVCNFSM6AAAAABKTWJUIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRGYZDIOJVGU . You are receiving this because you were mentioned.Message ID: @.***>

[bigbrozer]

Thanks for replying despite I have removed my comment 😇 It was late here and did not read fully what you did until I seen this is not DNS related 😄

[PadenZach]

Oh no worries! I just thought github was being weird haha

Also seems like we may have another WSL issue here: https://github.com/astral-sh/rye/issues/1220

[PadenZach]

Hmmm okay, this is getting more confusing:

1. Normal install script does not bootstrap (eg: curl -sSf https://rye.astral.sh/get | bash)
2. When I build my own project, with my 'fix' it works great!... However, this happens whether or not I have the 'disable revocation check' setting enabled or disabled :/
3. Running cargo build --git https://github.com/astral-sh/rye to build also seems to work. Closing the PR since it doesnt seem to change the correct behavior.

[mc51]

Hmmm okay, this is getting more confusing:

1. Normal install script does not bootstrap (eg: `curl -sSf https://rye.astral.sh/get | bash`)

2. When I build my own project, with my 'fix' it works great!... However, this happens whether or not I have the 'disable revocation check' setting enabled or disabled :/

3. Running `cargo build --git https://github.com/astral-sh/rye` to build also seems to work. Closing the PR since it doesnt seem to change the correct behavior.

I can reproduce this. Building myself solves the issue without code changes. Also, it doesn't seem to be related to MITM, as I can turn my vpn off (which does MITM) and I still have the issue.

[mc51]

Seems that I finally found a solution for this. Please let me know if this also solves your issue @PadenZach.

Turns out, this is in fact a DNS issue. Being in WSL2 and using a company VPN usually requires changed to DNS via the /etc/resolv.conf file to regain connectivity. (here's a description of this). In my case, changing the order of the nameservers in the file solves this issue. If I use a public dns server as a first entry, things go back to normal.

Now, the wild thing about this is that has never been an issue anywhere else. So, it seems that the host name resolution of the pre-built rye is behaving weirdly. It would be interesting to know why. Also why isn't this a problem when building it oneself? I suspected that this might be due to differences in the linked libcurl version. This differs in the systems between the GitHub Actions build and my own build. However, as I understand it, the curl-rust dependency is built which a static libcurl. So again, this is weird!

[PadenZach]

My first two name server entries are already public entries (cloud flare DNS fwiw) so unfortunately this doesn't fully explain the issue on my end.

On Fri, Jul 12, 2024, 10:05 AM Michael @.***> wrote:

Seems that I finally found a solution for this. Please let me know if this also solves your issue @PadenZach https://github.com/PadenZach.

Turns out, this is in fact a DNS issue https://miro.medium.com/v2/resize:fit:599/0*AiREdqoPWbJMgB_0.jpg. Being in WSL2 and using a company VPN usually requires changed to DNS via the /etc/resolv.conf file to regain connectivity. (here's https://learn.microsoft.com/en-us/windows/wsl/troubleshooting#wsl-has-no-network-connectivity-once-connected-to-a-vpn a description of this). In my case, changing the order of the nameservers in the file solves this issue. If I use a public dns server as a first entry, things go back to normal.

Now, the wild thing about this is that has never been an issue anywhere else. So, it seems that the host name resolution of the pre-built rye is behaving weirdly. It would be interesting to know why. Also why isn't this a problem when building it oneself? I suspected that this might be due to differences in the linked libcurl version. This differs in the systems between the GitHub Actions build and my own build. However, as I understand it, the curl-rust dependency is built which a static libcurl. So again, this is weird!

— Reply to this email directly, view it on GitHub https://github.com/astral-sh/rye/issues/1212#issuecomment-2225786867, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKBR3QPCAPWL3BHSQK4GGDZL7WBXAVCNFSM6AAAAABKTWJUIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVG44DMOBWG4 . You are receiving this because you were mentioned.Message ID: @.***>

[mc51]

Some interesting things: If you self build rye, it will be linked to /lib/x86_64-linux-gnu/libresolv.so.2 and /lib/x86_64-linux-gnu/libnss_dns.so.2 which are relevant for dns. However, the pre built release isn't linked to those.
Also, the pre built release seems to be requesting all dns servers in /etc/resolv.conf in parallel, while the self built one is only using one server.

I don't understand why this happens, but this seems to explain the diverging behavior I ran strace -o /tmp/wtf -f rye sync against both binaries to find out those things.

/e: I finally figured out, what's going on. Turns out, the official linux release is built for target x86_64-unknown-linux-musl so instead of glibc, musl libc is being used. Hence, the dns resolution under the hood works differently. This explains the aforementioned observations. If I build locally using this target, I can reproduce my issues! Now the last part of the puzzle is to find out, how the dns resolution with musl libc fails. But this is an issue with that library and not rye. However, one could think about changing the build target for linux in the official release. Why was musl chosen in the first place? (is it a smaller binary or has better portability?)

Here are some descriptions of the idiosyncrasies of musl dns resolution which fall in line with my observations here:

For instance, musl performs parallel querying of name servers and can’t switch to sequential one like glibc. So if you start the Docker daemon with --dns=172.17.42.1 --dns=10.0.2.15, where the former is the local DNS server and the latter is used for external DNS resolving, there is no guarantee that 172.17.42.1 will be tried first, which will lead to unforeseen failures.

and

musl supports DNS over UDP (user datagram protocol) only, and doesn’t support DNS over TCP, which is used for packets larger than 512 bytes. It also doesn’t support the feature specified by the RFC standard, which allows to increase the size of the UDP packet above 512 bytes (limit for DNS over UDP) via the Extension Mechanism for DNS (EDNS). This leads to limited support for big DNS packets in musl, so in case of large DNS responses, musl resolver exits.

When querying githubusercontent.com (where rye tries to download the internals from) (e.g. via dig githubusercontent.com ANY) I receive back 1018 bytes using some nameservers (but not others). Thus, the musl targeted build will probably fail this request. This could probably explain your issue @PadenZach .

/e2: Last remark. The issue seems to have been fixed with musl libc >= 1.2.4. I tried building locally using that version, and in fact, it resolves the issue. Unfortunately, cross currently only supports version 1.2.3 for the x86_64-unknown-linux-musl target. So making that change to the build pipeline is not trivial (yet).