search

rust-lang / rustup

The Rust toolchain installer
https://rust-lang.github.io/rustup/
Apache License 2.0
6.17k stars 889 forks source link

Proxy: Cannot download files #2523

Closed pJunger closed 3 years ago

pJunger commented 4 years ago

Problem Any command like update, check. etc. fails with operation timed out if a proxy is set with a leading http://.

Steps

  1. Set proxy with export http_proxy=http://gateway.something.net:10402 (& export https_proxy=http://gateway.something.net:10402)
  2. Run rustup update
  3. Receive 
    error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-updateMGVR4p/release-stable.toml'
error: caused by: failed to make network request
error: caused by: error sending request for url (https://static.rust-lang.org/rustup/release-stable.toml): operation timed out
error: caused by: operation timed out

Possible Solution(s) Workaround is possible by removing the http://

export http_proxy=gateway.something.net:10402
export https_proxy=gateway.something.net:10402
rustup update

Notes OS:

Ubuntu 20.04.1 (in WSL2)

Output of rustup --version:

rustup 1.22.1 (b01adbbc3 2020-07-08)

Output of rustup show:

Default host: x86_64-unknown-linux-gnu rustup home: /home/me/.rustup

stable-x86_64-unknown-linux-gnu (default) rustc 1.47.0 (18bf6b4f0 2020-10-07)

Edit: The * was meant only for obfuscation, replaced it with "something".

kinnison commented 4 years ago

I would guess that this would likely be a problem in env_proxy if it's anywhere. @inejge what do you think it might be?

dzhang-b commented 4 years ago

HI, I have the similar problem that using rustup in a environment required proxy.

Here is the debug output:

[0] # rustup --version
rustup 1.22.1 (b01adbbc3 2020-07-08)

Verbose Output

[0] # rustup --verbose update
verbose: read metadata version: '12'
info: no updatable toolchains installed
info: checking for self-updates
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-updateag7RVu/release-stable.toml'
error: caused by: failed to make network request
error: caused by: error sending request for url (https://static.rust-lang.org/rustup/release-stable.toml): error trying to connect: unexpected EOF
error: caused by: error trying to connect: unexpected EOF
error: caused by: unexpected EOF

My current environment has these variables set

http_proxy=http://proxy...
ftp_proxy=ftp://proxy-dmz...
ALL_PROXY=socks5://proxy...
socks_proxy=proxy-dmz....
https_proxy=https://proxy...

Using the RUSTUP_USE_CURL=1 given the following output:

[0] # RUSTUP_USE_CURL=1 rustup --verbose update
verbose: read metadata version: '12'
info: no updatable toolchains installed
info: checking for self-updates
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-updatedc1VH6/release-stable.toml'
error: caused by: error during download
error: caused by: [35] SSL connect error

The related issues so far:

  1. https://github.com/rust-lang/rustup/issues/35
  2. https://github.com/rust-lang/rustup/issues/1736
  3. https://github.com/rust-lang/rustup/issues/1328
inejge commented 4 years ago

I can't reproduce the reported behavior when running the following program:

// env_proxy = "0.4"
// url = "2"

use env_proxy::for_url;
use url::Url;

fn main() {
    let manifest = "https://static.rust-lang.org/rustup/release-stable.toml";
    let proxy = for_url(&Url::parse(manifest).unwrap());
    dbg!(proxy.to_url());
}

The program mimics rustup's usage of env_proxy. Both

https_proxy=http://gateway.*.net:10402 ./proxy-test

and

https_proxy=gateway.*.net:10402 ./proxy-test

produce the same output:

[src/main.rs:10] proxy.to_url() = Some(
    "http://gateway.*.net:10402/",
)

I may try with rustup itself later, but I don't have access to WSL2, so I can't recreate the exact environment. As an aside, gateway.*.net is one very funky hostname, but it shouldn't matter if it's properly defined in /etc/hosts or the local resolver.

X-21 commented 3 years ago

I would guess that this would likely be a problem in env_proxy if it's anywhere. @inejge what do you think it might be?

I have a issue when install rust and I guess it's like this problem about proxy.

$ curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh

error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' to '/home/edc/.rustup/tmp/e6kk8puoa1s49au8_file'
error: caused by: failed to make network request
error: caused by: error sending request for url (https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256): operation timed out
error: caused by: operation timed out
$ echo $http_proxy
http://192.168.10.1:10899/

# edc @ ubuntu in ~ [15:45:06] 
$ echo $https_proxy
https://192.168.10.1:10899/

And I turn off the Ubuntu's system proxy, restart and the installation script would have successed.

1328

kinnison commented 3 years ago

@X-21 If you're in a position to reproduce the issue, I'd appreciate it if you could. If you copy your rustup binary as rustup-init then it will save you constantly redownloading the installer during tests. For example if you do:

cp $(which rustup) /tmp/rustup-init
strace -f -s1024 -o /tmp/trace.log /tmp/rustup-init

And then once it has failed as above, you can attach trace.log to a comment (do not paste it in, attach it please, it'll be quite big) and we can look to see if we can see what's going on.

X-21 commented 3 years ago

#

@X-21 If you're in a position to reproduce the issue, I'd appreciate it if you could. If you copy your rustup binary as rustup-init then it will save you constantly redownloading the installer during tests. For example if you do:

cp $(which rustup) /tmp/rustup-init
strace -f -s1024 -o /tmp/trace.log /tmp/rustup-init

And then once it has failed as above, you can attach trace.log to a comment (do not paste it in, attach it please, it'll be quite big) and we can look to see if we can see what's going on.

Sure, I am pleasure to help that.

cp $(which rustup) /tmp/rustup-init
strace -f -s1024 -o /tmp/trace.log /tmp/rustup-init

Then it was failed as expected and generated the trace.log which I attached. trace.log

kinnison commented 3 years ago

Okay, so from the looks of that trace, it connected to your proxy and sent the clienthello etc. Your proxy then did not respond. Are you sure the https proxy is actually working? Could you also do the following for me to see how curl deals with things?

strace -f -s1024 -o /tmp/curltrace.log curl https://sh.rustup.rs > /dev/null

Then attach curltrace.log please?

X-21 commented 3 years ago

Okay, so from the looks of that trace, it connected to your proxy and sent the clienthello etc. Your proxy then did not respond. Are you sure the https proxy is actually working? Could you also do the following for me to see how curl deals with things?

strace -f -s1024 -o /tmp/curltrace.log curl https://sh.rustup.rs > /dev/null

Then attach curltrace.log please?

Yes I am sure the proxy is working. I can't visit some entertainment website without the proxy. 1

And the curltrace.log: curltrace.log

kinnison commented 3 years ago

Okay, curl looks like it used HTTP not HTTPs to talk to the proxy on your system. Fascinating. If I set similar environment variables and run curl, it uses HTTPs to connect, not HTTP, same as rustup does.

On the other hand, wget interestingly uses HTTP by default when it tries to use a proxy.

Your curl is 7.47.0 whereas mine is 7.64.0. I wonder if this is simply the case that older versions of curl, and wget, always use http to connect to https proxies, whereas newer ones (and rustup) use https, and whatever proxying software is on that server can't cope with the choice between the two? Sadly your proxy doesn't announce itself in the 'connection established' response which makes it harder to guess.

Do you know what proxy software is in use?

X-21 commented 3 years ago

I use proxy by ubuntu's Network proxy like that: 32

so all request(http/https/socks5/ftp) will be transmitted to another machine in local network, and this machine will handle these request to a proxy software named shadowsocks what is a proxy work in Transport Layer.

I try curl https://www.google.com and capture it: 33330

X-21 commented 3 years ago

I use proxy by ubuntu's Network proxy like that: 32

so all request(http/https/socks5/ftp) will be transmitted to another machine in local network, and this machine will handle these request to a proxy software named shadowsocks what is a proxy work in Transport Layer.

I try curl https://www.google.com and capture it: 33330

and I capture it when download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' : 3232323180112 I guess the problem is casued by the Server Name of request for download file from static.rust-lang.org, which is doesn't indicate the destination domain.

curl google: 123333330826

download form rust-lang.org: bbb23180920

kinnison commented 3 years ago

The SNI is not relevant here, it's that we're doing SSL to the proxy, which the proxy is not responding to properly.

Is the machine 192.168.10.1 running shadowsocks itself? If so, which version of the shadowsocks server?

X-21 commented 3 years ago

The SNI is not relevant here, it's that we're doing SSL to the proxy, which the proxy is not responding to properly.

Is the machine 192.168.10.1 running shadowsocks itself? If so, which version of the shadowsocks server?

Yes I know it was doing SSL to proxy, but I think it should do SSL to static.rust-lang.org via shadowsocks. Yes the machine in 192.168.10.1 is running shadowsocks and it's version is lastest - https://github.com/shadowsocks/shadowsocks-windows/releases/tag/4.3.3.0

kinnison commented 3 years ago

I agree that it should do SSL to static.rust-lang.org. It would achieve that by establishing the link to the proxy and then asking it to connect through to port 443 on static.rust-lang.org and then it'd establish SSL over that. Having read through shadowsocks' code, it uses privoxy as its underlying http proxy. Having been through privoxy's source code it has no SSL capability at all. Given that, your https_proxy environment variable should be set to http://server:port/ not to https://server:port/ Given that shadowsocks doesn't support anything but HTTP and SOCKS proxying, you also shouldn't be setting it in the ftp proxy slot in your config most likely.

I'd suggest not setting it in the https proxy slot, clients should fall back to http_proxy from there IIRC.

pJunger commented 3 years ago

I also added a strace log from running rustup self update for my configuration: rustup-trace-update.log

X-21 commented 3 years ago

I agree that it should do SSL to static.rust-lang.org. It would achieve that by establishing the link to the proxy and then asking it to connect through to port 443 on static.rust-lang.org and then it'd establish SSL over that. Having read through shadowsocks' code, it uses privoxy as its underlying http proxy. Having been through privoxy's source code it has no SSL capability at all. Given that, your https_proxy environment variable should be set to http://server:port/ not to https://server:port/ Given that shadowsocks doesn't support anything but HTTP and SOCKS proxying, you also shouldn't be setting it in the ftp proxy slot in your config most likely.

I'd suggest not setting it in the https proxy slot, clients should fall back to http_proxy from there IIRC.

Thanks, it is a socks5 proxy as you said. So I change the https_proxy to 'http://192.168.10.1:10899' make rustup and other commands working. I also have the confusion why rustup can't work but others can. I only set socks slot and http slot initially, but some commands like curl https://www.google.com/ -o 1.html would not be working and curl http://www.google.com/ -o 1.html is ok. So I set https slot to 'https://192.168.10.1:10899' and those would be fine. Actully I should have set https_proxy to 'http://192.168.10.1:10899' not to ‘https://ip:port’ , both two setting were working therefore I did not care these setting anymore.

Piping commented 3 years ago

I can confirm the behavior and right fix for rustup 1.23.1.

The fix is to change https_proxy environment variable to use http://proxy_version_url.

Then rustup is able talk to static.rust-lang.org.

Is it a bug that rustup does not use https://proxy_url correctly?

kinnison commented 3 years ago

No, this is a bug that your proxy doesn't support https:// proxy urls properly, and you've not encountered it otherwise because your OS is running old enough versions of curl that it doesn't support them properly either. Is there a chance you can unset the https proxy variable? Most things will then fall back to the all proxy anyway, which should be set to an http:// url which will then work.

monk279 commented 2 years ago

@pJunger what does "something" exactly mean?