Cargo behind a proxy - Githubissues

JP-Ellis commented 10 years ago

I am using trying to use Cargo from behind a proxy, and it is unable to fetch external repositories despite having all the appropriate environment variables and configurations.

More specifically, I have the following environment variables set:

export HTTP_PROXY=proxy.example.com:8000
export HTTPS_PROXY=proxy.example.com:8000
export FTP_PROXY=proxy.example.com:8000

along with their lowercase counterparts. Git and Cargo both have in their configuration:

[http]
proxy = proxy.example.com:8000

[https]
proxy = proxy.example.com:8000

and git from the command line works fine; however, when Cargo attempts to fetch a repository, I encounter

$ cargo build
    Updating git repository `https://github.com/pistondevelopers/piston`
Unable to update https://github.com/pistondevelopers/piston

Caused by:
  failed to fetch into /home/user/.cargo/git/db/piston-10111ca7958cf505

Caused by:
  [2] Failed to connect to github.com: No route to host

yet if I run git on its own, it works fine:

$ git clone https://github.com/pistondevelopers/piston
Cloning into 'piston'...
remote: Counting objects: 1959, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 1959 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1959/1959), 2.19 MiB | 575 KiB/s, done.
Resolving deltas: 100% (1252/1252), done.

tomjakubowski commented 10 years ago

Typically these proxy variables are given in the form of URLs (e.g. http://proxy.example.com:8000 or https://proxy.example.com:8443). It's possible that libgit2 is more strict about this than git or curl is.

JP-Ellis commented 10 years ago

I had actually thought of that, and I did try this with all the relevant proxy variables set with the full URLs. I.e., ~/.gitconfig and ~/.cargo/config containing:

[http]
proxy = "http://proxy.example.com:8000/"

[https]
proxy = "https://proxy.example.com:8000/"

and the environment variables being

export HTTP_PROXY=http://proxy.example.com:8000/
export HTTPS_PROXY=https://proxy.example.com:8000/
export FTP_PROXY=ftp://proxy.example.com:8000/

but the result remains the same.

tomjakubowski commented 10 years ago

Taking a quick look through libgit2's source, it looks like they handle reading HTTP proxy settings in a private-looking function, but only ever call this function from the winhttp transport.

And in fact, @alexcrichton has opened an issue for that here: libgit2/libgit2#2555

alexcrichton commented 10 years ago

Aha yes indeed! I would very much like cargo to support proxies. I believe that all of our own personal HTTP requests are routed through proxies when necessary (mostly when dealing with the registry), but as @tomjakubowski found out this has yet to move over to libgit2 on unix.

My plan at this point is to make a git2-curl package (or something like that) which provides an HTTP transport powered through curl-rust. That way we can do maximum configuration with curl-rust to support things like HTTP/SOCKS proxies, etc.

Thanks for opening this! It's good to have a tracker.

bartavelle commented 9 years ago

Is there a way to manually do the cloning ? I am new to rust, so I don't know where cargo is supposed to store the dependent repos ...

bartavelle commented 9 years ago

Ok I just cloned it and used a file:/// type url ..

hamstergene commented 9 years ago

Until this issue is fixed, you can use alias cargo='http_proxy= https_proxy= proxychains4 cargo' to workaround the problem, it works well for me. On OS X, proxychains can be installed via brew install proxychains-ng and configured by editing /usr/local/etc/proxychains.conf.

aldanor commented 9 years ago

Same story here -- behind a corporate proxy with SSL certificate injection (and both HTTP_PROXY/HTTPS_PROXY set to the same address):

$ cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  SSL error: error:140E0114:SSL routines:SSL_shutdown:uninitialized

E.g., I normally get git and wget working by settings http.sslverify=false and --no-check-certificate as defaults, but with cargo there doesn't seem a way around this.

Is this also caused by libgit2? If not and if it's caused by ssl verification, it'd be nice to have an option to ignore it (same as in `git).

alexcrichton commented 9 years ago

@aldanor yes that's coming from libgit2, but I don't think that it's coming from http.sslverify as I believe that libgit2 just doesn't handle connecting to HTTP proxies.

They did have support for http.sslverify awhile back, but it looks like they semi-recently removed it. We could always add support for it in Cargo as well though!

Are you sure that the only error there is http.sslverify, and not the fact that cargo isn't connecting to a proxy though?

aldanor commented 9 years ago

@alexcrichton Nope, not sure it's exactly the same problem (other than the OP's error was "no route to host" and not SSL-related), wonder how to confirm it? It's just I've faced this same problem with most tools and package managers (git, npm, wget, pip to name a few) and in each case it was solved by disabling ssl verification, hence the assumption it was also the problem here.

If I enable http.sslverify on git, it yields:

$ git config --global http.sslverify true
$ git clone https://github.com/rust-lang/cargo.git
Cloning into 'cargo'...
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://github.com/rust-lang/cargo.git/info/refs
fatal: HTTP request failed

which is a different SSL routine and a different error code, hm...

So maybe libgit2 fails at connecting to proxy in the first place, like you said.

alexcrichton commented 9 years ago

Ok, thanks for checking @aldanor! I've opened https://github.com/rust-lang/cargo/issues/1180 as well to track http.sslverify which is somewhat separable from this issue. It's definitely something we should fix!

Jojoshua commented 9 years ago

+1

C:\master\src>cargo build Updating registry https://github.com/rust-lang/crates.io-index An unknown error occurred

To learn more, run the command again with --verbose.

C:\master\src>cargo build --verbose Updating registry https://github.com/rust-lang/crates.io-index An unknown error occurred

Caused by: failed to send request: A connection with the server could not be established

alexcrichton commented 9 years ago

Ok, as a status update to this I've finished binding libgit2's custom transport API and I've written a simple HTTP backend using libcurl: https://crates.io/crates/git2-curl.

I'll try to hook this up into cargo and then we should get proxy support for free because libcurl supports it out of the box. Unfortunately the implementation is not super efficient (reads an entire network operation, aka repository, into memory) due to the current design of curl-rust, but it should serve as an adequate enough stopgap for now.

rohitjoshi commented 9 years ago

Great!! This would defiantly help/

aldanor commented 9 years ago

Wonder if this is supposed to work now? (not quite clear from the merge mentioned above). Are cargo/libgit2 now expected to respect HTTP_PROXY? Has anyone had any success with that?

// Weird thing, I'm not getting a "connection refused" error while using the same proxies as before (were previously causing "ssl certificate failed").

alexcrichton commented 9 years ago

@aldanor yeah this should in theory work now, could you gist the errors you're seeing?

worker2345234 commented 9 years ago

Hi, I'm working behind our company firewall using a Windows 7 PC. I've no futher Network skills. I took the Proxy IP from my Internet Explorer and tried:

C:\Data\Projects\rust\rustprj>cargo --version
cargo 0.3.0-nightly (06dbe65 2015-05-29) (built 2015-06-01)

C:\Data\Projects\rust\rustprj>SET http_proxy=http://10.3.3.35:8080/

C:\Data\Projects\rust\rustprj>SET https_proxy=https://10.3.3.35:8080/

C:\Data\Projects\rust\rustprj>cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [12] SSL connect error

C:\Data\Projects\rust\rustprj>

Any ideas?

THX, Mark

alexcrichton commented 9 years ago

@worker2345234 could you try also setting the .cargo/config values? Trying this out locally it does indeed use the proxy, so it could be that your proxy is rejecting the connections regardless.

worker2345234 commented 9 years ago

Ok, I created a small .cargo/config file:

[http]
proxy = "10.3.3.35:8080"  # HTTP proxy to use for HTTP requests (defaults to none)
timeout = 60000      # Timeout for each HTTP request, in milliseconds

The cargo result is the same as yesterday. I tried to load https://github.com/rust-lang/crates.io-index directly in the InternetExplorer, works. A screenshot from the InternetExplorer-Settings: proxy

alexcrichton commented 9 years ago

@worker2345234 I'm not sure how libcurl recognizes the protocol of the URL, but could you try starting the url with http:// just to make sure that libcurl thinks it's a HTTP proxy?

aldanor commented 9 years ago

@alexcrichton With proxy set both in [http] section in .cargo/config and in HTTP{S,}_PROXY environment variables, I get the following (after a very long wait):

$ cargo build -v
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  SSL error: error:140E0114:SSL routines:SSL_shutdown:uninitialized

Pretty weird, given that I can curl the same URL from the same shell just fine.... Is it some http request that fails or a git request? I remember I had to turn off ssl verification for some tools to make them work (due to the proxy doing something with the certificate), not sure it's relevant here. I also have both https and http proxy pointing to the same host (http://some-proxy:8080), again not sure if relevant.

alexcrichton commented 9 years ago

@aldanor it's difficult to say without more information, because it could be that the proxy is getting hit, but just isn't working. It would be useful if you could follow the network traffic (e.g. via wireshark) to see where it's all going, because if Cargo's using the proxy (which it is if those env vars and config sections are set), then it's either a bug with Cargo's libcurl or perhaps the proxy.

worker2345234 commented 9 years ago

Ok, I tried in .cargo/config:

proxy = "http://10.3.3.35:8080"  # HTTP proxy to use for HTTP requests (defaults to none)

Result:

and

proxy = "https://10.3.3.35:8080"  # HTTP proxy to use for HTTP requests (defaults to none)

Result:

I recorded the Network traffic with wireshark, but as mentioned above, sorry I've got no network skills and I can't interpret the content. Unfortunately I can't see a way how to provide the .pcapng files. I will send them to you by email and hope, it's ok for you and they give you a clue to get cargo to hop over my company proxy.

aldanor commented 9 years ago

Unfortunately, I don't have sudo rights at work so can't use Wireshark, I've tried the proxychains method above though and got

[proxychains] DLL init: proxychains-ng 4.9
    Updating registry `https://github.com/rust-lang/crates.io-index`
[proxychains] Strict chain  ...  10.170.129.90:8080  ...  github.com:443  ...  OK
Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  The SSL certificate is invalid

... although git clone from https works just

The sslverify=false git config option doesn't seem to affect this either...

Can cargo respect git's sslverify option properly?

alexcrichton commented 9 years ago

@aldanor ah that's covered by https://github.com/rust-lang/cargo/issues/1180

@worker2345234 ah unfortunately I don't think I can infer much from that info either, my best guess is that something about the proxy isn't working correctly, (e.g. it doesn't like the connect or the destination host), but it's tough to say what's going on :(

aldanor commented 9 years ago

@alexcrichton Oh, indeed -- I see I was already subscribed to #1180 but that was almost half a year ago. Are there any plans (however distant) to revive that?

Thanks!

alexcrichton commented 9 years ago

I don't have any current plans to implement it soon, but I think it should be relatively easy to do, and I'd be willing to help walk you through if you're interested in working on it!

XuefengWu commented 9 years ago

I doest use proxy, but I also meet "The SSL certificate is invalid"

cargo 0.3.0 (35c05ae 2015-06-17) (built 2015-06-17) mac os x 10.10.3 openssl 1.0.2c

$ cargo build -v
    Updating registry `https://github.com/rust-lang/crates.io-index`
failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [16] The SSL certificate is invalid

screen shot 2015-06-20 at 3 20 06 pm

worker2345234 commented 7 years ago

Hi there! from time to time I give cargo/rust a try always hoping that I can also use it at work. Currently I'm running on my local machine a ntlm proxy for authtification to our company proxy.

I just downloaded the nightly rust installer:

C:\Temp>cargo new --bin testrust
C:\Temp\testrust>cargo version
cargo 0.13.0-nightly (9c47815 2016-11-04)

I added a dependency, so my cargo.toml is

[package]
name = "testrust"
version = "0.1.0"
authors = ["Mark <mark.ziegler@sew-eurodrive.de>"]

[dependencies]
time = "0.1.12"

and my .cargo/config:

[http]
    proxy = "http://localhost:3128/"
  sslVerify = "false"

[https]
    proxy = "https://localhost:3128/"
  sslVerify = "false"

Then building the Hello World aborts:

    Updating registry `https://github.com/rust-lang/crates.io-index`
thread 'main' panicked at 'non-utf8 error: Utf8Error { valid_up_to: 116 }', ../s
rc/libcore\result.rs:799
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'callback has panicked, and continuing to unwind into
C is not safe, so aborting the process', C:/msys64/slave/nightly-dist-cargo-win-
msvc-64/cargo-home\registry\src\github.com-88ac128001ac3a9a\git2-0.4.4\src\panic
.rs:35
stack backtrace:
   0:        0x13fd1cdb4 - <unknown>
   1:        0x13fd1abfb - <unknown>
   2:        0x13fd1b5bd - <unknown>
   3:        0x13fc16873 - <unknown>
   4:        0x13fc19118 - <unknown>
   5:      0x7fef87a691f - _unDNameEx
   6:      0x7fef879ea49 - _FrameUnwindFilter
   7:      0x7fef879f1f0 - CxxThrowException
   8:      0x7fef879ec7c - _GetPlatformExceptionInfo
   9:      0x7fef879f5d6 - _CxxFrameHandler2
  10:         0x778f7f0c - RtlDecodePointer
  11:         0x778e7d4b - RtlUnwindEx
  12:         0x776e052d - RtlUnwindEx
  13:      0x7fef879f544 - SetThrowImageBase
  14:      0x7fef879d657 - _DestructExceptionObject
  15:      0x7fef879d9c6 - _DestructExceptionObject
  16:      0x7fef879ed63 - _GetPlatformExceptionInfo
  17:      0x7fef879f5d6 - _CxxFrameHandler2
  18:         0x778f7e8c - RtlDecodePointer
  19:         0x778e84ce - RtlUnwindEx
  20:         0x778e8ac7 - RtlRaiseException
  21:      0x7fefd6ea06c - RaiseException
  22:      0x7fef879ef62 - CxxThrowException
  23:        0x13fd1f35a - <unknown>
  24:        0x13fd1b6cb - <unknown>
  25:        0x13fd1b661 - <unknown>
  26:        0x13fd1b456 - <unknown>
  27:        0x13fd1b374 - <unknown>
  28:        0x13fd1b30b - <unknown>
  29:        0x13fd27905 - <unknown>
  30:        0x13fc349a3 - <unknown>
  31:        0x13fc3654c - <unknown>
  32:        0x13fc35da8 - <unknown>
  33:        0x13fc37f4f - <unknown>
  34:        0x13f9c3293 - <unknown>
  35:        0x13f9c4096 - <unknown>
  36:        0x13fc190b5 - <unknown>
  37:        0x13fd2e669 - <unknown>
  38:        0x13fd7276b - git_libgit2_shutdown
  39:        0x13fd2e206 - <unknown>
  40:        0x13fd5cea2 - git_remote_connect
  41:        0x13fd5d9be - git_remote_fetch
  42:        0x13fc1e28b - <unknown>
  43:        0x13fbf13ea - <unknown>
  44:        0x13fb4b340 - <unknown>
  45:        0x13fb6104f - <unknown>
  46:        0x13fb661bf - <unknown>
  47:        0x13fb66453 - <unknown>
  48:        0x13fa67ba0 - <unknown>
  49:        0x13fa6bcab - <unknown>
  50:        0x13fa21f92 - <unknown>
  51:        0x13fa56cf1 - <unknown>
  52:        0x13fa5975b - <unknown>
  53:        0x13fa52fba - <unknown>
  54:        0x13fb36647 - <unknown>
  55:        0x13fb352ac - <unknown>
  56:        0x13fa7eafc - <unknown>
  57:        0x13fa81504 - <unknown>
  58:        0x13fa7e704 - <unknown>
  59:        0x13fac677e - <unknown>
  60:        0x13f93f10f - <unknown>
  61:        0x13f960934 - <unknown>
  62:        0x13f94a461 - <unknown>
  63:        0x13f93c9e5 - <unknown>
  64:        0x13f948885 - <unknown>
  65:        0x13fd1f281 - <unknown>
  66:        0x13fd1a63a - <unknown>
  67:        0x13fe16f82 - git_refdb_backend_fs
  68:         0x776c59cc - BaseThreadInitThunk
thread panicked while panicking. aborting.

If there is a smart programmer out there able to fix it and just need more Information...please feel free to ask me. If I have to change anything on my machine to get it to work I'm lucky about any hint.

alexcrichton commented 7 years ago

Interesting! @worker2345234 what's your locale on Windows? Is it perhaps non-english?

worker2345234 commented 7 years ago

You're right Alex, it is a german Windows.

worker2345234 commented 7 years ago

By default the german Windows (it is still Windows Vista) uses cp1252.

alexcrichton commented 7 years ago

@worker2345234 ah ok in that case I think you're hitting an assert fixed in https://github.com/alexcrichton/curl-rust/commit/09f044bfc14f6c6274c554287ee97b07e65de988, which is being updated for Cargo in https://github.com/rust-lang/cargo/pull/3272, so hopefully once that lands you'll be able to at least get a better error message.

The problem here is unfortunately that libcurl is telling us something went wrong, and we had a tough time getting that information out to you. With https://github.com/rust-lang/cargo/pull/3272 you'll probably still hit an error, but it shouldn't be a panic.

minecrawler commented 7 years ago

@alexcrichton Hi there, I have the same problem and a similar setup as @worker2345234 .

Windows 7 x64 (set to en_US locale)
cargo 0.16.0-nightly (3568be9 2016-11-26)
latest CNTLM up and running on localhost:3128
git, npm and others all use the application->cntlm->corporate proxy->internet chain successfully
Cargo hangs on Updating registry https://github.com/rust-lang/crates.io-index

my ~/.cargo/config file:

[http]
proxy = "http://localhost:3128"
sslVerify = "false"

[https]
proxy = "https://localhost:3128"
sslVerify = "false"

When I add a local registry, I get the following error message:

 Downloading aho-corasick v0.5.3 (registry file:///C:/GitHub/crates.io-index.git)
error: unable to get packages from source

Caused by:
  [35] SSL connect error (schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - The revocation function was unable to check revocation for the certificate.)

cosmo0920 commented 7 years ago

@minecrawler It seems that it is cargo regression. Does this workaround work for your environment? https://github.com/rust-lang/cargo/issues/598#issuecomment-56089462

alexcrichton commented 7 years ago

@minecrawler the [https] section isn't read by Cargo, but it looks like [http] is being picked up and the proxy is being used. It then looks like Cargo is rejecting the SSL certificate coming back, which presumably means that the corporate proxy is intercepting TLS traffic. It also looks like the certificate presented isn't valid according for your system.

In short, it looks like the problem is that Cargo asserts the validity of certificates but the certificate it's working with isn't valid. Cargo doesn't currently respect sslVerify like git does. Unfortunately the http.cainfo option which is normally used to configure what certificates Cargo trusts doesn't work on Windows (it's not compatible with schannel).

The tl;dr; I believe is that on Windows right now there's no way to get Cargo to accept an invalidate SSL certificate. Ideally the relevant certificates would be added to your OS certificate store, but we still need to implement custom validation.

minecrawler commented 7 years ago

@cosmo0920 Thank you for your hint, but unfortunately that workaround does not work for me.

@alexcrichton Thank you for the explanation. There is just one thing I do not understand: Why would it work on an OS other than Windows? How would you go about it on Linux and why shouldn't something like that be a viable option on Windows?

alexcrichton commented 7 years ago

Each platform has its own method of establishing and validating SSL connections. On Windows the HTTP library we're using, libcurl, doesn't read cainfo but on Linux/OSX it does.

worker2345234 commented 7 years ago

I just downloaded the nightly aaaaannnnddd.... finally it works (after almost 2 years....trying,trying,trying)!

First time that I could call successfully e.g. 'cargo install racer' , thanks for the #3699 fix.

Highfive Alex, I'm sure this will give rust a kick into company environements.

Here a brief summery how I could get it to work (Win7 inside my company behind company Firewall with NTLM authentification):

1.) download and install CNTLM proxy on my PC to manage the authentification for our company firewall 2.) in c:\Users\foo.cargo\config I added:

[http]
proxy = "127.0.0.1:3128" # use my cntlm proxy for authentification
timeout = 60000     # Timeout for each HTTP request, in milliseconds
check-revoke = false # please ignore the man in the middle attack of our IT department

@Alex: Would be nice, if you will give the rustup team a hint that they also implement a check-revoke=false option

alexcrichton commented 7 years ago

@worker2345234 glad it's working, thanks for the confirmation!

Want to file an issue over at rust-lang-nursery/rustup.rs for the revocation issue with rustup?

worker2345234 commented 7 years ago

rustup also uses curl lib to download the binaries. That's why the rustup tool often doesn't work at companies. As workaround I always download the binary rust-installers but the prefered way would be rustup. Especially for cross compiling... I guess you might explain them better why it is important to turn the ssl-revocation off.

alexcrichton commented 7 years ago

Oh sure, an issue is just a good tracking point for integration

minecrawler commented 7 years ago

was excited to hear that someone got it working, however it's still not working for me :(

PS C:\> cargo --version
cargo 0.18.0-nightly (5db6d64 2017-03-03)
PS C:\> cargo install amethyst_tools --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): [12/-20] early EOF
warning: spurious network error (1 tries remaining): [12/-20] early EOF
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [12/-20] early EOF

with cargo config:

[http]
proxy = "http://localhost:3128"
timeout = 60000
check-revoke = false
sslVerify = false

worker2345234 commented 7 years ago

I used the same cargo Version: C:\>cargo --version cargo 0.18.0-nightly (5db6d64 2017-03-03) But I had another error message: error: [35] SSL connect error (schannel: next InitializeSecurityContext failed: Unknown error (0x80092012....

worker2345234 commented 7 years ago

My error message with check-revoke=true is: C:\>cargo install --force rustfmt Updating registry 'https://github.com/rust-lang/crates.io-index' warning: spurious network error (1 tries remaining): [12/-2] [35] SSL connect error (schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - Die Sperrfunktion konnte keine Sperrp r?fung f?r das Zertifikat durchf?hren.) error: failed to fetch 'https://github.com/rust-lang/crates.io-index'

with check-revoke=false, everything is ok

aldanor commented 7 years ago

Can confirm that the issue has been resolved for me -- Linux, SSL_CERT_FILE and HTTP_PROXY both set; curl backend seems to pick it up fine, git-sourced crates work fine as well; no changes to .cargo/config required. Whichever was the change that fixed it, it seems to work now.

alexcrichton commented 7 years ago

@minecrawler oh dear, and to confirm you placed the configuration in ~/.cargo/config, right?

minecrawler commented 7 years ago

@alexcrichton yes, I should have edited the right one, because before I upgraded, I had the local registry configured in that file, which worked (at least for compilation)

alexcrichton commented 7 years ago

@minecrawler you may be running into a different issue then unfortunately :(

minecrawler commented 7 years ago

@alexcrichton is there any way to get more information? For example the reply from the proxy or the reply from the destination? That's the stuff I would expect from --verbose! I don't really want to start reverse-engineering the packet contents, but I think they might provide important information regarding the actual problem...

rust-lang / cargo

Cargo behind a proxy #636