Iron Port TLS 1.0 proxy, pub get fails

crodier commented 8 years ago

I am behind an _Iron Port firewall _in the office, on Windows. Could be me, but I found and tried 'https_proxy' and 'http_proxy', and versions of the Dart SDK from 1.12 to 1.15.0 Dev today, including 1.12, 1.13., 1.14, and 1.15.0 Dev, from 1/28/2016 (latest.)

I am trying to run the Dart, Angular2, getting started. Only getting packages from pub.dartlang.org.

On 15.0.0 from Dev today, the error is 407, Proxy Authentication Required..

The pub get trace logs are attached.

pub-get-proxy.zip

If I set https_proxy, get the same issue, I can set both without or with my password to the Iron Port proxy address. like https_proxy=iport:80, https_proxy=crodier:password@iport:80. Also with and without http_proxy, which has no impact on Dev.

The interesting bits:

npm works from behind the proxy, without any configuration
Maven and other http fetching tools, are ok behind the proxy, without any configuration (usually)
git has issues, not with git clone git protocol, but git https protocol
Between my windows machine and the Iron Port firewall, _TLS 1.0_ is used.
My guess: TLS 1.0 between me and the proxy is no longer supported by BoringSSL, or needs flags to be allowed.
In our case, TLS 1.0 is only between my machine and the Iron Port, on a closed network, which is not great, but not in the wild
The certificate used is the Iron Port certificate, for my organization, which has been given trust by root authorities.
Windows 7 Professional

I reviewed the alternative, manually downloading the packages, but not being able to run pub get is a major deterrent.

While I could investigate with the networking team, I doubt this leads to any changes in the proxy setup in my organization. It is also difficult to justify that investigation, with other tools working ok, and the proxy generally working, and being a vendor (Iron Port) firewall.

Dart looks to be an incredible platform, and I post in the hopes this is helpful. I also apologize, but I doubt I will be able to learn the code and work on the issue my self in the SDK etc. I can offer to re-test if there are changes to Dev, With guidance, I may be able to work on the sdk issue itself.

Best regards, Chris

kevmoo commented 8 years ago

@nex3 would you look at this quick? I believe you've dealt w/ firewall issues before...

nex3 commented 8 years ago

To be honest, all I know about proxy issues is that they're the domain of dart:io. Pub doesn't do anything explicit with them at all.

kevmoo commented 8 years ago

@whesse is there anything about the new BoringSSL impl that could cause this?

whesse commented 8 years ago

If 1.12 is also failing in the same way, then the problem is not BoringSSL. I don't really know much about proxies. Someone else should look at this, and ask the right questions to the bug reporter. @sgjesse @mkustermann

This does need to be addressed, so I think the pub team should follow up on this, and find someone who can figure this out.

mkustermann commented 8 years ago

Based on the pub get --trace logs I would assume that dart:ios HttpClient found environment variables which tell which proxy to use and then that proxy requires authentication but no authentication handler was set.

@crodier Could you unset all the environment variables described here to make pub use direct connections and try again? At least that documentation mentions that a username:password@hostname:port value should in theory work.

dart:io's HttpClient does have support for this via the HttpClient.authenticateProxy setter. I assume the pub client doesn't use this functionality so the request fails.

The functionality for finding a proxy or disabling the use of proxies can be configured via HttpClient.findProxy. I also assume that pub doesn't disable proxies but rather let dart:io use it's defaults.

@sgjesse Maybe you have more ideas?

crodier commented 8 years ago

Thanks for looking everyone. I did attach both the with https_proxy environment variable set and unset, as two zip files. I believe it logs the value in the one where set, and not in the other. Same result in both, on the latest 1.15 dev build.

Maven (java) had s similar SSL issue, where adding the iron port certificate to the trust store resolved the problem. Forcing java to trust the iron port certificate. Could be because our iron port cert resolves to a different root authority than java is aware of. That's my next guess, because I agree that if it doesn't work in 1.12, than it should not be BoringSSL related, but something that would fail across dart.io versions.

If this theory is not easily testable, I can try to build and debug into the sdk.

On Tue, Feb 2, 2016, 1:15 PM Martin Kustermann notifications@github.com wrote:

Based on the pub get --trace logs I would assume that dart:ios HttpClient found environment variables which tell which proxy to use and then that proxy requires authentication but no authentication handler was set.

@crodier https://github.com/crodier Could you unset all the environment variables described here https://api.dartlang.org/1.14.0/dart-io/HttpClient/findProxyFromEnvironment.html to make pub use direct connections and try again?

dart:io's HttpClient does have support for this via the HttpClient.authenticateProxy https://api.dartlang.org/1.14.0/dart-io/HttpClient/authenticateProxy.html setter. I assume the pub client doesn't use this functionality so the request fails.

The functionality for finding a proxy or disabling the use of proxies can be configured via HttpClient.findProxy https://api.dartlang.org/1.14.0/dart-io/HttpClient/findProxy.html. I also assume that pub doesn't disable proxies but rather let dart:io use it's defaults.

@sgjesse https://github.com/sgjesse Maybe you have more ideas?

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/sdk/issues/25615#issuecomment-178731066.

sgjesse commented 8 years ago

@crodier Have you tried to run a simple dart:io program accessing an https URL? E.g.

import 'dart:io';

main() async {
  HttpClient client = new HttpClient();
  var request = await client.getUrl(Uri.parse("https://www.google.com/"));
  var response = await request.close();
  print(response.headers);
  client.close(force: true);
}

I assume that will give the same error?

Do you know what type of authentication the proxy uses? dart:io support basic and digest.

Regarding the certificate, does the proxy server require an HTTPS connection? If so we are out of luck, as that is not supported. The proxy configuration we support are PROXY and DIRECT as described in https://en.wikipedia.org/wiki/Proxy_auto-config (the string returned from findProxy use that format (we don't handle PAC files - just use the same string format returned from the PAC JavaScript code) and there is no way of specifying HTTPS.

crodier commented 8 years ago

Thanks. Tried it, this uncovers the actual issie!

Connecting to https google.com, as in the example code.

Error: CERTIFICATE_VERIFY_FAILED.

The proxy is https. It issues certificates, proxied, for everything we access internally, and has been trusted by the root authority.

This puts the remote certificate (pub.dart) in the certificate chain, beneath ours. It is a transparent proxy, web wise.

Our certificate is valid. The connection is tls 1.0. I am looking now for the client code to tell dart http:// client to:

trust our certificate
trust tls 1.0

If you can update the example I can try it. I expect that fixes it. If it does, then a command.line flag would be necessary for pub get to do the same for pub commands.

In maven, this was done by importing our certificate into a trust store, and then telling java to use that trust store. Similar problem, I believe need the same for dart pub command. Either npm ignores cer issues or has a workaround for tls 1.0 amd certificate chains that works for.our Iron Port vendor proxy. This is a huge vendor and typical in finance, captures all internal network activity for storage and review.

On Wed, Feb 3, 2016, 11:59 AM Søren Gjesse notifications@github.com wrote:

@crodier https://github.com/crodier Have you tried to run a simple dart:io program accessing an https URL? E.g.

import 'dart:io';

main() async { HttpClient client = new HttpClient(); var request = await client.getUrl(Uri.parse("https://www.google.com/")); var response = await request.close(); print(response.headers); client.close(force: true); }

I assume that will give the same error?

Do you know what type of authentication the proxy uses? dart:io support basic and digest.

Regarding the certificate, does the proxy server require an HTTPS connection? If so we are out of luck, as that is not supported. The proxy configuration we support are PROXY and DIRECT as described in https://en.wikipedia.org/wiki/Proxy_auto-config (the string returned from findProxy https://api.dartlang.org/1.14.0/dart-io/HttpClient/findProxy.html use that format (we don't handle PAC files - just use the same string format returned from the PAC JavaScript code) and there is no way of specifying HTTPS.

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/sdk/issues/25615#issuecomment-179345550.

crodier commented 8 years ago

I added our iron port PEM to the SecurityContext, and connected to https google.com, in the example.

Now question becomes primarily, how to tell pub get to do the same?

Follow up is why it must be added when we are trusted by a major root authority.

Npm works, as does the web. I don't think the certificate chain is being honored or similar.

On Wed, Feb 3, 2016, 6:08 PM Chris Rodier chrisrodier@gmail.com wrote:

Thanks. Tried it, this uncovers the actual issie!

Connecting to https google.com, as in the example code.

Error: CERTIFICATE_VERIFY_FAILED.

The proxy is https. It issues certificates, proxied, for everything we access internally, and has been trusted by the root authority.

This puts the remote certificate (pub.dart) in the certificate chain, beneath ours. It is a transparent proxy, web wise.

Our certificate is valid. The connection is tls 1.0. I am looking now for the client code to tell dart http:// client to:

trust our certificate

trust tls 1.0

If you can update the example I can try it. I expect that fixes it. If it does, then a command.line flag would be necessary for pub get to do the same for pub commands.

In maven, this was done by importing our certificate into a trust store, and then telling java to use that trust store. Similar problem, I believe need the same for dart pub command. Either npm ignores cer issues or has a workaround for tls 1.0 amd certificate chains that works for.our Iron Port vendor proxy. This is a huge vendor and typical in finance, captures all internal network activity for storage and review.

On Wed, Feb 3, 2016, 11:59 AM Søren Gjesse notifications@github.com wrote:

@crodier https://github.com/crodier Have you tried to run a simple dart:io program accessing an https URL? E.g.

import 'dart:io';

main() async { HttpClient client = new HttpClient(); var request = await client.getUrl(Uri.parse("https://www.google.com/")); var response = await request.close(); print(response.headers); client.close(force: true); }

I assume that will give the same error?

Do you know what type of authentication the proxy uses? dart:io support basic and digest.

Regarding the certificate, does the proxy server require an HTTPS connection? If so we are out of luck, as that is not supported. The proxy configuration we support are PROXY and DIRECT as described in https://en.wikipedia.org/wiki/Proxy_auto-config (the string returned from findProxy https://api.dartlang.org/1.14.0/dart-io/HttpClient/findProxy.html use that format (we don't handle PAC files - just use the same string format returned from the PAC JavaScript code) and there is no way of specifying HTTPS.

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/sdk/issues/25615#issuecomment-179345550.

sgjesse commented 8 years ago

@crodier The default root certificates used by the Dart VM is build-in, and is pulled from a separate repository (https://github.com/dart-lang/root_certificates/blob/master/certdata.pem), and you CA depend on is obviously not in that list.

Regarding pub there are currently no (afaik @munificent or @nex3 correct me if I am wrong) options to specify root certificates or to allow untrusted certificates. So changes will be needed to either pub or dart:io.

There is work in progress on using the root certificates installed on the machine instead of the build-in set, but it will take some time before that lands.

crodier commented 8 years ago

Thanks for helping me!

I agree. I need to say I think both are necessary, and will br prohibitive to any would be corporate Dart users, or many. Maybe enough to warrant priority.

I will look into who our root authority is. How do I check that?

On Thu, Feb 4, 2016, 10:28 AM Søren Gjesse notifications@github.com wrote:

@crodier https://github.com/crodier The default root certificates used by the Dart VM is build-in, and is pulled from a separate repository ( https://github.com/dart-lang/root_certificates/blob/master/certdata.pem), and you CA depend on is obviously not in that list.

Regarding pub there are currently no (afaik @munificent https://github.com/munificent or @nex3 https://github.com/nex3 correct me if I am wrong) options to specify root certificates or to allow untrusted certificates. So changes will be needed to either pub or dart:io .

There is work in progress on using the root certificates installed on the machine instead of the build-in set, but it will take some time before that lands.

— Reply to this email directly or view it on GitHub https://github.com/dart-lang/sdk/issues/25615#issuecomment-179898595.

sgjesse commented 8 years ago

In Chrome you can right-click on the green lock icon for the HTTPS connection, then click on the "Connection" tab and click "Certificate information". The click the "Details" tab which should have the "Certificate Hierarchy" at the top.

andregs commented 6 years ago

On npm we can set strict-ssl false (on git http.sslVerify false) to disable SSL certificate verification. Something like that would be helpful in pub for situations like this one, right?

mkustermann commented 6 years ago

@zanderso / @nex3

This doesn't seem to be a dart:io issue, since dart:io provides mechanisms for the Dart programmer to override which certificates are trusted via the SecurityContext and HttpClient.badCertificateCallback.

This seems to be more of a feature request for Pub, namely to configure pub get to accept untrusted server certificates. Maybe we should move this to the Pub issue tracker?

dnfield commented 6 years ago

I just opened https://github.com/dart-lang/pub/issues/1882 because I'm facing a similar issue with pub (no proxy, but self signed CA).

I'd really like to see the ability to tell Dart to use a self signed CA, but at the least it'd be nice to be able to tell pub to accept one. However, this issue would be an issue with any Dart program I try to run on my machine that wants to do an HTTPS connection and has no way of knowing that I have a self-signed certificate I want it to trust. This will be a major issue for corporate users behind BlueCoat etc.

natebosch commented 6 years ago

@mkustermann - it's dart:io which is honoring https_proxy, it seems like it could also be a VM level flag to handle certificate overrides etc - why do we think it should be the responsibility of every Dart script that wants to make an https request to handle SecurityContext?

zanderso commented 6 years ago

The command line VM has options for overriding the source of trusted root certificates. See: https://github.com/dart-lang/sdk/blob/master/runtime/bin/main_options.cc#L207

natebosch commented 6 years ago

Sounds good. Should be able to use this with DART_VM_OPTIONS=--root-certs-file=some_file pub get

@crodier - if this is still an issue for you please try with DART_VM_OPTIONS and let us know if that doesn't work for you.

Since this issue is so old I'll close for now and we can reopen if we get a report that this doesn't work.

stephenrooban commented 5 years ago

Sounds good. Should be able to use this with DART_VM_OPTIONS=--root-certs-file=some_file pub get

@crodier - if this is still an issue for you please try with DART_VM_OPTIONS and let us know if that doesn't work for you.

Since this issue is so old I'll close for now and we can reopen if we get a report that this doesn't work.

Hi,

can we pass this argument along with flutter doctor ?

FlareDev0 commented 5 years ago

Hi Natebosch, Can you please explain how to use DART_VM_OPTIONS=--root-certs-file=some_file pub get?

yathit commented 5 years ago

Android have option to use user cert store. I have included <certificates src="user" /> to trust user added CAs, but it does not work in Flutter app.

zanderso commented 5 years ago

@yathit Could you file a new issue for that?

Aaron009 commented 4 years ago

@crodier Have you tried to run a simple dart:io program accessing an https URL? E.g.
import 'dart:io';

main() async {
  HttpClient client = new HttpClient();
  var request = await client.getUrl(Uri.parse("https://www.google.com/"));
  var response = await request.close();
  print(response.headers);
  client.close(force: true);
}
I assume that will give the same error?

Do you know what type of authentication the proxy uses? dart:io support basic and digest.

Regarding the certificate, does the proxy server require an HTTPS connection? If so we are out of luck, as that is not supported. The proxy configuration we support are PROXY and DIRECT as described in https://en.wikipedia.org/wiki/Proxy_auto-config (the string returned from findProxy use that format (we don't handle PAC files - just use the same string format returned from the PAC JavaScript code) and there is no way of specifying HTTPS.

I tested it and I replaced the link with www.baidu.com without any issues, but replaced the following link and reported a HandshakeException: Handshake error in client (OS Error: CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:352))

import 'dart:io';

main() async {
  HttpClient client = new HttpClient();
  var request = await client.getUrl(Uri.parse('https://shop.io.mi-img.com/app/shop/img?id=shop_88f929c5731967cbc8339cfae1f5f0ec.jpeg'));
  var response = await request.close();
  print(response.headers);
  client.close(force: true);
}

I'm sure no proxy is set。

dart-lang / sdk

Iron Port TLS 1.0 proxy, pub get fails #25615