search

libwww-perl / LWP-Protocol-https

Provide https support for LWP::UserAgent
https://metacpan.org/pod/LWP::Protocol::https
Other
16 stars 35 forks source link

LWP::Protocol::https::Socket: SSL connect attempt failed error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/local/share/perl/5.18.2/LWP/Protocol/http.pm line 47 #20

Closed ne0ark closed 7 years ago

ne0ark commented 9 years ago

Log: https://gist.github.com/ne0ark/b890a48adb113def7558

noxxi commented 9 years ago

Please make sure that you are not behind some SSL intercepting middlebox or firewall, which is often used in companies. The certificate for this middlebox is not included in the list of trusted CA's from Mozilla::CA, so the verification will fail.

DrHyde commented 9 years ago

I get similar problems, and there are no proxies involved. I think it's because Apache are using some shiny new extension to SSL that my openssl libraries don't support.

noxxi commented 9 years ago

"Shiny new" might be that the server requires Server Name Indication and fails without. Do you still use OpenSSL 0.9.8 or lower?

DrHyde commented 9 years ago

yeah

flocsy commented 9 years ago

I also get this error when checking ssl certificate of monitoring.us-east-1.amazonaws.com:443 dpkg -l openssl => 1.0.1f

noxxi commented 9 years ago

I also get this error when checking ssl certificate of monitoring.us-east-1.amazonaws.com:443 dpkg -l openssl => 1.0.1f

I don't get this error on Ubuntu 14.04 with native LWP so something is different between your and my setup. It might help if you would add enough details about your setup so that the problem can be reproduced, i.e. version of perl, LWP, LWP::Protocol::https, Mozilla::CA your operating system and if you use the LWP shipped with the OS or not (at least Debian/Ubuntu have their own patches for validation).

brynbstn7 commented 8 years ago

-- same error--

I'm using: perl: v5.10.1 (just re-installed lwp::protocol::https) linux: Red Hat Enterprise Linux Server release 6.4 (Santiago) apache: Apache/2.2.15 (Unix)

Any successes resolving this one?

noxxi commented 8 years ago

This can be a problem of the certificate or of the CA on the system. It can be related to the (unknown) version of OpenSSL you are using or it can has something to do with SNI. Without any information about the URL you've tried to access I can not look further into it.

brynbstn7 commented 8 years ago

OpenSSL 1.0.1e-fips 11 Feb 2013

The host site requires TLS1.2, and something > RC4; we have confirmation that we are using TLS1.2 and a newer cipher

What is SNI, and what are the commands for getting the info?

If you want to work offline I can send you some of the more sensitve info like the URL. It requires authentication. I don't feel comfortable with running that perl command to get your email ...

thanks

noxxi commented 8 years ago

see http://noxxi.de/ for contact information.

akirpa commented 7 years ago

Hi all!

For solving this issue please use solution provided at https://rt.cpan.org/Public/Bug/Display.html?id=112145

Problem exists on webservers with multiple ssl servers, for instance, https://www.zonemaster.net/ and with some other servers.

Test with and without proxy and lwp-request: No patch setenv https_proxy=http://1.2.3.4:3128 lwp-request "https://www.zonemaster.net/" - not work (with proxy) and report SSL upgrade failed: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed at /usr/local/lib/perl5/site_perl/LWP/Protocol/http.pm line 205.

unsetenv https_proxy lwp-request "https://www.zonemaster.net/" - work without proxy

After apply next patch

vvvvv
--- https.pm.orig 2014-04-18 19:33:26.000000000 +0300
+++ https.pm 2017-02-19 03:51:54.885346000 +0200
@@ -153,6 +153,7 @@
  my ($self,$sock,$url) = @_;
  $sock = LWP::Protocol::https::Socket->start_SSL( $sock,
      SSL_verifycn_name => $url->host,
+     SSL_hostname => $url->host,
      $self->_extra_sock_opts,
  );
  $@ = LWP::Protocol::https::Socket->errstr if ! $sock;

^^^^

lwp-request "https://www.zonemaster.net/" - work with and without proxy (squid-3.5.24)

For additional verify I test with and without proxy next sites: lwp-request "https://www.google.com/" lwp-request "https://rt.cpan.org/Public/Bug/Display.html?id=112145" lwp-request "https://translate.google.com/?hl=en&tab=TT" lwp-request "https://validator.w3.org/" lwp-request "https://www.merriam-webster.com/" lwp-request "https://en.wikipedia.org/wiki/Main_Page" lwp-request "https://rt.perl.org/Public/" lwp-request "https://ecosystem.atlassian.net/" lwp-request "https://github.com" lwp-request "https://metacpan.org/"

problem is not detected.

Best regards, Alexander Kirpa

noxxi commented 7 years ago

It is unknown if this is really the original problem behind this issue because not enough information were provided. But if this is the underlying problem then it was already multiple times reported and apart from the bug report from 2016 there is the pull request from 2015 https://github.com/libwww-perl/LWP-Protocol-https/pull/24 with a fix and another one from 2014 https://github.com/libwww-perl/LWP-Protocol-https/pull/17 with a better fix.

akirpa commented 7 years ago

Dear Steffen!

My software based perl LWP used for verifying of content of webpages on some websites. After applying of patch my software retrieve approx 12K pages from 6 websites without problems.

If research tcpdump files with and without patch we see - patch provide during "SSL upgrade" sent to webserver additioan field: Extension: server_name Type: server_name (0x0000) Server Name Indication extension Server Name Type: host_name (0) Server Name: www.zonemaster.net

Web servers that use single key/cert pair for ALL websites possible simple ignore this field or can use it w/o problems. For webservers that use separate key/cert pairs for each website only this patch (or similar) can use "host_name" field and correctly determine required key/cert pair for setup correct SSL connection.

Regarding pull #17 - poor solution: verify_hostname => 0 - is not applicable setting as minimum in common case.

Regarding pull #24 - correct and more flexible and worked solution, but this pull still open.

IMHO need close pull #24. Current edition of LWP-Protocol-https is 6.06 from 18 Apr 2014 - too many time await commit pull #24 for new 6.07 (for example) edition.

I apply patch (pull #24) on my unix servers as minimum until it will be a new version that work correctly.

Best regards, Alexander Kirpa

noxxi commented 7 years ago

Regarding pull #17 - poor solution: verify_hostname => 0 - is not applicable setting as minimum in common case.

I'm not sure why you refer to verify_hostname => 0 since #17 does not include any such change (and it would be a very bad idea to do this by default). The only difference between #24 and #17 is that #17 sets SSL_hostname additionally to SSL_verify_name and #24 sets SSL_hostname instead of SSL_verify_name. I've explained in https://github.com/libwww-perl/LWP-Protocol-https/pull/24#issuecomment-280901483 why the approach of #17 should be the preferred one.

akirpa commented 7 years ago

17 vs #24?

  1. 24 more short

  2. 24 still verify cert of web site

  3. 24 still correct work regardless direct or over proxy

    please try retrive https://0-it.com/ <- additional server at https://github.com with incorrect cert: setenv https_proxy..... lwp-request https://0-it.com/ SSL upgrade failed: hostname verification failed at /usr/local/lib/perl5/site_perl/LWP/Protocol/http.pm line 205. unsetenv https_proxy lwp-request https://0-it.com/ Can't connect to 0-it.com:443

hostname verification failed at /usr/local/lib/perl5/site_perl/LWP/Protocol/http.pm line 46.

Both reply correct

  1. 17 more long and do not provide any additional features.

    If write in https.pm next code: SSL_verifycn_name => $url->host, SSL_hostname => $url->host, SSL_verifycn_name => 'github.com', $self->_extra_sock_opts, and try lwp-request https://0-it.com/ Content of https://0-it.com/ retrieved w/o problem, we do not receive any error. In other words #17 do not prevent change value of 'SSL_verifycn_name'

Regarding all above I prefer #24.

Best regards, Alexander Kirpa

genio commented 7 years ago

Hi @akirpa,

I agree with @noxxi that #17 is preferable to #24 in that it does the right thing. It sets both SSL_hostname and SSL_verifycn_name; it does not in any way alter verify_hostname.

A release containing #17 has already been made and I'm going to close this issue out now since the release seems to have corrected things. However, please don't hesitate to open another issue if you can show us where something is misbehaving.

Thanks, Chase