search

benoitc / hackney

simple HTTP client in Erlang
Other
1.34k stars 427 forks source link

bad_cert error #624

Closed benbro closed 4 years ago

benbro commented 4 years ago

I'm getting a TLS error when sending a request hackney master from today OTP 23.

I'll try to get more info

hackney:request(get, Url, Headers, <<>>, [{with_body, true}, {recv_timeout, 5000}]).

TLS client: In state certify at /ssl/src/ssl_handshake.erl: 1847 generated CLIENT ALERT: Fatal - Handshake Failure - {bad_cert,hostname_check_failed}
yjh0502 commented 4 years ago

Here's a code for reproduction

hackney:get(<<"https://contacts.google.com/">>).

Trace dump. It seems that it fails to handle wildcard certificates.

(foo@127.0.0.1)46> hackney:get(<<"https://contacts.google.com/">>).
{error,{tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}
}
(<0.517.0>) call public_key:pkix_verify_hostname({'OTPCertificate',
    {'OTPTBSCertificate',v3,321731345735978692271199214973930037793,
        {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
        {rdnSequence,
            [[{'AttributeTypeAndValue',{2,5,4,6},"US"}],
             [{'AttributeTypeAndValue',
                  {2,5,4,10},
                  {printableString,"Google Trust Services"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,3},
                  {printableString,"GTS CA 1O1"}}]]},
        {'Validity',{utcTime,"200428074341Z"},{utcTime,"200721074341Z"}},
        {rdnSequence,
            [[{'AttributeTypeAndValue',{2,5,4,6},"US"}],
             [{'AttributeTypeAndValue',
                  {2,5,4,8},
                  {printableString,"California"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,7},
                  {printableString,"Mountain View"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,10},
                  {printableString,"Google LLC"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,3},
                  {utf8String,<<"*.google.com">>}}]]},
        {'OTPSubjectPublicKeyInfo',
            {'PublicKeyAlgorithm',
                {1,2,840,10045,2,1},
                {namedCurve,{1,2,840,10045,3,1,7}}},
            {'ECPoint',
                <<4,15,69,78,47,12,167,136,154,185,36,255,87,80,220,241,171,
                  110,221,62,127,130,38,48,167,18,159,129,138,39,157,125,6,
                  46,211,226,80,59,206,108,45,46,91,50,206,125,235,134,6,124,
                  140,41,43,71,97,222,240,202,248,183,152,0,33,106,52>>}},
        asn1_NOVALUE,asn1_NOVALUE,
        [{'Extension',{2,5,29,15},true,[digitalSignature]},
         {'Extension',{2,5,29,37},false,[{1,3,6,1,5,5,7,3,1}]},
         {'Extension',
             {2,5,29,19},
             true,
             {'BasicConstraints',false,asn1_NOVALUE}},
         {'Extension',
             {2,5,29,14},
             false,
             <<160,137,132,208,190,181,43,113,58,250,115,5,221,228,44,197,138,
               218,71,239>>},
         {'Extension',
             {2,5,29,35},
             false,
             {'AuthorityKeyIdentifier',
                 <<152,209,248,110,16,235,207,155,236,96,159,24,144,27,160,235,
                   125,9,253,43>>,
                 asn1_NOVALUE,asn1_NOVALUE}},
         {'Extension',
             {1,3,6,1,5,5,7,1,1},
             false,
             [{'AccessDescription',
                  {1,3,6,1,5,5,7,48,1},
                  {uniformResourceIdentifier,"http://ocsp.pki.goog/gts1o1"}},
              {'AccessDescription',
                  {1,3,6,1,5,5,7,48,2},
                  {uniformResourceIdentifier,
                      "http://pki.goog/gsr2/GTS1O1.crt"}}]},
         {'Extension',
             {2,5,29,17},
             false,
             [{dNSName,"*.google.com"},
              {dNSName,"*.android.com"},
              {dNSName,"*.appengine.google.com"},
              {dNSName,"*.bdn.dev"},
              {dNSName,"*.cloud.google.com"},
              {dNSName,"*.crowdsource.google.com"},
              {dNSName,"*.g.co"},
              {dNSName,"*.gcp.gvt2.com"},
              {dNSName,"*.gcpcdn.gvt1.com"},
              {dNSName,"*.ggpht.cn"},
              {dNSName,"*.gkecnapps.cn"},
              {dNSName,"*.google-analytics.com"},
              {dNSName,"*.google.ca"},
              {dNSName,"*.google.cl"},
              {dNSName,"*.google.co.in"},
              {dNSName,"*.google.co.jp"},
              {dNSName,"*.google.co.uk"},
              {dNSName,"*.google.com.ar"},
              {dNSName,"*.google.com.au"},
              {dNSName,"*.google.com.br"},
              {dNSName,"*.google.com.co"},
              {dNSName,"*.google.com.mx"},
              {dNSName,"*.google.com.tr"},
              {dNSName,"*.google.com.vn"},
              {dNSName,"*.google.de"},
              {dNSName,"*.google.es"},
              {dNSName,"*.google.fr"},
              {dNSName,"*.google.hu"},
              {dNSName,"*.google.it"},
              {dNSName,"*.google.nl"},
              {dNSName,"*.google.pl"},
              {dNSName,"*.google.pt"},
              {dNSName,"*.googleadapis.com"},
              {dNSName,"*.googleapis.cn"},
              {dNSName,"*.googlecnapps.cn"},
              {dNSName,"*.googlecommerce.com"},
              {dNSName,"*.googlevideo.com"},
              {dNSName,"*.gstatic.cn"},
              {dNSName,"*.gstatic.com"},
              {dNSName,"*.gstaticcnapps.cn"},
              {dNSName,"*.gvt1.com"},
              {dNSName,"*.gvt2.com"},
              {dNSName,"*.metric.gstatic.com"},
              {dNSName,"*.urchin.com"},
              {dNSName,"*.url.google.com"},
              {dNSName,"*.wear.gkecnapps.cn"},
              {dNSName,"*.youtube-nocookie.com"},
              {dNSName,"*.youtube.com"},
              {dNSName,"*.youtubeeducation.com"},
              {dNSName,"*.youtubekids.com"},
              {dNSName,"*.yt.be"},
              {dNSName,"*.ytimg.com"},
              {dNSName,"android.clients.google.com"},
              {dNSName,"android.com"},
              {dNSName,"developer.android.google.cn"},
              {dNSName,"developers.android.google.cn"},
              {dNSName,"g.co"},
              {dNSName,"ggpht.cn"},
              {dNSName,"gkecnapps.cn"},
              {dNSName,"goo.gl"},
              {dNSName,"google-analytics.com"},
              {dNSName,"google.com"},
              {dNSName,"googlecnapps.cn"},
              {dNSName,"googlecommerce.com"},
              {dNSName,"source.android.google.cn"},
              {dNSName,"urchin.com"},
              {dNSName,"www.goo.gl"},
              {dNSName,"youtu.be"},
              {dNSName,"youtube.com"},
              {dNSName,"youtubeeducation.com"},
              {dNSName,"youtubekids.com"},
              {dNSName,"yt.be"}]},
         {'Extension',
             {2,5,29,32},
             false,
             [{'PolicyInformation',{2,23,140,1,2,2},asn1_NOVALUE},
              {'PolicyInformation',{1,3,6,1,4,1,11129,2,5,3},asn1_NOVALUE}]},
         {'Extension',
             {2,5,29,31},
             false,
             [{'DistributionPoint',
                  {fullName,
                      [{uniformResourceIdentifier,
                           "http://crl.pki.goog/GTS1O1.crl"}]},
                  asn1_NOVALUE,asn1_NOVALUE}]},
         {'Extension',
             {1,3,6,1,4,1,11129,2,4,2},
             false,
             <<4,129,243,0,241,0,118,0,178,30,5,204,139,162,205,138,32,78,
               135,102,249,43,185,138,37,32,103,107,218,250,112,231,178,73,
               83,45,239,139,144,94,0,0,1,113,191,244,205,188,0,0,4,3,0,71,
               48,69,2,33,0,144,114,30,77,37,127,69,205,184,10,100,30,228,
               150,174,69,22,71,220,235,113,91,89,63,7,125,229,72,183,122,
               218,124,2,32,5,151,41,191,70,54,82,202,241,181,54,210,20,108,
               91,75,222,248,210,48,89,171,18,59,229,208,49,19,129,131,27,
               76,0,119,0,94,167,115,249,223,86,192,231,181,54,72,125,208,
               73,224,50,122,145,154,12,132,161,18,18,132,24,117,150,129,
               113,69,88,0,0,1,113,191,244,205,218,0,0,4,3,0,72,48,70,2,33,
               0,145,145,9,20,141,220,37,184,27,211,90,152,124,246,95,54,
               217,148,216,254,36,210,237,219,45,52,97,97,169,151,10,29,2,
               33,0,245,74,82,1,217,44,128,116,111,176,51,180,159,54,181,58,
               6,53,137,39,177,213,26,158,222,115,161,210,239,9,101,74>>}]},
    {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
    <<103,178,222,42,157,134,197,197,19,8,52,58,217,182,180,138,149,239,238,
      179,148,196,63,244,31,249,12,164,204,78,45,164,41,185,236,175,70,211,26,
      215,126,97,181,10,59,114,254,3,90,78,130,222,154,62,197,72,8,33,44,231,
      12,12,112,82,18,211,71,198,115,200,122,223,103,103,82,144,138,8,185,101,
      252,235,36,123,127,209,39,164,117,174,26,35,177,156,129,164,204,76,62,
      138,233,8,33,91,181,116,0,12,250,81,59,52,128,194,233,169,191,193,213,
      166,132,31,25,46,187,125,77,146,246,205,205,37,192,171,112,5,81,82,20,
      99,237,56,164,190,51,157,38,241,43,118,78,150,102,249,209,42,229,154,63,
      179,37,62,170,32,114,28,19,120,179,33,219,178,14,198,97,66,28,30,158,
      214,243,60,20,82,25,115,117,115,170,82,19,38,243,206,29,255,74,24,244,
      31,34,21,32,58,90,150,1,253,243,69,134,47,189,16,79,129,251,0,79,109,20,
      138,62,61,136,231,124,171,62,185,146,200,198,46,229,194,8,176,205,177,
      15,97,73,61,239,39,80,104,78,40,143,69,239,88,244,172>>},[{dns_id,"contacts.google.com"}],[]) ({ssl_certificate,
                                                                                                      verify_hostname,
                                                                                                      4})
=NOTICE REPORT==== 16-May-2020::10:05:16.721529 ===
TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure
 - {bad_cert,hostname_check_failed}
(<0.517.0>) returned from public_key:pkix_verify_hostname/3 -> false
(foo@127.0.0.1)47>
benoitc commented 4 years ago

thanks. I'm working on it :)

On Sat 16 May 2020 at 12:08 Jihyun Yu notifications@github.com wrote:

Here's a code for reproduction

hackney:get(<<"https://contacts.google.com/">>).

Trace dump. It seems that it fails to handle wildcard certificates.

(foo@127.0.0.1)46> hackney:get(<<"https://contacts.google.com/">>). {error,{tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}} } (<0.517.0>) call public_key:pkix_verify_hostname({'OTPCertificate', {'OTPTBSCertificate',v3,321731345735978692271199214973930037793, {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'}, {rdnSequence, [[{'AttributeTypeAndValue',{2,5,4,6},"US"}], [{'AttributeTypeAndValue', {2,5,4,10}, {printableString,"Google Trust Services"}}], [{'AttributeTypeAndValue', {2,5,4,3}, {printableString,"GTS CA 1O1"}}]]}, {'Validity',{utcTime,"200428074341Z"},{utcTime,"200721074341Z"}}, {rdnSequence, [[{'AttributeTypeAndValue',{2,5,4,6},"US"}], [{'AttributeTypeAndValue', {2,5,4,8}, {printableString,"California"}}], [{'AttributeTypeAndValue', {2,5,4,7}, {printableString,"Mountain View"}}], [{'AttributeTypeAndValue', {2,5,4,10}, {printableString,"Google LLC"}}], [{'AttributeTypeAndValue', {2,5,4,3}, {utf8String,<<".google.com">>}}]]}, {'OTPSubjectPublicKeyInfo', {'PublicKeyAlgorithm', {1,2,840,10045,2,1}, {namedCurve,{1,2,840,10045,3,1,7}}}, {'ECPoint', <<4,15,69,78,47,12,167,136,154,185,36,255,87,80,220,241,171, 110,221,62,127,130,38,48,167,18,159,129,138,39,157,125,6, 46,211,226,80,59,206,108,45,46,91,50,206,125,235,134,6,124, 140,41,43,71,97,222,240,202,248,183,152,0,33,106,52>>}}, asn1_NOVALUE,asn1_NOVALUE, [{'Extension',{2,5,29,15},true,[digitalSignature]}, {'Extension',{2,5,29,37},false,[{1,3,6,1,5,5,7,3,1}]}, {'Extension', {2,5,29,19}, true, {'BasicConstraints',false,asn1_NOVALUE}}, {'Extension', {2,5,29,14}, false, <<160,137,132,208,190,181,43,113,58,250,115,5,221,228,44,197,138, 218,71,239>>}, {'Extension', {2,5,29,35}, false, {'AuthorityKeyIdentifier', <<152,209,248,110,16,235,207,155,236,96,159,24,144,27,160,235, 125,9,253,43>>, asn1_NOVALUE,asn1_NOVALUE}}, {'Extension', {1,3,6,1,5,5,7,1,1}, false, [{'AccessDescription', {1,3,6,1,5,5,7,48,1}, {uniformResourceIdentifier,"http://ocsp.pki.goog/gts1o1"}}, {'AccessDescription', {1,3,6,1,5,5,7,48,2}, {uniformResourceIdentifier, "http://pki.goog/gsr2/GTS1O1.crt"}}]}, {'Extension', {2,5,29,17}, false, [{dNSName,".google.com"}, {dNSName,".android.com"}, {dNSName,".appengine.google.com"}, {dNSName,".bdn.dev"}, {dNSName,".cloud.google.com"}, {dNSName,".crowdsource.google.com"}, {dNSName,".g.co"}, {dNSName,".gcp.gvt2.com"}, {dNSName,".gcpcdn.gvt1.com"}, {dNSName,".ggpht.cn"}, {dNSName,".gkecnapps.cn"}, {dNSName,".google-analytics.com"}, {dNSName,".google.ca"}, {dNSName,".google.cl"}, {dNSName,".google.co.in"}, {dNSName,".google.co.jp"}, {dNSName,".google.co.uk"}, {dNSName,".google.com.ar"}, {dNSName,".google.com.au"}, {dNSName,".google.com.br"}, {dNSName,".google.com.co"}, {dNSName,".google.com.mx"}, {dNSName,".google.com.tr"}, {dNSName,".google.com.vn"}, {dNSName,".google.de"}, {dNSName,".google.es"}, {dNSName,".google.fr"}, {dNSName,".google.hu"}, {dNSName,".google.it"}, {dNSName,".google.nl"}, {dNSName,".google.pl"}, {dNSName,".google.pt"}, {dNSName,".googleadapis.com"}, {dNSName,".googleapis.cn"}, {dNSName,".googlecnapps.cn"}, {dNSName,".googlecommerce.com"}, {dNSName,".googlevideo.com"}, {dNSName,".gstatic.cn"}, {dNSName,".gstatic.com"}, {dNSName,".gstaticcnapps.cn"}, {dNSName,".gvt1.com"}, {dNSName,".gvt2.com"}, {dNSName,".metric.gstatic.com"}, {dNSName,".urchin.com"}, {dNSName,".url.google.com"}, {dNSName,".wear.gkecnapps.cn"}, {dNSName,".youtube-nocookie.com"}, {dNSName,".youtube.com"}, {dNSName,".youtubeeducation.com"}, {dNSName,".youtubekids.com"}, {dNSName,".yt.be"}, {dNSName,"*.ytimg.com"}, {dNSName,"android.clients.google.com"}, {dNSName,"android.com"}, {dNSName,"developer.android.google.cn"}, {dNSName,"developers.android.google.cn"}, {dNSName,"g.co"}, {dNSName,"ggpht.cn"}, {dNSName,"gkecnapps.cn"}, {dNSName,"goo.gl"}, {dNSName,"google-analytics.com"}, {dNSName,"google.com"}, {dNSName,"googlecnapps.cn"}, {dNSName,"googlecommerce.com"}, {dNSName,"source.android.google.cn"}, {dNSName,"urchin.com"}, {dNSName,"www.goo.gl"}, {dNSName,"youtu.be"}, {dNSName,"youtube.com"}, {dNSName,"youtubeeducation.com"}, {dNSName,"youtubekids.com"}, {dNSName,"yt.be"}]}, {'Extension', {2,5,29,32}, false, [{'PolicyInformation',{2,23,140,1,2,2},asn1_NOVALUE}, {'PolicyInformation',{1,3,6,1,4,1,11129,2,5,3},asn1_NOVALUE}]}, {'Extension', {2,5,29,31}, false, [{'DistributionPoint', {fullName, [{uniformResourceIdentifier, "http://crl.pki.goog/GTS1O1.crl"}]}, asn1_NOVALUE,asn1_NOVALUE}]}, {'Extension', {1,3,6,1,4,1,11129,2,4,2}, false, <<4,129,243,0,241,0,118,0,178,30,5,204,139,162,205,138,32,78, 135,102,249,43,185,138,37,32,103,107,218,250,112,231,178,73, 83,45,239,139,144,94,0,0,1,113,191,244,205,188,0,0,4,3,0,71, 48,69,2,33,0,144,114,30,77,37,127,69,205,184,10,100,30,228, 150,174,69,22,71,220,235,113,91,89,63,7,125,229,72,183,122, 218,124,2,32,5,151,41,191,70,54,82,202,241,181,54,210,20,108, 91,75,222,248,210,48,89,171,18,59,229,208,49,19,129,131,27, 76,0,119,0,94,167,115,249,223,86,192,231,181,54,72,125,208, 73,224,50,122,145,154,12,132,161,18,18,132,24,117,150,129, 113,69,88,0,0,1,113,191,244,205,218,0,0,4,3,0,72,48,70,2,33, 0,145,145,9,20,141,220,37,184,27,211,90,152,124,246,95,54, 217,148,216,254,36,210,237,219,45,52,97,97,169,151,10,29,2, 33,0,245,74,82,1,217,44,128,116,111,176,51,180,159,54,181,58, 6,53,137,39,177,213,26,158,222,115,161,210,239,9,101,74>>}]}, {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'}, <<103,178,222,42,157,134,197,197,19,8,52,58,217,182,180,138,149,239,238, 179,148,196,63,244,31,249,12,164,204,78,45,164,41,185,236,175,70,211,26, 215,126,97,181,10,59,114,254,3,90,78,130,222,154,62,197,72,8,33,44,231, 12,12,112,82,18,211,71,198,115,200,122,223,103,103,82,144,138,8,185,101, 252,235,36,123,127,209,39,164,117,174,26,35,177,156,129,164,204,76,62, 138,233,8,33,91,181,116,0,12,250,81,59,52,128,194,233,169,191,193,213, 166,132,31,25,46,187,125,77,146,246,205,205,37,192,171,112,5,81,82,20, 99,237,56,164,190,51,157,38,241,43,118,78,150,102,249,209,42,229,154,63, 179,37,62,170,32,114,28,19,120,179,33,219,178,14,198,97,66,28,30,158, 214,243,60,20,82,25,115,117,115,170,82,19,38,243,206,29,255,74,24,244, 31,34,21,32,58,90,150,1,253,243,69,134,47,189,16,79,129,251,0,79,109,20, 138,62,61,136,231,124,171,62,185,146,200,198,46,229,194,8,176,205,177, 15,97,73,61,239,39,80,104,78,40,143,69,239,88,244,172>>},[{dns_id,"contacts.google.com"}],[]) ({ssl_certificate, verify_hostname, 4}) =NOTICE REPORT==== 16-May-2020::10:05:16.721529 === TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure

  • {bad_cert,hostname_check_failed} (<0.517.0>) returned from public_key:pkix_verify_hostname/3 -> false (foo@127.0.0.1)47>

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benoitc/hackney/issues/624#issuecomment-629621595, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAADRIUGAWRW66GYDLLRZULRRZQ3FANCNFSM4NC2A63A .

-- Sent from my Mobile

yjh0502 commented 4 years ago

It works when server_name_indication is specified, for example

hackney:request(get, <<"https://contacts.google.com/">>, [], <<>>, [{ssl_options, [{server_name_indication, "contacts.google.com"}]}]).
benbro commented 4 years ago

It works probably just because you override ssl_options. Try with "yahoo.com" and it will also work.

benoitc commented 4 years ago

there maybe a conflict with the validation in otp 23. I'm debugging....

On Sat 16 May 2020 at 15:24 benbro notifications@github.com wrote:

It works probably just because you override ssl_options. Try with " yahoo.com" and it will also work.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/benoitc/hackney/issues/624#issuecomment-629645178, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAADRITS6SVLQXZY237QYNLRR2HX7ANCNFSM4NC2A63A .

-- Sent from my Mobile

benoitc commented 4 years ago

Here's a code for reproduction

hackney:get(<<"https://contacts.google.com/">>).

Trace dump. It seems that it fails to handle wildcard certificates.

(foo@127.0.0.1)46> hackney:get(<<"https://contacts.google.com/">>).
{error,{tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}
}
(<0.517.0>) call public_key:pkix_verify_hostname({'OTPCertificate',
    {'OTPTBSCertificate',v3,321731345735978692271199214973930037793,
        {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
        {rdnSequence,
            [[{'AttributeTypeAndValue',{2,5,4,6},"US"}],
             [{'AttributeTypeAndValue',
                  {2,5,4,10},
                  {printableString,"Google Trust Services"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,3},
                  {printableString,"GTS CA 1O1"}}]]},
        {'Validity',{utcTime,"200428074341Z"},{utcTime,"200721074341Z"}},
        {rdnSequence,
            [[{'AttributeTypeAndValue',{2,5,4,6},"US"}],
             [{'AttributeTypeAndValue',
                  {2,5,4,8},
                  {printableString,"California"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,7},
                  {printableString,"Mountain View"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,10},
                  {printableString,"Google LLC"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,3},
                  {utf8String,<<"*.google.com">>}}]]},
        {'OTPSubjectPublicKeyInfo',
            {'PublicKeyAlgorithm',
                {1,2,840,10045,2,1},
                {namedCurve,{1,2,840,10045,3,1,7}}},
            {'ECPoint',
                <<4,15,69,78,47,12,167,136,154,185,36,255,87,80,220,241,171,
                  110,221,62,127,130,38,48,167,18,159,129,138,39,157,125,6,
                  46,211,226,80,59,206,108,45,46,91,50,206,125,235,134,6,124,
                  140,41,43,71,97,222,240,202,248,183,152,0,33,106,52>>}},
        asn1_NOVALUE,asn1_NOVALUE,
        [{'Extension',{2,5,29,15},true,[digitalSignature]},
         {'Extension',{2,5,29,37},false,[{1,3,6,1,5,5,7,3,1}]},
         {'Extension',
             {2,5,29,19},
             true,
             {'BasicConstraints',false,asn1_NOVALUE}},
         {'Extension',
             {2,5,29,14},
             false,
             <<160,137,132,208,190,181,43,113,58,250,115,5,221,228,44,197,138,
               218,71,239>>},
         {'Extension',
             {2,5,29,35},
             false,
             {'AuthorityKeyIdentifier',
                 <<152,209,248,110,16,235,207,155,236,96,159,24,144,27,160,235,
                   125,9,253,43>>,
                 asn1_NOVALUE,asn1_NOVALUE}},
         {'Extension',
             {1,3,6,1,5,5,7,1,1},
             false,
             [{'AccessDescription',
                  {1,3,6,1,5,5,7,48,1},
                  {uniformResourceIdentifier,"http://ocsp.pki.goog/gts1o1"}},
              {'AccessDescription',
                  {1,3,6,1,5,5,7,48,2},
                  {uniformResourceIdentifier,
                      "http://pki.goog/gsr2/GTS1O1.crt"}}]},
         {'Extension',
             {2,5,29,17},
             false,
             [{dNSName,"*.google.com"},
              {dNSName,"*.android.com"},
              {dNSName,"*.appengine.google.com"},
              {dNSName,"*.bdn.dev"},
              {dNSName,"*.cloud.google.com"},
              {dNSName,"*.crowdsource.google.com"},
              {dNSName,"*.g.co"},
              {dNSName,"*.gcp.gvt2.com"},
              {dNSName,"*.gcpcdn.gvt1.com"},
              {dNSName,"*.ggpht.cn"},
              {dNSName,"*.gkecnapps.cn"},
              {dNSName,"*.google-analytics.com"},
              {dNSName,"*.google.ca"},
              {dNSName,"*.google.cl"},
              {dNSName,"*.google.co.in"},
              {dNSName,"*.google.co.jp"},
              {dNSName,"*.google.co.uk"},
              {dNSName,"*.google.com.ar"},
              {dNSName,"*.google.com.au"},
              {dNSName,"*.google.com.br"},
              {dNSName,"*.google.com.co"},
              {dNSName,"*.google.com.mx"},
              {dNSName,"*.google.com.tr"},
              {dNSName,"*.google.com.vn"},
              {dNSName,"*.google.de"},
              {dNSName,"*.google.es"},
              {dNSName,"*.google.fr"},
              {dNSName,"*.google.hu"},
              {dNSName,"*.google.it"},
              {dNSName,"*.google.nl"},
              {dNSName,"*.google.pl"},
              {dNSName,"*.google.pt"},
              {dNSName,"*.googleadapis.com"},
              {dNSName,"*.googleapis.cn"},
              {dNSName,"*.googlecnapps.cn"},
              {dNSName,"*.googlecommerce.com"},
              {dNSName,"*.googlevideo.com"},
              {dNSName,"*.gstatic.cn"},
              {dNSName,"*.gstatic.com"},
              {dNSName,"*.gstaticcnapps.cn"},
              {dNSName,"*.gvt1.com"},
              {dNSName,"*.gvt2.com"},
              {dNSName,"*.metric.gstatic.com"},
              {dNSName,"*.urchin.com"},
              {dNSName,"*.url.google.com"},
              {dNSName,"*.wear.gkecnapps.cn"},
              {dNSName,"*.youtube-nocookie.com"},
              {dNSName,"*.youtube.com"},
              {dNSName,"*.youtubeeducation.com"},
              {dNSName,"*.youtubekids.com"},
              {dNSName,"*.yt.be"},
              {dNSName,"*.ytimg.com"},
              {dNSName,"android.clients.google.com"},
              {dNSName,"android.com"},
              {dNSName,"developer.android.google.cn"},
              {dNSName,"developers.android.google.cn"},
              {dNSName,"g.co"},
              {dNSName,"ggpht.cn"},
              {dNSName,"gkecnapps.cn"},
              {dNSName,"goo.gl"},
              {dNSName,"google-analytics.com"},
              {dNSName,"google.com"},
              {dNSName,"googlecnapps.cn"},
              {dNSName,"googlecommerce.com"},
              {dNSName,"source.android.google.cn"},
              {dNSName,"urchin.com"},
              {dNSName,"www.goo.gl"},
              {dNSName,"youtu.be"},
              {dNSName,"youtube.com"},
              {dNSName,"youtubeeducation.com"},
              {dNSName,"youtubekids.com"},
              {dNSName,"yt.be"}]},
         {'Extension',
             {2,5,29,32},
             false,
             [{'PolicyInformation',{2,23,140,1,2,2},asn1_NOVALUE},
              {'PolicyInformation',{1,3,6,1,4,1,11129,2,5,3},asn1_NOVALUE}]},
         {'Extension',
             {2,5,29,31},
             false,
             [{'DistributionPoint',
                  {fullName,
                      [{uniformResourceIdentifier,
                           "http://crl.pki.goog/GTS1O1.crl"}]},
                  asn1_NOVALUE,asn1_NOVALUE}]},
         {'Extension',
             {1,3,6,1,4,1,11129,2,4,2},
             false,
             <<4,129,243,0,241,0,118,0,178,30,5,204,139,162,205,138,32,78,
               135,102,249,43,185,138,37,32,103,107,218,250,112,231,178,73,
               83,45,239,139,144,94,0,0,1,113,191,244,205,188,0,0,4,3,0,71,
               48,69,2,33,0,144,114,30,77,37,127,69,205,184,10,100,30,228,
               150,174,69,22,71,220,235,113,91,89,63,7,125,229,72,183,122,
               218,124,2,32,5,151,41,191,70,54,82,202,241,181,54,210,20,108,
               91,75,222,248,210,48,89,171,18,59,229,208,49,19,129,131,27,
               76,0,119,0,94,167,115,249,223,86,192,231,181,54,72,125,208,
               73,224,50,122,145,154,12,132,161,18,18,132,24,117,150,129,
               113,69,88,0,0,1,113,191,244,205,218,0,0,4,3,0,72,48,70,2,33,
               0,145,145,9,20,141,220,37,184,27,211,90,152,124,246,95,54,
               217,148,216,254,36,210,237,219,45,52,97,97,169,151,10,29,2,
               33,0,245,74,82,1,217,44,128,116,111,176,51,180,159,54,181,58,
               6,53,137,39,177,213,26,158,222,115,161,210,239,9,101,74>>}]},
    {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
    <<103,178,222,42,157,134,197,197,19,8,52,58,217,182,180,138,149,239,238,
      179,148,196,63,244,31,249,12,164,204,78,45,164,41,185,236,175,70,211,26,
      215,126,97,181,10,59,114,254,3,90,78,130,222,154,62,197,72,8,33,44,231,
      12,12,112,82,18,211,71,198,115,200,122,223,103,103,82,144,138,8,185,101,
      252,235,36,123,127,209,39,164,117,174,26,35,177,156,129,164,204,76,62,
      138,233,8,33,91,181,116,0,12,250,81,59,52,128,194,233,169,191,193,213,
      166,132,31,25,46,187,125,77,146,246,205,205,37,192,171,112,5,81,82,20,
      99,237,56,164,190,51,157,38,241,43,118,78,150,102,249,209,42,229,154,63,
      179,37,62,170,32,114,28,19,120,179,33,219,178,14,198,97,66,28,30,158,
      214,243,60,20,82,25,115,117,115,170,82,19,38,243,206,29,255,74,24,244,
      31,34,21,32,58,90,150,1,253,243,69,134,47,189,16,79,129,251,0,79,109,20,
      138,62,61,136,231,124,171,62,185,146,200,198,46,229,194,8,176,205,177,
      15,97,73,61,239,39,80,104,78,40,143,69,239,88,244,172>>},[{dns_id,"contacts.google.com"}],[]) ({ssl_certificate,
                                                                                                      verify_hostname,
                                                                                                      4})
=NOTICE REPORT==== 16-May-2020::10:05:16.721529 ===
TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure
 - {bad_cert,hostname_check_failed}
(<0.517.0>) returned from public_key:pkix_verify_hostname/3 -> false
(foo@127.0.0.1)47>

Here's a code for reproduction

hackney:get(<<"https://contacts.google.com/">>).

Trace dump. It seems that it fails to handle wildcard certificates.

(foo@127.0.0.1)46> hackney:get(<<"https://contacts.google.com/">>).
{error,{tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}
}
(<0.517.0>) call public_key:pkix_verify_hostname({'OTPCertificate',
    {'OTPTBSCertificate',v3,321731345735978692271199214973930037793,
        {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
        {rdnSequence,
            [[{'AttributeTypeAndValue',{2,5,4,6},"US"}],
             [{'AttributeTypeAndValue',
                  {2,5,4,10},
                  {printableString,"Google Trust Services"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,3},
                  {printableString,"GTS CA 1O1"}}]]},
        {'Validity',{utcTime,"200428074341Z"},{utcTime,"200721074341Z"}},
        {rdnSequence,
            [[{'AttributeTypeAndValue',{2,5,4,6},"US"}],
             [{'AttributeTypeAndValue',
                  {2,5,4,8},
                  {printableString,"California"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,7},
                  {printableString,"Mountain View"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,10},
                  {printableString,"Google LLC"}}],
             [{'AttributeTypeAndValue',
                  {2,5,4,3},
                  {utf8String,<<"*.google.com">>}}]]},
        {'OTPSubjectPublicKeyInfo',
            {'PublicKeyAlgorithm',
                {1,2,840,10045,2,1},
                {namedCurve,{1,2,840,10045,3,1,7}}},
            {'ECPoint',
                <<4,15,69,78,47,12,167,136,154,185,36,255,87,80,220,241,171,
                  110,221,62,127,130,38,48,167,18,159,129,138,39,157,125,6,
                  46,211,226,80,59,206,108,45,46,91,50,206,125,235,134,6,124,
                  140,41,43,71,97,222,240,202,248,183,152,0,33,106,52>>}},
        asn1_NOVALUE,asn1_NOVALUE,
        [{'Extension',{2,5,29,15},true,[digitalSignature]},
         {'Extension',{2,5,29,37},false,[{1,3,6,1,5,5,7,3,1}]},
         {'Extension',
             {2,5,29,19},
             true,
             {'BasicConstraints',false,asn1_NOVALUE}},
         {'Extension',
             {2,5,29,14},
             false,
             <<160,137,132,208,190,181,43,113,58,250,115,5,221,228,44,197,138,
               218,71,239>>},
         {'Extension',
             {2,5,29,35},
             false,
             {'AuthorityKeyIdentifier',
                 <<152,209,248,110,16,235,207,155,236,96,159,24,144,27,160,235,
                   125,9,253,43>>,
                 asn1_NOVALUE,asn1_NOVALUE}},
         {'Extension',
             {1,3,6,1,5,5,7,1,1},
             false,
             [{'AccessDescription',
                  {1,3,6,1,5,5,7,48,1},
                  {uniformResourceIdentifier,"http://ocsp.pki.goog/gts1o1"}},
              {'AccessDescription',
                  {1,3,6,1,5,5,7,48,2},
                  {uniformResourceIdentifier,
                      "http://pki.goog/gsr2/GTS1O1.crt"}}]},
         {'Extension',
             {2,5,29,17},
             false,
             [{dNSName,"*.google.com"},
              {dNSName,"*.android.com"},
              {dNSName,"*.appengine.google.com"},
              {dNSName,"*.bdn.dev"},
              {dNSName,"*.cloud.google.com"},
              {dNSName,"*.crowdsource.google.com"},
              {dNSName,"*.g.co"},
              {dNSName,"*.gcp.gvt2.com"},
              {dNSName,"*.gcpcdn.gvt1.com"},
              {dNSName,"*.ggpht.cn"},
              {dNSName,"*.gkecnapps.cn"},
              {dNSName,"*.google-analytics.com"},
              {dNSName,"*.google.ca"},
              {dNSName,"*.google.cl"},
              {dNSName,"*.google.co.in"},
              {dNSName,"*.google.co.jp"},
              {dNSName,"*.google.co.uk"},
              {dNSName,"*.google.com.ar"},
              {dNSName,"*.google.com.au"},
              {dNSName,"*.google.com.br"},
              {dNSName,"*.google.com.co"},
              {dNSName,"*.google.com.mx"},
              {dNSName,"*.google.com.tr"},
              {dNSName,"*.google.com.vn"},
              {dNSName,"*.google.de"},
              {dNSName,"*.google.es"},
              {dNSName,"*.google.fr"},
              {dNSName,"*.google.hu"},
              {dNSName,"*.google.it"},
              {dNSName,"*.google.nl"},
              {dNSName,"*.google.pl"},
              {dNSName,"*.google.pt"},
              {dNSName,"*.googleadapis.com"},
              {dNSName,"*.googleapis.cn"},
              {dNSName,"*.googlecnapps.cn"},
              {dNSName,"*.googlecommerce.com"},
              {dNSName,"*.googlevideo.com"},
              {dNSName,"*.gstatic.cn"},
              {dNSName,"*.gstatic.com"},
              {dNSName,"*.gstaticcnapps.cn"},
              {dNSName,"*.gvt1.com"},
              {dNSName,"*.gvt2.com"},
              {dNSName,"*.metric.gstatic.com"},
              {dNSName,"*.urchin.com"},
              {dNSName,"*.url.google.com"},
              {dNSName,"*.wear.gkecnapps.cn"},
              {dNSName,"*.youtube-nocookie.com"},
              {dNSName,"*.youtube.com"},
              {dNSName,"*.youtubeeducation.com"},
              {dNSName,"*.youtubekids.com"},
              {dNSName,"*.yt.be"},
              {dNSName,"*.ytimg.com"},
              {dNSName,"android.clients.google.com"},
              {dNSName,"android.com"},
              {dNSName,"developer.android.google.cn"},
              {dNSName,"developers.android.google.cn"},
              {dNSName,"g.co"},
              {dNSName,"ggpht.cn"},
              {dNSName,"gkecnapps.cn"},
              {dNSName,"goo.gl"},
              {dNSName,"google-analytics.com"},
              {dNSName,"google.com"},
              {dNSName,"googlecnapps.cn"},
              {dNSName,"googlecommerce.com"},
              {dNSName,"source.android.google.cn"},
              {dNSName,"urchin.com"},
              {dNSName,"www.goo.gl"},
              {dNSName,"youtu.be"},
              {dNSName,"youtube.com"},
              {dNSName,"youtubeeducation.com"},
              {dNSName,"youtubekids.com"},
              {dNSName,"yt.be"}]},
         {'Extension',
             {2,5,29,32},
             false,
             [{'PolicyInformation',{2,23,140,1,2,2},asn1_NOVALUE},
              {'PolicyInformation',{1,3,6,1,4,1,11129,2,5,3},asn1_NOVALUE}]},
         {'Extension',
             {2,5,29,31},
             false,
             [{'DistributionPoint',
                  {fullName,
                      [{uniformResourceIdentifier,
                           "http://crl.pki.goog/GTS1O1.crl"}]},
                  asn1_NOVALUE,asn1_NOVALUE}]},
         {'Extension',
             {1,3,6,1,4,1,11129,2,4,2},
             false,
             <<4,129,243,0,241,0,118,0,178,30,5,204,139,162,205,138,32,78,
               135,102,249,43,185,138,37,32,103,107,218,250,112,231,178,73,
               83,45,239,139,144,94,0,0,1,113,191,244,205,188,0,0,4,3,0,71,
               48,69,2,33,0,144,114,30,77,37,127,69,205,184,10,100,30,228,
               150,174,69,22,71,220,235,113,91,89,63,7,125,229,72,183,122,
               218,124,2,32,5,151,41,191,70,54,82,202,241,181,54,210,20,108,
               91,75,222,248,210,48,89,171,18,59,229,208,49,19,129,131,27,
               76,0,119,0,94,167,115,249,223,86,192,231,181,54,72,125,208,
               73,224,50,122,145,154,12,132,161,18,18,132,24,117,150,129,
               113,69,88,0,0,1,113,191,244,205,218,0,0,4,3,0,72,48,70,2,33,
               0,145,145,9,20,141,220,37,184,27,211,90,152,124,246,95,54,
               217,148,216,254,36,210,237,219,45,52,97,97,169,151,10,29,2,
               33,0,245,74,82,1,217,44,128,116,111,176,51,180,159,54,181,58,
               6,53,137,39,177,213,26,158,222,115,161,210,239,9,101,74>>}]},
    {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
    <<103,178,222,42,157,134,197,197,19,8,52,58,217,182,180,138,149,239,238,
      179,148,196,63,244,31,249,12,164,204,78,45,164,41,185,236,175,70,211,26,
      215,126,97,181,10,59,114,254,3,90,78,130,222,154,62,197,72,8,33,44,231,
      12,12,112,82,18,211,71,198,115,200,122,223,103,103,82,144,138,8,185,101,
      252,235,36,123,127,209,39,164,117,174,26,35,177,156,129,164,204,76,62,
      138,233,8,33,91,181,116,0,12,250,81,59,52,128,194,233,169,191,193,213,
      166,132,31,25,46,187,125,77,146,246,205,205,37,192,171,112,5,81,82,20,
      99,237,56,164,190,51,157,38,241,43,118,78,150,102,249,209,42,229,154,63,
      179,37,62,170,32,114,28,19,120,179,33,219,178,14,198,97,66,28,30,158,
      214,243,60,20,82,25,115,117,115,170,82,19,38,243,206,29,255,74,24,244,
      31,34,21,32,58,90,150,1,253,243,69,134,47,189,16,79,129,251,0,79,109,20,
      138,62,61,136,231,124,171,62,185,146,200,198,46,229,194,8,176,205,177,
      15,97,73,61,239,39,80,104,78,40,143,69,239,88,244,172>>},[{dns_id,"contacts.google.com"}],[]) ({ssl_certificate,
                                                                                                      verify_hostname,
                                                                                                      4})
=NOTICE REPORT==== 16-May-2020::10:05:16.721529 ===
TLS client: In state certify at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure
 - {bad_cert,hostname_check_failed}
(<0.517.0>) returned from public_key:pkix_verify_hostname/3 -> false
(foo@127.0.0.1)47>

is this on OTP 23?

benoitc commented 4 years ago

I am reproducing it on erlang 23 but not erlang 22. fix is coming.

benoitc commented 4 years ago

Still investigating. There is something weird happeing with OTP 23 when checking the hostname:

68> ssl:connect("contacts.google.com", 443, [{cacertfile, certifi:cacertfile()}, {server_name_indication, "contacts.google.com"},  {reuse_sessions, false}, {verify, verify_peer}, {depth, 99}]).                                               
2020-05-20T06:14:46.932618+02:00 notice: TLS client: In state wait_cert_cr at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure, - {bad_cert,hostname_check_failed}
{error,{tls_alert,{handshake_failure,"TLS client: In state wait_cert_cr at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}}
69> ssl:connect("contacts.google.com", 443, [{cacertfile, certifi:cacertfile()}, {server_name_indication, "google.com"},  {reuse_sessions, false}, {verify, verify_peer}, {depth, 99}]).         
{ok,{sslsocket,{gen_tcp,#Port<0.66>,tls_connection,
                        undefined},
               [<0.1076.0>,<0.1075.0>]}}
70>

afaik, contacts.google.com return *.google.com. as DN in their certificate. So the issue is in checking the wildcard domain in SSL certificates. Same issue with ssl_verify_fun

voltone commented 4 years ago

This is due to https://bugs.erlang.org/browse/ERL-1232

Prior to 23, OTP's own hostname verification was not active when a custom verify_fun was used. In 23, the built-in hostname check is performed prior to calling the verification function.

By default OTP does not permit wildcards in SAN extensions, unless you opt-in using {customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}

(or, to be precise, by default it recognises wildcards only in Subject CN, and only when no SAN extension is present)

benoitc commented 4 years ago

fixed by 7cbe1f6f9a2c464a5ec4dcccca0a7ed41005b9af . Thanks @voltone :)

lasseebert commented 4 years ago

For the record: I got hit by both this issue and #623 after upgrading to otp-23. I just tested with the current master that contains fixes for both and it works now :+1:

msudgh commented 3 years ago

Seems the issue is reproducible, any comments on the referenced issue? https://github.com/getsentry/sentry-elixir/issues/443