[Open Data vNext] Intermittent SSL certificate issues

[zachomedia]

It looks like that one of the two servers (api-web-2) serving openapi.data.uwaterloo.ca is either missing the SSL cert or is not configured with SNI correctly.

On the command line, this results in intermittent failures if you hit the broken server. In your browser, it's not possible to load the page when you get the broken server because of HSTS.

Broken

❯ openssl s_client -servername openapi.data.uwaterloo.ca -connect openapi.data.uwaterloo.ca:443 </dev/null
CONNECTED(00000005)
depth=0 CN = api-web-2.NEXUS.UWATERLOO.CA
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = api-web-2.NEXUS.UWATERLOO.CA
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=api-web-2.NEXUS.UWATERLOO.CA
   i:/CN=api-web-2.NEXUS.UWATERLOO.CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIC/DCCAeSgAwIBAgIQMxv63ikQTJJM1muFJAbarzANBgkqhkiG9w0BAQsFADAn
MSUwIwYDVQQDExxhcGktd2ViLTIuTkVYVVMuVVdBVEVSTE9PLkNBMB4XDTE4MDUw
ODE5MjUyNloXDTE5MDUwODAwMDAwMFowJzElMCMGA1UEAxMcYXBpLXdlYi0yLk5F
WFVTLlVXQVRFUkxPTy5DQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJMBps1SyuCZl9h/5Gmvca2FUVSEEzNMDiaQLC7HoL1N7NGoA4605Cg/FC8A0bQS
0jD0B4InLwaUOtQD847n7spdcC1bbRFRiy5UwWY5/gh8iQ4EeFVWWJjgn4d7cB3b
knFYvewHwdn9iSwsljloeNSiKEdtWaLhlnazp8wbe+uItHBPw+wwogvcY+uK21hq
G0/v26KW+rk+B0RYz+XKNLsAESuy2tb4Liju1CVoSaKffi3QMZbx2nK+H72LNNVO
Mc1A65TJMkfxKjv59VcFESlYUhCxhugfEYP+pZXVy3zUv6jGWYG4ZBiLkpzKaY5B
LHNzdvOA09fEdRV30Jwaa5UCAwEAAaMkMCIwCwYDVR0PBAQDAgQwMBMGA1UdJQQM
MAoGCCsGAQUFBwMBMA0GCSqGSIb3DQEBCwUAA4IBAQBmYO6rlJuSUPN+qiZpogvL
Ki7mGVYIrTVBI6xeqfvGLU8r8jxPOFKUIXW6YfC5BIr5+fYM27USbSERZ5HxXU1e
CRaXZbldfuaGf//462vU/uiadgUhdZrdvg5dottYVoeeR2KpKKXwDH2x6NWjdQ04
qmDGvu8qzCgiP6mVsxm0ulVSW363mFWlnAApzS3a052n74/aUjL7wg45ebLfdrXT
+hHrxWGxSLQ4UHb3knJzk17GT5nBLJfrcEgsMNqF9PHdZC0nesXKAFOapeqo6wJE
ln3+ii7oBywh6TaF3AXkcPd4asdEBO0a89dWiuMTJ6FEcivO8a89wZO3s/2GENOW
-----END CERTIFICATE-----
subject=/CN=api-web-2.NEXUS.UWATERLOO.CA
issuer=/CN=api-web-2.NEXUS.UWATERLOO.CA
---
No client certificate CA names sent
---
SSL handshake has read 1252 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 581900002F9E2C7C98A9554206070B12E53EA70CBD281290DAF381EE24F41A20
    Session-ID-ctx:
    Master-Key: 75BE0F94528352D34B80FDCF0D474B613EA39430400C3D2FCFE9EEC476A401137C2ABDD8B7FAB9C00E834D401C70200B
    Start Time: 1536274340
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

Working

❯ openssl s_client -servername openapi.data.uwaterloo.ca -connect openapi.data.uwaterloo.ca:443 </dev/null
CONNECTED(00000005)
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Organization Validation CA - SHA256 - G2
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=CA/ST=Ontario/L=Waterloo/O=University of Waterloo/CN=openapi.data.uwaterloo.ca
   i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHSTCCBjGgAwIBAgIMPpn2RE5PYXVkW0oEMA0GCSqGSIb3DQEBCwUAMGYxCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTwwOgYDVQQDEzNH
bG9iYWxTaWduIE9yZ2FuaXphdGlvbiBWYWxpZGF0aW9uIENBIC0gU0hBMjU2IC0g
RzIwHhcNMTgwMjA1MTUwNjAyWhcNMTkwMjA2MTUwNjAyWjB3MQswCQYDVQQGEwJD
QTEQMA4GA1UECBMHT250YXJpbzERMA8GA1UEBxMIV2F0ZXJsb28xHzAdBgNVBAoT
FlVuaXZlcnNpdHkgb2YgV2F0ZXJsb28xIjAgBgNVBAMTGW9wZW5hcGkuZGF0YS51
d2F0ZXJsb28uY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCoJIjQ
vjfKXRDEZ6pU9cQCFWzHaOtJlUWFyOWHa8E9kJiXAM4fZdDLDDiJO+DYwo1468Xt
S/4RcSHnc5lO6DTGf7sK1zcNXXIt3xjhktsjOSFNVhifo2EMveiI2O4FJ0PVydcJ
lk8gAPsNidrcKx9WrvLeTC5cud8J/4n/zxlFRVDLd8f/KR18LnmVrIbObPoUx/Qc
wT40+eKjA10cupAM/neOHdOC4+bT1uHXkBcHRySz1XS2GOPHizcbv4+mwccn7kSt
QtrhBYjvi2c9U+CubHblfn4wRImJHO7JDNgaC+wkecf1z21ZZg4HYqLNAh3inHRu
Yd+HIus159lnTGr5AgMBAAGjggPkMIID4DAOBgNVHQ8BAf8EBAMCBaAwgaAGCCsG
AQUFBwEBBIGTMIGQME0GCCsGAQUFBzAChkFodHRwOi8vc2VjdXJlLmdsb2JhbHNp
Z24uY29tL2NhY2VydC9nc29yZ2FuaXphdGlvbnZhbHNoYTJnMnIxLmNydDA/Bggr
BgEFBQcwAYYzaHR0cDovL29jc3AyLmdsb2JhbHNpZ24uY29tL2dzb3JnYW5pemF0
aW9udmFsc2hhMmcyMFYGA1UdIARPME0wQQYJKwYBBAGgMgEUMDQwMgYIKwYBBQUH
AgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMAgGBmeB
DAECAjAJBgNVHRMEAjAAMEkGA1UdHwRCMEAwPqA8oDqGOGh0dHA6Ly9jcmwuZ2xv
YmFsc2lnbi5jb20vZ3MvZ3Nvcmdhbml6YXRpb252YWxzaGEyZzIuY3JsMCQGA1Ud
EQQdMBuCGW9wZW5hcGkuZGF0YS51d2F0ZXJsb28uY2EwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBTPgN+KOVIoeMsJ+3pSvL23Adcu+zAf
BgNVHSMEGDAWgBSW3mHxvRwWKVMcwMx9O4MAQOYafDCCAfYGCisGAQQB1nkCBAIE
ggHmBIIB4gHgAHYA3esdK3oNT6Ygi4GtgWhwfi6OnQHVXIiNPRHEzbbsvswAAAFh
ZoC3PwAABAMARzBFAiA6X8V5CjIZoFkrNWJ5ykkzKNgeFwaI7ACyIT2CqrfW5AIh
AKZIjtbv9/9ztdhbp6gVBNwV2ueIXp6M88rlDW2IpzTrAHYAVhQGmi/XwuzT9eG9
RLI+x0Z2ubyZEVzA75SYVdaJ0N0AAAFhZoC3fQAABAMARzBFAiEAuTgcDcStKdgW
zaHP40s+5wrdiU2EvRwx4ogTHb7DfXYCIB2JgaqlbLecG9p4T5a9dsOhAan/EWSc
xdSdnNAoR5V7AHYApLkJkLQYWBSHuxOizGdwCjw1mAT5G9+443fNDsgN3BAAAAFh
ZoC6DwAABAMARzBFAiB0pMCYdI6AVjvAbFyfu09u9y4mvQKzsZDYlaL0Vr32twIh
AMuX4a5Pnd2HX53vZNZJJo9FfVHWiyeQGFu9C45iXt1sAHYA7ku9t3XOYLrhQmkf
q+GeZqMPfl+wctiDAMR7iXqo/csAAAFhZoC9BAAABAMARzBFAiEAiZv6Gz3rO1Aj
xr6G8C5jEuF7dsy2fX5l73QEA2UkKoYCIEooeVPslJGN0bFtqj/BMEcPz74wVVqb
xqat1XjRRaFRMA0GCSqGSIb3DQEBCwUAA4IBAQA4Nnb2AAdCv0KpWZiYBM4vfTNp
QdGXguAM36D2CZb5mnCJFEm30UlQLwuAMAirPQ+Z4JBAMYjKpPENnXR5sSrDLJcs
YZV8lIbEQ/CTiMBpO21a/TyfUcED0VCeA8rQ7tyQgA4eQ0xY+Uvp3UPqiqxW9OfD
MA3/LW2c0eWlJaueYW1G+pjsWWrMElxPJgngDuWy83prb60kqWnCHQ4jFQmWDAS8
2jZSOucHz1/YSx3FRvPfIRsZOOsKqlJ2zijqXhKvO3A3wQAMGjcEM+Nq+G7DJ3p9
5+7GXzl8S2AFPcuuy5A6/Z9UrRKlEpGUKpDh1LtsvC5NW7Od3Ke+21tPN5dV
-----END CERTIFICATE-----
subject=/C=CA/ST=Ontario/L=Waterloo/O=University of Waterloo/CN=openapi.data.uwaterloo.ca
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
---
No client certificate CA names sent
---
SSL handshake has read 3493 bytes and written 478 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: DE270000F0152A772E1CCDB9C43BF349DC897221DD94D047F8271B1D5205BD25
    Session-ID-ctx:
    Master-Key: 322BB886A63A4FE2D7F096AC3C461903EF1E7D3EC45634CFF15C19E344CB1DFE31FC721C34364C2C1E38C6952E65C2B5
    Start Time: 1536274322
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[sbobkin]

Thanks for the report @zachomedia -- it looks like one of the servers isn't handling SNI properly as you've guessed. Going to look into it today : )

[sbobkin]

The IIS bindings to a central certificate store got dropped, which was causing the issue. That's now resolved. Thanks for the feedback!