search

cal-itp / data-infra

Cal-ITP data infrastructure
https://docs.calitp.org/data-infra
GNU Affero General Public License v3.0
48 stars 13 forks source link

Analyst no longer sees GTFS-RT data from Big Blue Bus #3452

Closed cal-itp-sentry[bot] closed 2 months ago

cal-itp-sentry[bot] commented 2 months ago

Sentry Issue: CAL-ITP-DATA-INFRA-29ZN

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1133)
(4 additional frame(s) were not displayed)
...
  File "urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "ssl.py", line 1074, in _create
    self.do_handshake()
  File "ssl.py", line 1343, in do_handshake
    self._sslobj.do_handshake()

MaxRetryError: HTTPSConnectionPool(host='cleverapi.bigbluebus.com', port=443): Max retries exceeded with url: /gtfsrt/vehicles (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1133)')))
  File "requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))

SSLError: HTTPSConnectionPool(host='cleverapi.bigbluebus.com', port=443): Max retries exceeded with url: /gtfsrt/vehicles (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1133)')))
  File "gtfs_rt_archiver_v3/tasks.py", line 145, in fetch
    extract, content = download_feed(

RTFetchException: SSLError  https://cleverapi.bigbluebus.com/gtfsrt/vehicles
  File "gtfs_rt_archiver_v3/tasks.py", line 122, in inner
    return f(*args, **kwargs)
  File "gtfs_rt_archiver_v3/tasks.py", line 172, in fetch
    raise RTFetchException(config.url, cause=e, status_code=status_code) from e

Unhandled exception in task d7aa53fe-b8fe-4ddb-88bf-609b4018bb44.
ohrite commented 2 months ago

Symptoms

Sentry records that the RT Downloader cannot make an SSL connection to the Santa Monica Big Blue Bus GTFS-RT API server. Further investigation shows that there is an SSL certificate error, as indicated above.

Root Cause

The cleverapi.bigbluebus.com certificate chain is incomplete, as per SSL Shopper and Qualys SSL Labs.

The missing certificate means that curl fails to validate the SSL connection, even when the latest ca-certificates package is installed on the RT Downloader image, which is currently running Debian 12.

Immediate Remediation

PR #3457 pins the current cleverapi.bigbluebus.com certificate. This approach allows curl to connect, and does not interfere with connections to other servers (tested against api.511.org).

Frequency Analysis

A pinned certificate will expire or be rotated at some future date, and this issue will continue to occur. This is because the SSL certificate chain is misconfigured. Big Blue Bus should ensure that their SSL certificate chain is complete so that we (and presumably others) are able to connect to it. The issue can be considered resolved when SSL analysis tools report that it is complete.

evansiroky commented 2 months ago

@ohrite thanks so much for the incident report! I'll pass this along to the transit agency.

In your opinion, is this more of a problem with the data source we are downloading from or a problem with our method of making requests from our pipeline?

themightychris commented 2 months ago

@evansiroky they're not using a widely-accepted certificate, or are failing to bundle a necessary intermediary certificate with it.

It fails a standard checker: https://www.sslshopper.com/ssl-checker.html#hostname=https://cleverapi.bigbluebus.com/ and will fail on any debian-based systems, and fails in my local Chrome too.

This issue likely affects some and potentially many RT clients too, they need to fix their cert

As long as we're updating our Docker image periodically, our method of making requests will be pretty representative of typical consumers

ohrite commented 2 months ago

@evansiroky In my opinion, this is an issue with a misconfigured data source. If this was a consumer-facing website, it'd either pop a warning screen, have a broken lock in the URL bar, fail to load, or load just fine. There would be user error reports, in any case, and this would not be an acceptable setup.

As an example of a complete chain, you can run this command and see error code 0 (OK):

$ openssl s_client -showcerts -connect cal-itp.org:443
Connecting to 172.67.167.71
CONNECTED(00000006)
depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R4
verify return:1
depth=1 C=US, O=Google Trust Services, CN=WE1
verify return:1
depth=0 CN=cal-itp.org
verify return:1
---
Certificate chain
 0 s:CN=cal-itp.org
   i:C=US, O=Google Trust Services, CN=WE1
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Aug 21 01:18:52 2024 GMT; NotAfter: Nov 19 01:18:51 2024 GMT
-----BEGIN CERTIFICATE-----
MIIDozCCA0mgAwIBAgIQXbVhflMIPccN2Flg2YUhDzAKBggqhkjOPQQDAjA7MQsw
CQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQwwCgYD
VQQDEwNXRTEwHhcNMjQwODIxMDExODUyWhcNMjQxMTE5MDExODUxWjAWMRQwEgYD
VQQDEwtjYWwtaXRwLm9yZzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABPLzB5qC
XatgAtcHhqmzfLKlH15plA/R36PGGRfcfKdGpm6qWcYyiPDPHbrV85LIvuWCIYiC
ilPbAVwdpHW+oR2jggJSMIICTjAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0lBAwwCgYI
KwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU+86BZnkoBpSHWyMplGEA
FdTX1IcwHwYDVR0jBBgwFoAUkHeSNWfE/6jMqeZ72YB5e8yT+TgwXgYIKwYBBQUH
AQEEUjBQMCcGCCsGAQUFBzABhhtodHRwOi8vby5wa2kuZ29vZy9zL3dlMS9YYlUw
JQYIKwYBBQUHMAKGGWh0dHA6Ly9pLnBraS5nb29nL3dlMS5jcnQwJQYDVR0RBB4w
HIILY2FsLWl0cC5vcmeCDSouY2FsLWl0cC5vcmcwEwYDVR0gBAwwCjAIBgZngQwB
AgEwNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2MucGtpLmdvb2cvd2UxL2h5Vk95
WUt4ejlFLmNybDCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AO7N0GTV2xrOxVy3
nbTNE6Iyh0Z8vOzew1FIWUZxH7WbAAABkXK61oMAAAQDAEYwRAIgLDqvLATnxIYR
36XKVJ1MDa7RYLuzVmf59uNGFzl/jUsCIENrotY/FU5sZVZ+dX6p9cbReWS6968W
boLgvrkygPWxAHYA3+FW66oFr7WcD4ZxjajAMk6uVtlup/WlagHRwTu+UlwAAAGR
crrXegAABAMARzBFAiAnk9rVb7w9wsRmZ/q4zy04nHgyUnJ4hQ0tM1JS9fA64QIh
AIsSHgfmxQ0OUkc3C+5qVuMKEzoP4W9jpuGQM1spiJNRMAoGCCqGSM49BAMCA0gA
MEUCIQCxNWymnGYaD8O/QgKhFhzDDao0OmYJKy+N0y9rVzXIVgIgG6VS6tqouMJm
xmyjoqQnVKJs50053WjerWFGFgqljXw=
-----END CERTIFICATE-----
 1 s:C=US, O=Google Trust Services, CN=WE1
   i:C=US, O=Google Trust Services LLC, CN=GTS Root R4
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Dec 13 09:00:00 2023 GMT; NotAfter: Feb 20 14:00:00 2029 GMT
-----BEGIN CERTIFICATE-----
MIICnzCCAiWgAwIBAgIQf/MZd5csIkp2FV0TttaF4zAKBggqhkjOPQQDAzBHMQsw
CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU
MBIGA1UEAxMLR1RTIFJvb3QgUjQwHhcNMjMxMjEzMDkwMDAwWhcNMjkwMjIwMTQw
MDAwWjA7MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZp
Y2VzMQwwCgYDVQQDEwNXRTEwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARvzTr+
Z1dHTCEDhUDCR127WEcPQMFcF4XGGTfn1XzthkubgdnXGhOlCgP4mMTG6J7/EFmP
LCaY9eYmJbsPAvpWo4H+MIH7MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggr
BgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU
kHeSNWfE/6jMqeZ72YB5e8yT+TgwHwYDVR0jBBgwFoAUgEzW63T/STaj1dj8tT7F
avCUHYwwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzAChhhodHRwOi8vaS5wa2ku
Z29vZy9yNC5jcnQwKwYDVR0fBCQwIjAgoB6gHIYaaHR0cDovL2MucGtpLmdvb2cv
ci9yNC5jcmwwEwYDVR0gBAwwCjAIBgZngQwBAgEwCgYIKoZIzj0EAwMDaAAwZQIx
AOcCq1HW90OVznX+0RGU1cxAQXomvtgM8zItPZCuFQ8jSBJSjz5keROv9aYsAm5V
sQIwJonMaAFi54mrfhfoFNZEfuNMSQ6/bIBiNLiyoX46FohQvKeIoJ99cx7sUkFN
7uJW
-----END CERTIFICATE-----
 2 s:C=US, O=Google Trust Services LLC, CN=GTS Root R4
   i:C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 15 03:43:21 2023 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
-----BEGIN CERTIFICATE-----
MIIDejCCAmKgAwIBAgIQf+UwvzMTQ77dghYQST2KGzANBgkqhkiG9w0BAQsFADBX
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UE
CxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIzMTEx
NTAzNDMyMVoXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoT
GUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFI0
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE83Rzp2iLYK5DuDXFgTB7S0md+8Fhzube
Rr1r1WEYNa5A3XP3iZEwWus87oV8okB2O6nGuEfYKueSkWpz6bFyOZ8pn6KY019e
WIZlD6GEZQbR3IvJx3PIjGov5cSr0R2Ko4H/MIH8MA4GA1UdDwEB/wQEAwIBhjAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUwAwEB/zAd
BgNVHQ4EFgQUgEzW63T/STaj1dj8tT7FavCUHYwwHwYDVR0jBBgwFoAUYHtmGkUN
l8qJUC99BM00qP/8/UswNgYIKwYBBQUHAQEEKjAoMCYGCCsGAQUFBzAChhpodHRw
Oi8vaS5wa2kuZ29vZy9nc3IxLmNydDAtBgNVHR8EJjAkMCKgIKAehhxodHRwOi8v
Yy5wa2kuZ29vZy9yL2dzcjEuY3JsMBMGA1UdIAQMMAowCAYGZ4EMAQIBMA0GCSqG
SIb3DQEBCwUAA4IBAQAYQrsPBtYDh5bjP2OBDwmkoWhIDDkic574y04tfzHpn+cJ
odI2D4SseesQ6bDrarZ7C30ddLibZatoKiws3UL9xnELz4ct92vID24FfVbiI1hY
+SW6FoVHkNeWIP0GCbaM4C6uVdF5dTUsMVs/ZbzNnIdCp5Gxmx5ejvEau8otR/Cs
kGN+hr/W5GvT1tMBjgWKZ1i4//emhA1JG1BbPzoLJQvyEotc03lXjTaCzv8mEbep
8RqZ7a2CPsgRbuvTPBwcOMBBmuFeU88+FSBX6+7iP0il8b4Z0QFqIwwMHfs/L6K1
vepuoxtGzi4CZ68zJpiq1UvSqTbFJjtbD4seiMHl
-----END CERTIFICATE-----
---
Server certificate
subject=CN=cal-itp.org
issuer=C=US, O=Google Trust Services, CN=WE1
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2823 bytes and written 399 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

However, the same command against cleverapi.bigbluebus.com returns error code 21 (unable to verify the first certificate):

$openssl s_client -showcerts -connect cleverapi.bigbluebus.com:443
Connecting to 52.24.181.110
CONNECTED(00000005)
depth=0 CN=*.bigbluebus.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=*.bigbluebus.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN=*.bigbluebus.com
verify return:1
---
Certificate chain
 0 s:CN=*.bigbluebus.com
   i:C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul  2 00:00:00 2024 GMT; NotAfter: Aug  2 23:59:59 2025 GMT
-----BEGIN CERTIFICATE-----
MIIGOTCCBSGgAwIBAgIRAJXyFcnDDisopOpfbsjH2jEwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yNDA3MDIwMDAwMDBaFw0yNTA4MDIyMzU5NTlaMBsxGTAXBgNVBAMMECou
YmlnYmx1ZWJ1cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDQ
1EdTJWElpqWaxrJl+BSK48S68VzADdHvepxgfAAHmVOXdOPzeGBvlpZ3wt7ADo77
2z8Hgm/ihlVslxxRSjomuhIkSBc6bKCLmiXgNrHjzmqwKFYnB5LAqwpk1qtsKred
aS4Jzv93M/Yk1atrxLtj1mp2onSk7169XjEfLymxSqWxFM3MSSwFgHYK/jRh0Vkr
P79kB51ZMNPq+VhxY72mhKYXxgofWLiQUB6fvYKT4nyDWR5UkwPzoBuYQBYlPaq+
i1jVjDfRfI/vOsMEwZyvuX0kDOaHKzhsib82FkHZNexqEGKgJzzCeSX7nDCQqeka
DbN6hkqUkKiDgb9X5ZjdAgMBAAGjggMBMIIC/TAfBgNVHSMEGDAWgBSNjF7EVK2K
4Xfpm/mbBeG4AY1h4TAdBgNVHQ4EFgQUt3v/V7GcJ+/HyZFMeKdJtnhZ5hAwDgYD
VR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEG
CCsGAQUFBwMCMEkGA1UdIARCMEAwNAYLKwYBBAGyMQECAgcwJTAjBggrBgEFBQcC
ARYXaHR0cHM6Ly9zZWN0aWdvLmNvbS9DUFMwCAYGZ4EMAQIBMIGEBggrBgEFBQcB
AQR4MHYwTwYIKwYBBQUHMAKGQ2h0dHA6Ly9jcnQuc2VjdGlnby5jb20vU2VjdGln
b1JTQURvbWFpblZhbGlkYXRpb25TZWN1cmVTZXJ2ZXJDQS5jcnQwIwYIKwYBBQUH
MAGGF2h0dHA6Ly9vY3NwLnNlY3RpZ28uY29tMCsGA1UdEQQkMCKCECouYmlnYmx1
ZWJ1cy5jb22CDmJpZ2JsdWVidXMuY29tMIIBfQYKKwYBBAHWeQIEAgSCAW0EggFp
AWcAdQDd3Mo0ldfhFgXnlTL6x5/4PRxQ39sAOhQSdgosrLvIKgAAAZB0KZbEAAAE
AwBGMEQCIED43lSXZGdno6+hHvfHyUaS+pHQlhrQGsVj23g/Ak2oAiBnmgiEQd4c
N0pDbfxTIn68T6freKHUyaCphP8RSpuzLQB2AA3h8jAr0w3BQGISCepVLvxHdHyx
1+kw7w5CHrR+Tqo0AAABkHQplqQAAAQDAEcwRQIgIbKndPknvKpFajERgeYNHneG
PFG+KGfvt6MNPPyCiQcCIQDlklaxZxQzrfWhqQ66AHlTuK4fbtiLIGQ5UKMP5DBc
aQB2ABLxTjS9U3JMhAYZw48/ehP457Vih4icbTAFhOvlhiY6AAABkHQplqQAAAQD
AEcwRQIgMlTZFYzdMA4asHwz3m4G6zn6gxxGeadRkmlL+jWzQGgCIQD1ftlNyH7k
MiUb1QxFGTnGfZtCe+QGnL0LNaHPB0h0DjANBgkqhkiG9w0BAQsFAAOCAQEAsVIH
+6IJHJwFLDKZHgIjAo4IcdRsKqY2fCcGEabT3eFPvKOIVIh+S69mhxLXXVkAblzU
H5Z7usGF0NhIW9ABsF+cT1k6LPyyiBZdP/Y7CMeYl/+FcE4r7vf6hN9GvizmVIWo
rzONlDR/Dpk0mGewCcKge1AwcU0MUJCEmA+gUfVRvJPBVSJOWJSUSKydpeXkAOd+
Zy5DzEVNXD6giM9khLXHSVYX7zzCHPL0MAg9OFyqtDQXg7DxXliM7ZxaT1PUg+cP
FmcThQD1JrHhcN5/YN8C+MXoGPZSRWpvgp0eeIiHiEhKIqtgx64i5xuasZJa4uaU
uJgYV8gETZvNYSGrvw==
-----END CERTIFICATE-----
---
Server certificate
subject=CN=*.bigbluebus.com
issuer=C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2157 bytes and written 412 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE
evansiroky commented 2 months ago

Upon receiving the root cause analysis, this can be marked as complete

evansiroky commented 1 month ago

Referencing our proposal to add a best practice to the GTFS spec of requiring "good" HTTPS https://github.com/google/transit/issues/496