search

hashicorp / terraform-provider-consul

Terraform Consul provider
https://www.terraform.io/docs/providers/consul/
Mozilla Public License 2.0
125 stars 113 forks source link

keybase.io certificate issued May 24, 2021 not trusted by provider #266

Closed quinndiggity closed 3 years ago

quinndiggity commented 3 years ago

Terraform Version

Terraform v0.15.4 on linux_amd64

Affected Resource(s)

Please list the resources as a list, for example:

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

resource "consul_acl_token" "issue_report" {
  description = "vault-agent"
  policies    = [
     ...
  ]
  local       = false
}

data "consul_acl_token_secret_id" "issue_report" {
  accessor_id = consul_acl_token.issue_report.id
  pgp_key     = "keybase:..." # keybase username
}

output "consul_acl_token_secret_id_issue_report" {
  value     = data.consul_acl_token_secret_id.issue_report.encrypted_secret_id
  sensitive = true
}

Debug Output

ā”‚ Error: Error retrieving Public Key for keybase:...: Get "https://keybase.io/_/api/1.0/user/lookup.json?usernames=...&fields=public_keys": x509: certificate signed by unknown authority

Expected Behavior

Encrypt acl tokens via keybase pgp key

Actual Behavior

keybase.io has a new certificate issued yesterday, this provider does not use the keybase client (which embeds their CA cert https://github.com/keybase/keybase-issues/issues/2371#issuecomment-233137409 ) https://github.com/hashicorp/terraform-provider-consul/blob/b8fe8859dc3cf8c4be888417d6f18e50a79c2243/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/vault/helper/pgpkeys/keybase.go#L21 so it is throwing x509: certificate signed by unknown authority on attempts to pull public keys from keybase

quinndiggity commented 3 years ago

openssl s_client -showcerts -connect keybase.io:443 -servername keybase.io | openssl x509 -noout -text -in -

depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = keybase.io
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0b:08:14:b4:d2:28:dc:ca:05:f4:2e:6f:3a:86:f5:73
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
        Validity
            Not Before: May 24 00:00:00 2021 GMT
            Not After : May 24 23:59:59 2022 GMT
        Subject: CN = keybase.io
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:ae:3f:90:45:1d:a6:11:d5:f4:dc:83:51:22:92:
                    f7:cb:f5:f7:95:49:c6:4f:f0:df:12:1f:1a:b0:4a:
                    cf:1b:35:9f:54:8e:99:9e:d4:bf:b6:32:a0:d2:60:
                    6b:4a:cd:7e:5e:dc:96:4f:2c:b5:1b:a1:bd:29:7e:
                    a1:66:7d:7e:12:6c:3d:3a:11:ee:75:c9:72:b0:b3:
                    34:c3:0d:1b:6b:97:11:74:43:5c:a2:de:df:f1:c5:
                    3d:39:96:89:ac:48:56:28:d6:f4:55:db:3c:a3:b5:
                    9a:72:1d:7d:61:94:36:ab:01:47:c7:d0:93:26:f5:
                    7f:b8:99:fe:a7:67:c9:1b:90:46:07:64:cf:22:48:
                    da:c7:85:77:b6:25:c2:9d:1d:6d:f2:b3:53:a0:ac:
                    39:6e:08:a7:4f:a0:40:d5:35:54:ba:ee:4a:f8:b8:
                    0d:ca:8e:6f:ad:fa:21:8d:fc:47:7a:46:df:82:8b:
                    ae:36:45:d2:ed:c9:32:fc:92:ee:e1:d8:d4:64:72:
                    66:6b:69:a8:e6:44:90:10:b9:a1:0b:f4:34:db:0d:
                    8c:39:13:ea:90:76:ea:70:9a:9a:93:a6:13:20:3f:
                    94:16:40:d2:60:47:2e:c6:bd:53:29:e6:17:80:2e:
                    8d:bd:13:e6:02:ca:51:50:7e:a6:2e:a0:1a:a7:b0:
                    1e:15
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:8D:8C:5E:C4:54:AD:8A:E1:77:E9:9B:F9:9B:05:E1:B8:01:8D:61:E1

            X509v3 Subject Key Identifier: 
                29:1D:96:21:5E:17:3B:7E:9E:5C:5C:AB:A4:0D:91:C7:95:D2:3D:04
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Certificate Policies: 
                Policy: 1.3.6.1.4.1.6449.1.2.2.7
                  CPS: https://sectigo.com/CPS
                Policy: 2.23.140.1.2.1

            Authority Information Access: 
                CA Issuers - URI:http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt
                OCSP - URI:http://ocsp.sectigo.com

            X509v3 Subject Alternative Name: 
                DNS:keybase.io, DNS:www.keybase.io
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 46:A5:55:EB:75:FA:91:20:30:B5:A2:89:69:F4:F3:7D:
                                11:2C:41:74:BE:FD:49:B8:85:AB:F2:FC:70:FE:6D:47
                    Timestamp : May 24 15:40:30.526 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:E5:3D:65:E1:9F:3A:05:61:8D:71:EC:
                                AA:99:18:36:20:7C:DD:E9:29:2C:35:A9:1E:5E:E5:66:
                                5B:7A:F9:B9:1A:02:20:1B:89:E3:DD:8A:BE:18:D4:FA:
                                0E:E0:C1:36:BD:FC:E3:BA:07:2A:07:A6:75:7B:35:2B:
                                82:CA:26:9B:DD:6F:9A
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : DF:A5:5E:AB:68:82:4F:1F:6C:AD:EE:B8:5F:4E:3E:5A:
                                EA:CD:A2:12:A4:6A:5E:8E:3B:12:C0:20:44:5C:2A:73
                    Timestamp : May 24 15:40:30.416 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:22:D1:FE:1B:BD:81:9A:95:08:02:08:AD:
                                DA:0B:C9:28:91:3A:F2:FC:D2:63:02:00:AA:E3:55:96:
                                E2:30:97:62:02:21:00:F7:ED:4B:2B:82:09:CA:9A:BC:
                                74:F8:04:47:33:BB:08:BF:6C:87:77:30:6D:4D:A2:F7:
                                40:42:E8:7C:D5:5E:42
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 29:79:BE:F0:9E:39:39:21:F0:56:73:9F:63:A5:77:E5:
                                BE:57:7D:9C:60:0A:F8:F9:4D:5D:26:5C:25:5D:C7:84
                    Timestamp : May 24 15:40:30.433 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:D3:AC:3B:16:7A:38:04:AE:FE:16:AA:
                                DA:C7:12:02:81:9B:36:80:48:6D:A2:9A:E4:FF:56:08:
                                93:0D:40:F2:DE:02:20:16:12:BC:68:79:BE:14:0C:96:
                                A5:DE:B8:D6:B8:C7:29:3F:B7:68:49:A7:13:1F:80:C7:
                                7F:BB:68:EF:14:51:A8
    Signature Algorithm: sha256WithRSAEncryption
         08:8a:13:d5:aa:a1:08:e9:d3:db:0c:d1:77:05:05:b6:95:2d:
         6b:19:c1:32:98:b4:bb:83:56:2d:56:64:3f:05:3c:45:62:d9:
         e1:50:18:60:0d:22:b9:6c:65:50:fe:e5:fc:37:d2:6c:bb:cc:
         fc:40:a2:cc:3b:76:e8:f9:1e:41:bf:56:8f:ca:2d:81:6e:16:
         a1:61:c2:7d:10:6a:c8:02:20:83:4f:52:08:0b:a6:61:ea:f4:
         9e:2e:33:52:17:c0:49:ef:6e:48:17:02:5d:68:6b:92:ff:30:
         d6:04:24:86:00:97:dd:f5:70:b4:f6:1d:3b:b9:82:c7:86:51:
         bd:e1:c8:86:56:d5:7d:65:9f:d0:5d:47:1c:fd:0f:99:58:09:
         bd:01:5c:e0:81:14:f0:55:23:36:b7:80:2c:78:06:6c:10:03:
         a8:c4:03:fc:53:5b:fb:ea:1f:4b:b0:ba:ef:c6:da:d6:c6:a3:
         91:f8:62:ac:26:b5:e4:25:89:11:25:b4:63:a0:16:4d:30:f4:
         9e:db:5f:ec:b6:50:ce:26:0c:44:ea:cb:33:01:a6:ef:93:e1:
         5b:68:7b:51:7a:3d:08:1d:4b:02:f6:4d:be:fc:74:9f:bb:b6:
         86:31:6b:1f:89:5a:05:b0:dc:6f:32:95:eb:f6:a8:f9:da:08:
         92:3c:5b:d4
quinndiggity commented 3 years ago

curl 'https://keybase.io/_/api/1.0/user/lookup.json?usernames=...&fields=public_keys'

{"status":{"code":100,"desc":"bad list value","fields":{"usernames":"bad list value"},"name":"INPUT_ERROR"}}

username redacted; note, no CA issues due to system trust chain

quinndiggity commented 3 years ago

openssl s_client -showcerts -connect keybase.io:443 -servername keybase.io

    ...
    Start Time: 1621960286
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no

Verify return code: 0 (ok), so it seems to be an issue originating with hashicorp/terraform-provider-consul (vs, local workstation's truststore or other similar details)

quinndiggity commented 3 years ago

Noticing that https://github.com/hashicorp/terraform-plugin-sdk had internal/vault/helper removed recently (https://github.com/hashicorp/terraform-plugin-sdk/commit/db1d38ffc8c79fe75184a701aca4416b260ef867) - I'm not certain where the best location to report this issue is, but will file on hashicorp/terraform-plugin-sdk as well just in case

quinndiggity commented 3 years ago

Guh, this may be due to flakiness on keybase.io 's side šŸ˜© I am able to get the error to occur when using curl, but inconsistently (terraform makes a dozen or so calls, so it's easy to see why the flakiness would affect it consistently):

curl -vvvv -i 'https://keybase.io/_/api/1.0/user/lookup.json?usernames=...&fields=public_keys'

*   Trying 34.192.7.122:443...
* Connected to keybase.io (34.192.7.122) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=keybase.io
*  start date: May 24 00:00:00 2021 GMT
*  expire date: May 24 23:59:59 2022 GMT
*  subjectAltName: host "keybase.io" matched cert's "keybase.io"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
...
{"status":{"code":0,...

curl -vvvv -i 'https://keybase.io/_/api/1.0/user/lookup.json?usernames=polymesh_network&fields=public_keys'

*   Trying 34.192.7.122:443...
* Connected to keybase.io (34.192.7.122) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

wait a few seconds for keepalive to timeout, try again, and 50/50 chance of 200/valid or x509/unable to get local issuer certificate šŸ˜’

quinndiggity commented 3 years ago

yeah, looks like keybase.io has updated only a portion of their instances šŸ˜© as to where to report this... curl -k -vvvv -i 'https://keybase.io/_/api/1.0/user/lookup.json?usernames=...&fields=public_keys'

*   Trying 34.192.7.122:443...
* Connected to keybase.io (34.192.7.122) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=keybase.io
*  start date: May 24 00:00:00 2021 GMT
*  expire date: May 24 23:59:59 2022 GMT
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ac6e16d580)
> GET /_/api/1.0/user/lookup.json?usernames=...&fields=public_keys HTTP/2
> Host: keybase.io
> user-agent: curl/7.74.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
HTTP/2 200 
< date: Tue, 25 May 2021 17:52:16 GMT
date: Tue, 25 May 2021 17:52:16 GMT
< content-type: application/json; charset=utf-8
content-type: application/json; charset=utf-8
< content-length: 8818
content-length: 8818
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-allow-methods: GET
access-control-allow-methods: GET
< access-control-allow-headers: Content-Type, Authorization, Content-Length, X-Requested-With
access-control-allow-headers: Content-Type, Authorization, Content-Length, X-Requested-With
< access-control-allow-credentials: false
access-control-allow-credentials: false
< cache-control: no-store
cache-control: no-store
< etag: W/"2272-pOxrMOt8G06rdQ9lPtwOLq8BJM4"
etag: W/"2272-pOxrMOt8G06rdQ9lPtwOLq8BJM4"
< strict-transport-security: max-age=31536000; includeSubdomains; preload
strict-transport-security: max-age=31536000; includeSubdomains; preload

< 
{"status":{"code":0,"name":"OK"}...
quinndiggity commented 3 years ago

Workaround for anyone running into this in the mean time (swap ... for your keybase username): curl -s -k 'https://keybase.io/_/api/1.0/user/lookup.json?usernames=...&fields=public_keys' | jq -r .them[0].public_keys.primary.bundle | gpg --dearmor | base64 --wrap=0 and use the base64 encoded value in place of:

- pgp_key     = "keybase:..."
+ pgp_key     = "  ___base64 encoded value___   "
quinndiggity commented 3 years ago

or: curl -s -k 'https://keybase.io/_/api/1.0/user/lookup.json?usernames=some-keybase-username&fields=public_keys' | jq -r .them[0].public_keys.primary.bundle | gpg --dearmor > some-keybase-username.pgp and:

pgp_key = filebase64("${path.module}/some-keybase-username.pgp")
remilapeyre commented 3 years ago

Hi @quinndiggity, I noticed this issue too and had found the issue you reported on their bug tracker. I find it weird that this post dates from 2016 and that the cert has only been deployed to some of their servers, for now many of them are still using a valid cert that expires in 2022. I will contact them to find out more.

remilapeyre commented 3 years ago

Ok, I found https://github.com/keybase/keybase-issues/issues/4033 which indicates that this was an issue with their servers and not the client we are using. I reintroduced the test in the CI and everything pass now: https://github.com/remilapeyre/terraform-provider-consul/runs/2673539272 so it seems that the issue has been fixed.

I will close this issue for now, since we have a test of the keybase integration in our unit tests we should catch it if this happens again. Please let me know if you still encounter the bug.