Certificate authentication with colon-separated SAN URI fails in 22.1.6

[jonstjohn]

Describe the problem

If a client certificate that includes a colon-separated SAN URI is used to authenticate with CRDB in 22.1.6, the following errors occurs:

ERROR: invalid tenant URI SAN mycompany:sv:rootclient:dev:usw1

This issue appeared in 22.1.6 and looks like it is related to https://github.com/cockroachdb/cockroach/pull/84371 .

To Reproduce

Generate a client certificate with a colon-separated SAN URI.

Setup a simple directory structure to hold keys and configuration materials:

mkdir auth auth/certs auth/openssl auth/safe

Put the cluster's ca.key in auth/safe and ca.crt in auth/certs.

Initialize the openssl index and serial files:

touch auth/openssl/index.txt
echo '01' > auth/openssl/serial.txt

Create a ca configuration file in auth/openssl/ca.cnf:

# OpenSSL CA configuration file
[ ca ]
default_ca = CA_default

[ CA_default ]
default_days = 365
database = auth/openssl/index.txt
serial = auth/openssl/serial.txt
default_md = sha256
copy_extensions = copy
unique_subject = no

# Used to create the CA certificate.
[ req ]
prompt=no
distinguished_name = distinguished_name
x509_extensions = extensions

[ distinguished_name ]
organizationName = Cockroach
commonName = Cockroach CA

[ extensions ]
keyUsage = critical,digitalSignature,nonRepudiation,keyEncipherment,keyCertSign
basicConstraints = critical,CA:true,pathlen:1

# Common policy for nodes and users.
[ signing_policy ]
organizationName = supplied
commonName = optional

# Used to sign node certificates.
[ signing_node_req ]
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth

# Used to sign client certificates.
[ signing_client_req ]
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = clientAuth

Create a certificate configuration file inauth/openssl/sanuri.client.cnf:

[ req ]
prompt=no
distinguished_name = distinguished_name
req_extensions = extensions

[ distinguished_name ]
organizationName = Cockroach
commonName = sanuri

[ extensions ]
subjectAltName = DNS:root,URI:mycompany:sv:rootclient:dev:usw1

Next, generate a certificate using openssl and the cluster's ca key:

openssl genrsa -out auth/certs/client.sanuri.key 2048
openssl req -new -config auth/openssl/sanuri.client.cnf -key auth/certs/client.sanuri.key -out auth/openssl/client.sanuri.csr -batch
openssl ca -config auth/openssl/ca.cnf -keyfile auth/safe/ca.key -cert auth/certs/ca.crt -policy signing_policy -extensions signing_client_req -out auth/certs/client.sanuri.crt -outdir auth/certs/ -in auth/openssl/client.sanuri.csr -batch

Inspect the certificate:

openssl x509 -in auth/certs/client.sanuri.crt -text -noout | less

Create the user on the cluster:

CREATE USER sanuri;
GRANT ADMIN to sanuri;

Try to login using the cockroach client (where $CRDB_IP_EXTERNAL is the accessible IP):

cockroach sql --url "postgres://$CRDB_IP_EXTERNAL:26257?sslcert=.%2Fauth%2Fcerts%2Fclient.sanuri.crt&sslkey=.%2Fauth%2Fcerts%2Fclient.sanuri.key&sslmode=verify-full&sslrootcert=.%2Fauth%2Fcerts%2Fca.crt"

Note the error message.

ERROR: invalid tenant URI SAN mycompany:sv:rootclient:dev:usw1

For the login to be successful in 22.1.5 and earlier, a cert-principal-map and/or identity map needs to be configured, but the error is present even without that.

Expected behavior

This should work as it does in 22.1.5 and earlier versions.

Additional data / screenshots

None

Environment:

• CockroachDB version: 22.1.6
• Server OS: n/a
• Client app: cockroach client

Additional context

Unable to upgrade to 22.1.6.

Certificates are generated via a company-wide policy and the format cannot be changed.

Jira issue: CRDB-19225

[abarganier]

Thanks for the repro steps - I have a fix prepared & should have a PR up shortly. We'll have to backport this into v22.1.7 or do an extraordinary release to get it into v22.1. @jonstjohn let me know if you have a preference here (if they can fallback to v22.1.5 until v22.1.7, that's preferable, but if it's high priority we can expedite).

The v22.1.7 release is scheduled for publish on September 12th currently, but it depends on whether we can get the original PR + backport merged before September 6th (might be cutting it close...).

If we miss it, we can try pushing for an extraordinary release if it's time sensitive, or we'd wait for v22.1.8, which is scheduled for publish on October 3rd.

---

The fix itself is to simply relax our requirement on the URI SAN scheme/format. If we don't recognize the format, instead of erroring out and exiting, we'll fallback to the legacy behavior.

I did some tests - first, to reproduce:

$ ./cockroach sql --url "postgres://localhost:26257?sslcert=.%2Fauth%2Fcerts%2Fclient.sanuri.crt&sslkey=.%2Fauth%2Fcerts%2Fclient.sanuri.key&sslmode=verify-full&sslrootcert=.%2Fauth%2Fcerts%2Fca.crt"
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
ERROR: invalid tenant URI SAN mycompany:sv:rootclient:dev:usw1
SQLSTATE: 28000
Failed running "sql"

Then, with my proposed changes:

$ ./cockroach sql --url "postgres://localhost:26257?sslcert=.%2Fauth%2Fcerts%2Fclient.sanuri.crt&sslkey=.%2Fauth%2Fcerts%2Fclient.sanuri.key&sslmode=verify-full&sslrootcert=.%2Fauth%2Fcerts%2Fca.crt"
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v22.2.0-alpha.1-256-g9911916662-dirty (x86_64-apple-darwin21.6.0, built 2022/09/01 17:27:18, go1.18.4) (same version as client)
# Cluster ID: 2ab40009-e4cd-4e7d-8588-c87a88d9adcc
#
# Enter \? for a brief introduction.
#
root@localhost:26257/defaultdb>

I'm going to polish this up and get a PR out - I'll do my best to move quickly!

[jonstjohn]

Thanks for the update @abarganier ! Yes, they can use 22.1.5 until the fix comes out. Ideally, we would have the fix in 22.1.7 but if that isn't possible, a fix in 22.1.8 is acceptable as there isn't a pressing need to upgrade at the moment.

[blathers-crl[bot]]

Hi @knz, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[blathers-crl[bot]]

Hi @knz, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[knz]

this is a release blocker.

[abarganier]

@jonstjohn this has been resolved for both 22.2, and the next 22.1 release (22.1.8 22.1.7).

We now look through all the URI SANs included in the certificate for one with the expected crdb:// prefix. If we find a prefix match, we attempt to parse the URI SAN as before, in which case we take a strict approach where we throw an error if the format isn't the expected crdb://tenant/<tenant_id>/user/<tenant_username>.

In the case where none of the provided URI SANs have the expected crdb:// prefix, such as in your example, only then do we fallback to the global cert.

Thanks for reporting this issue. Marking as closed.

[jonstjohn]

Thanks @abarganier and others! Really appreciate the turn-around!

[rafiss]

update: v22.1.7 will include this change, since the release schedule shifted slightly which allowed this fix to make it in. it's currently targeted to go out next week.

[abarganier]

Great news @rafiss! Thanks for clarifying.