Provide client certs for machine id for direct connections to database

[stevenGravy]

What would you like Teleport to do?

Provide a documented or built method to get machine id client certs to connect directly to a database that trusts the Teleport CA. Currently you can configure Teleport database service and receive certs for that database access in machineid through the proxy. Those certs do not allow for direct access to that database.

What problem does this solve?

In certain cases for performance needs directly connecting to a resource like a database is required over going through a proxy and then the database service. Utilizing machine id certs that would allow short lived credentials while allowing for direct connections to the database resource.

Reproduce steps for attempting direct connection steps:

1. Configure a Self-Hosted Postgres database with database access as in https://goteleport.com/docs/database-access/guides/postgres-self-hosted/.

   For example for a dbserver issue certificates for SSL access

    tctl auth sign --format=db --host=dbserver--out=server --ttl=2190h

   Install the certs into Self-Hosted postgres. Confirm that you can connect via the database service via tsh db connect to the postgres user following all the steps

2. Configure a machine-id database access

   Make sure the machine-id host can directly connect to the database host and port for the Postgres db

   Follow this guide https://goteleport.com/docs/machine-id/access-guides/databases/ to issue a machine-id robot that for that database configured in the first step

   Confirm that postgres can ping via the https://goteleport.com/docs/machine-id/access-guides/databases/#step-44- configure-the-client-to-connect-to-the-database Go example with the Proxy.

3. Attempt a direct connection

   Now change instead of the Proxy to the direct connection to the database host and port in the Go example. Recompile and run. The output should look as the following.

   2024/02/28 21:38:50 Failed to Ping database: failed to connect to `host=dbserver user=postgres database=postgres`: 
   server error (FATAL: connection requires a valid client certificate (SQLSTATE 28000)).

[GavinFrazar]

I don't think we should implement this, because it's a big security issue.

Databases trust the db_client CA. Currently, the only Teleport component that uses db_client issued certs is the Teleport database service itself, and it uses an ephemeral cert/key pair for each connection (which it only keeps in memory), and only after it has checked Teleport RBAC to ensure the connection is allowed.

The problem with saving db_client certs to disk is they can then be used to access any database that trusts the db_client CA (for some databases like postgres it'll at least check the CN matches the internal db user), but this means that a machine id setup to get certs for database A, if compromised, could use those certs to access database B, even if the machine id identity never had access to that database in Teleport RBAC.

This is different from machine-id's current behavior that issues user CA certs, as those certs can only be used to access via Teleport connection, meaning that RBAC will always be enforced for them, so even if those certs get compromised the blast radius is limited to what that user has access to. i.e. if the machine id identity can only get certs for resource A, then those certs can only be used for resource A.

The workaround is to bring your own CA and make your database trust that CA and Teleport's db_client CA. We could improve here by letting user's manage custom CAs in Teleport, so they can easily create one custom CA per database (or cluster, like CockroachDB or Redis cluster):

• https://github.com/gravitational/teleport/issues/36537

[klizhentas]

@stevenGravy we should not then use db type certs, and treat the whole flow end to end as user creating their own certs and then, setting up Postgres as any other microservice for Machine ID, not DB specific, makes sense?

[GavinFrazar]

@klizhentas could you clarify what you mean? I don't understand this part specifically:

treat the whole flow end to end as user creating their own certs and then, setting up Postgres as any other microservice for Machine ID

Do you mean users issuing their own certs with some non-Teleport CA?

[lakshmna_Visainc]

This is what I understood, the Teleport Machine ID is generating certificates issued by the Teleport User CA. If we can configure the DB Server side (Postgres) certificates to also be issued by the same User CA, instead of the DB CA, it should work.

Like below, tctl auth sign --format=user --host=sl73vbtsdbd003.visa.com --out=server --ttl=2190h

Please correct if my understand is wrong.

Regards, Nambi

From: Gavin Frazar @.> Date: Tuesday, 19 March 2024 at 18:48 To: gravitational/teleport @.> Cc: Lakshmanan, Nambi @.>, Manual @.> Subject: Re: [gravitational/teleport] Provide client certs for machine id for direct connections to database (Issue #38774)

@klizhentashttps://github.com/klizhentas could you clarify what you mean? I don't understand this part specifically:

treat the whole flow end to end as user creating their own certs and then, setting up Postgres as any other microservice for Machine ID

Do you mean users issuing their own certs with some non-Teleport CA?

— Reply to this email directly, view it on GitHubhttps://github.com/gravitational/teleport/issues/38774#issuecomment-2007898502, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BDRURCOG2EH4C3VS47N4RRDYZCCADAVCNFSM6AAAAABD7A6476VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXHA4TQNJQGI. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[strideynet]

Hey Folks,

I'm fairly sure the new Workload Identity SPIFFE CA is probably the right route to pursue here. We can issue certificates with controllable CN from the SPIFFE CA. We can configure postgres to trust the SPIFFE CA for user authentication and then leverage https://www.postgresql.org/docs/current/auth-username-maps.html to map the SPIFFE client certs to the appropriate database user.

I have a few questions:

• Is this just for Postgres?
• Will standard Teleport Database Access also be used to connect to these databases?

[stevenGravy]

Hey Folks,

I'm fairly sure the new Workload Identity SPIFFE CA is probably the right route to pursue here. We can issue certificates with controllable CN from the SPIFFE CA. We can configure postgres to trust the SPIFFE CA for user authentication and then leverage https://www.postgresql.org/docs/current/auth-username-maps.html to map the SPIFFE client certs to the appropriate database user.

I have a few questions:

• Is this just for Postgres?
• Will standard Teleport Database Access also be used to connect to these databases?

The initial testing of this was Postgres. MySQL was the next priority. I've sent you in a email the full list.

Yes users could also require Teleport Database Access.

[strideynet]

Rough notes from my personal experiment with #40150 .

postgresql.conf

ssl_ca_file = '/opt/pg-stuff/svid_bundle.pem'

pg_hba.conf

hostssl all all 0.0.0.0/0 cert map=spiffe

pg_ident.conf

spiffe  /^(.*)\.mydb\.internal\.example\.com$ \1

To support both direct and proxied database access, the direct access usernames will need to be namespaced (e.g direct-) as pg_hba is fairly limited and the first matching block is used rather than falling through where multiple may match. Equally, IP based control could be used (only use the teleport database access where the originating IP is the db access agent).