Machine ID: Improve the dynamic access story

[strideynet]

The Problem

Right now, using Machine ID dynamically or with complex environments is problematic. This is because output credentials for Kubernetes Access, Database Access and Application Access must be pre-encoded with the target resource. This means if you are building out CI that needs to access a number of resources, you end up with a very complex configuration file with a large number of outputs. If you do not necessarily know the resources ahead of time, you need to repeatedly invoke tbot with a generated configuration for each resource.

Since we disable re-issue on the output credentials, it's not possible to naively use these with tsh (and it's built in re-issuing behaviour) as one might expect from their prior experience with tsh.

Solutions

Support access without Route attributes encoded in certs

• https://github.com/gravitational/teleport/issues/40405

Allow certificate re-issue

The Machine ID RFD specifies that re-issuing should be disabled:

Additionally, to further restrict renewals, impersonated certificates will be generated with a "disallow-reissue" flag to entirely prevent malicious renewals of a compromised secondary / end-user certificate.

Benefits of enabling re-issue:

• We are less reliant on building support into tbot (e.g tbot proxy db) - we can keep these for more specific machine-only use-cases and allow people to use the standard tsh commands. This also means that when new features are introduced to tsh, they'll immediately be available to machine users - it also means we can have specific use-cases in mind when adding commands to tbot.
• Dynamic use-cases are much simpler. Users can use tbot to produce a short-lived re-issuable identity and then use this freely with the existing behaviour of tsh to access any resource the roles of the identity allow.

Problems with enabling re-issue:

• tsh may need to be adjusted to provide a more natural experience when working with an identity file.
• tsh may need to be adjusted to behave better in a non-interactive environment.
• Users may have come to expect/rely on this disabled re-issue behaviour (e.g outputting credentials that are "locked" to a specific cluster). We can mitigate this by making this a new output type that users opt in to.
• Security risk as identified in the RFD - we'd need to ensure certs could not "escape" their role-impersonated state.

We would retain the existing output types for those who do not need dynamic access and would benefit from the generated configuration files available (e.g the generated kubeconfig or ssh_config) easily with Machine ID.

It ultimately appears to come down to whether or not we want Machine ID to be a separate and clean-cut from other Teleport tooling, or if we wish to richly integrate with our existing tools.

Machine ID Socket API

TBot could expose a socket hosting a gRPC service which could be used by commands that require "once off" credentials. tbot would then run the background as a service and maintain control over the issuing of certificates.

This is covered in more details by a draft RFD: https://github.com/gravitational/teleport/pull/24286

One off generation commands

This could be used with commands on tbot to produce credentials

# With a valid `tbot` instance already running on a host and exposing the local access gRPC API
curl  \
  --cert $(tbot single-use apps --format=cert --app=myapp) \
  --key $(tbot single-use apps --format=key --app=myapp) \
    myapp.teleport.example.com/api/users

tsh Integration

We could adapt the internals of tsh to support using this API when re-issuing is required. This allows people to enjoy the experience of tsh they are already familiar with - but avoid actually making our certificate re-issuable.

[pQraus]

Hi, dynamic application access via machine ID would be very nice for us. We have multiple (edge) devices. A REST API is deployed on every device to control them. We can (only) reach these APIs via teleport applications. All of them have the same label (kind: control-api). To control all devices automatically we would like to use a teleport machine ID identity which provides us access to all teleport applications with the label kind: control-api. Therefore, it would be nice if the machine ID identity creates one certificate for our edge-device-controller to access the REST APIs (see the picture below). Unfortunately, the machine ID can only create a certificate for one application. Currently we have ~10 edge devices but the number will increase. So, it would be nice if the machine id creates dynamically a new certificate when a new application with the label kind: control-api is joining the cluster.

(Initial slack thread https://goteleport.slack.com/archives/CEZH6UL64/p1686221633885649)

[strideynet]

Rewritten this ticket to cover the dynamic access story as a whole rather than specifically app access.

[pgreidanusupgrade]

My use case is similar to above, where we have a distributed set of Prometheus Exporter endpoints with a Teleport agent in the same network as the exporters, there will be multiple exporters per agent. Then we want to use Tbot to create a certificate that Prometheus at a central system can use to scrape via Teleport. We are looking upwards of 50+ exporters, so static configuration of one output exporter, which would be one secret per exporter, would be to much of a mess on the Tbot and the Prometheus configuration. Exporters and sites can come in and out dynamically, so static configuration would be a never ending chore.

Ideally we should have a single App certificate that can access all services that the bot has permissions to, as well as a second secret we can use to access the teleport API and get the list of those services, which would then create the prometheus configuration via a custom coded sidecar.

I don't see how reissue comes into play with my scenario however, all I would really want to see is a certificate generated that isn't restricted to a single app, which is the certificates that are generated from tsh/web?