Teleport 8.0 Test Plan

[russjones]

Manual Testing Plan

Below are the items that should be manually tested with each release of Teleport. These tests should be run on both a fresh install of the version to be released as well as an upgrade of the previous version of Teleport.

• [x] Adding nodes to a cluster @atburke

  • [x] Adding Nodes via Valid Static Token
  • [x] Adding Nodes via Valid Short-lived Tokens
  • [x] Adding Nodes via Invalid Token Fails
  • [x] Revoking Node Invitation

• [x] Labels @atburke

  • [x] Static Labels
  • [x] Dynamic Labels

• [x] Trusted Clusters @atburke

  • [x] Adding Trusted Cluster Valid Static Token
  • [x] Adding Trusted Cluster Valid Short-lived Token
  • [x] Adding Trusted Cluster Invalid Token
  • [x] Removing Trusted Cluster

• [x] RBAC @codingllama

  Make sure that invalid and valid attempts are reflected in audit log.

  • [x] Successfully connect to node with correct role
  • [x] Unsuccessfully connect to a node in a role restricting access by label
  • [x] Unsuccessfully connect to a node in a role restricting access by invalid SSH login
  • [x] Allow/deny role option: SSH agent forwarding
  • [x] Allow/deny role option: Port forwarding

• [x] Verify that custom PAM environment variables are available as expected. @xacrimon

• [x] Users @nklaassen With every user combination, try to login and signup with invalid second factor, invalid password to see how the system reacts.

  • [x] Adding Users Password Only
  • [x] Adding Users OTP
  • [x] Adding Users U2F
  • [x] Adding Users WebAuthn
  • [x] Managing MFA devices
  • [x] Add an OTP device with tsh mfa add
  • [x] Add a U2F device with tsh mfa add
  • [x] Verify that the U2F device works under WebAuthn
  • [x] Add a WebAuthn device with tsh mfa add
  • [x] List MFA devices with tsh mfa ls
  • [x] Remove an OTP device with tsh mfa rm
  • [x] Remove a U2F device with tsh mfa rm
  • [x] Remove a WebAuthn device with tsh mfa rm
  • [x] Attempt removing the last MFA device on the user
    • [x] with second_factor: on in auth_service, should fail
    • [x] with second_factor: optional in auth_service, should succeed
  • [x] #8861
  • [x] Login with MFA
  • [x] Add 2 OTP and 2 WebAuthn devices with tsh mfa add
  • [x] Login via OTP
  • [x] Login via WebAuthn
  • [x] Login OIDC
  • [x] Login SAML
  • [x] Login GitHub
  • [x] Deleting Users

• [x] Backends @Joerger

  • [x] Teleport runs with etcd
  • [x] Teleport runs with dynamodb
  • [x] Teleport runs with SQLite
  • [x] Teleport runs with Firestore

• [x] Session Recording @codingllama

  • [x] Session recording can be disabled
  • [x] Sessions can be recorded at the node
  • [x] Sessions in remote clusters are recorded in remote clusters
  • [x] Sessions can be recorded at the proxy
  • [x] Sessions on remote clusters are recorded in the local cluster
  • [x] Enable/disable host key checking.

• [ ] Audit Log @quinqu

  • [x] Failed login attempts are recorded
  • [x] Interactive sessions have the correct Server ID
  • [x] Server ID is the ID of the node in "session_recording: node" mode
  • [x] Server ID is the ID of the proxy in "session_recording: proxy" mode

  Node/Proxy ID may be found at /var/lib/teleport/host_uuid in the corresponding machine.

  Node IDs may also be queried via tctl nodes ls.

  • [x] Exec commands are recorded
  • [x] scp commands are recorded
  • [x] Subsystem results are recorded

  Subsystem testing may be achieved using both Recording Proxy mode and OpenSSH integration.

  Assuming the proxy is proxy.example.com:3023 and node1 is a node running OpenSSH/sshd, you may use the following command to trigger a subsystem audit log:

  sftp -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p" root@node1

• [x] Interact with a cluster using tsh @tcsc

  These commands should ideally be tested for recording and non-recording modes as they are implemented in a different ways.

  • [x] tsh ssh \<regular-node>
  • [x] tsh ssh \<node-remote-cluster>
  • [x] tsh ssh -A \<regular-node>
  • [x] tsh ssh -A \<node-remote-cluster>
  • [x] tsh ssh \<regular-node> ls
  • [x] tsh ssh \<node-remote-cluster> ls
  • [x] tsh join \<regular-node>
  • [x] tsh join \<node-remote-cluster>
  • [x] tsh play \<regular-node>
  • [x] tsh play \<node-remote-cluster>
  • [x] tsh scp \<regular-node>
  • [x] tsh scp \<node-remote-cluster>
  • [x] tsh ssh -L \<regular-node>
  • [x] tsh ssh -L \<node-remote-cluster>
  • [x] tsh ls
  • [x] tsh clusters

• [x] Interact with a cluster using ssh @quinqu Make sure to test both recording and regular proxy modes.

  • [x] ssh \<regular-node>
  • [x] ssh \<node-remote-cluster>
  • [x] ssh -A \<regular-node>
  • [x] ssh -A \<node-remote-cluster>
  • [x] ssh \<regular-node> ls
  • [x] ssh \<node-remote-cluster> ls
  • [x] scp \<regular-node>
  • [x] scp \<node-remote-cluster>
  • [x] ssh -L \<regular-node>
  • [x] ssh -L \<node-remote-cluster>

• [x] Interact with a cluster using the Web UI @tcsc

  • [x] Connect to a Teleport node
  • [x] Connect to a OpenSSH node
  • [x] Check agent forwarding is correct based on role and proxy mode. (See note below - @tcsc)

User accounting @xacrimon

• [x] Verify that active interactive sessions are tracked in /var/run/utmp on Linux.
• [x] Verify that interactive sessions are logged in /var/log/wtmp on Linux.

Combinations @tcsc

For some manual testing, many combinations need to be tested. For example, for interactive sessions the 12 combinations are below.

• [x] Connect to a OpenSSH node in a local cluster using OpenSSH.
• [x] Connect to a OpenSSH node in a local cluster using Teleport.
• [x] Connect to a OpenSSH node in a local cluster using the Web UI.
• [x] Connect to a Teleport node in a local cluster using OpenSSH.
• [x] Connect to a Teleport node in a local cluster using Teleport.
• [x] Connect to a Teleport node in a local cluster using the Web UI.
• [x] Connect to a OpenSSH node in a remote cluster using OpenSSH.
• [x] Connect to a OpenSSH node in a remote cluster using Teleport.
• [x] Connect to a OpenSSH node in a remote cluster using the Web UI.
• [x] Connect to a Teleport node in a remote cluster using OpenSSH.
• [x] Connect to a Teleport node in a remote cluster using Teleport.
• [x] Connect to a Teleport node in a remote cluster using the Web UI.

Teleport with EKS/GKE @smallinsky

• [x] Deploy Teleport on a single EKS cluster
• [x] Deploy Teleport on two EKS clusters and connect them via trusted cluster feature
• [x] Deploy Teleport Proxy outside of GKE cluster fronting connections to it (use this script to generate a kubeconfig)
• [x] Deploy Teleport Proxy outside of EKS cluster fronting connections to it (use this script to generate a kubeconfig)

Teleport with multiple Kubernetes clusters @r0mant

Note: you can use GKE or EKS or minikube to run Kubernetes clusters. Minikube is the only caveat - it's not reachable publicly so don't run a proxy there.

• [x] Deploy combo auth/proxy/kubernetes_service outside of a Kubernetes cluster, using a kubeconfig
  • [x] Login with tsh login, check that tsh kube ls has your cluster
  • [x] Run kubectl get nodes, kubectl exec -it $SOME_POD -- sh
  • [x] Verify that the audit log recorded the above request and session
• [x] Deploy combo auth/proxy/kubernetes_service inside of a Kubernetes cluster
  • [x] Login with tsh login, check that tsh kube ls has your cluster
  • [x] Run kubectl get nodes, kubectl exec -it $SOME_POD -- sh
  • [x] Verify that the audit log recorded the above request and session
• [x] Deploy combo auth/proxy_service outside of the Kubernetes cluster and kubernetes_service inside of a Kubernetes cluster, connected over a reverse tunnel
  • [x] Login with tsh login, check that tsh kube ls has your cluster
  • [x] Run kubectl get nodes, kubectl exec -it $SOME_POD -- sh
  • [x] Verify that the audit log recorded the above request and session
• [x] Deploy a second kubernetes_service inside of another Kubernetes cluster, connected over a reverse tunnel
  • [x] Login with tsh login, check that tsh kube ls has both clusters
  • [x] Switch to a second cluster using tsh kube login
  • [x] Run kubectl get nodes, kubectl exec -it $SOME_POD -- sh on the new cluster
  • [x] Verify that the audit log recorded the above request and session
• [x] Deploy combo auth/proxy/kubernetes_service outside of a Kubernetes cluster, using a kubeconfig with multiple clusters in it
  • [x] Login with tsh login, check that tsh kube ls has all clusters
• [x] Test Kubernetes screen in the web UI (tab is located on left side nav on dashboard):
  • [x] Verify that all kubes registered are shown with correct name and labels
  • [x] Verify that clicking on a rows connect button renders a dialogue on manual instructions with Step 2 login value matching the rows name column
  • [x] Verify searching for name or labels in the search bar works
  • [x] Verify you can sort by name colum

Teleport with FIPS mode @russjones

• [x] Perform trusted clusters, Web and SSH sanity check with all teleport components deployed in FIPS mode.

ACME @Joerger

• [x] Teleport can fetch TLS certificate automatically using ACME protocol.

Migrations @r0mant @russjones

• [x] Migrate trusted clusters from 2.4.0 to 2.5.0
  • [x] Migrate auth server on main cluster, then rest of the servers on main cluster SSH should work for both main and old clusters
  • [x] Migrate auth server on remote cluster, then rest of the remote cluster SSH should work

Command Templates

When interacting with a cluster, the following command templates are useful:

OpenSSH

# when connecting to the recording proxy, `-o 'ForwardAgent yes'` is required.
ssh -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p" \
  node.example.com

# the above command only forwards the agent to the proxy, to forward the agent
# to the target node, `-o 'ForwardAgent yes'` needs to be passed twice.
ssh -o "ForwardAgent yes" \
  -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p" \
  node.example.com

# when connecting to a remote cluster using OpenSSH, the subsystem request is
# updated with the name of the remote cluster.
ssh -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p@foo.com" \
  node.foo.com

Teleport

# when connecting to a OpenSSH node, remember `-p 22` needs to be passed.
tsh --proxy=proxy.example.com --user=<username> --insecure ssh -p 22 node.example.com

# an agent can be forwarded to the target node with `-A`
tsh --proxy=proxy.example.com --user=<username> --insecure ssh -A -p 22 node.example.com

# the --cluster flag is used to connect to a node in a remote cluster.
tsh --proxy=proxy.example.com --user=<username> --insecure ssh --cluster=foo.com -p 22 node.foo.com

Teleport with SSO Providers @benarent

• [x] G Suite install instructions work
  • [ ] G Suite Screenshots are up to date
• [ ] ActiveDirectoy install instructions work
  • [x] Active Directoy Screenshots are up to date
• [x] Okta install instructions work
  • [x] Okta Screenshots are up to date
• [x] OneLogin install instructions work
  • [x] OneLogin Screenshots are up to date
• [x] OIDC install instructions work
  • [x] OIDC Screenshots are up to date

Teleport Plugins @Joerger

• [x] Test receiving a message via Teleport Slackbot
• [x] Test receiving a new Jira Ticket via Teleport Jira

WEB UI @kimlisa @rudream @gzdunek

Main

For main, test with a role that has access to all resources.

Top Nav

• [x] Verify that cluster selector displays all (root + leaf) clusters
• [x] Verify that user name is displayed
• [x] Verify that user menu shows logout, help&support, and account settings (for local users)

Side Nav

• [x] Verify that each item has an icon
• [x] Verify that Collapse/Expand works and collapsed has icon >, and expand has icon v
• [x] Verify that it automatically expands and highlights the item on page refresh

Servers aka Nodes

• [x] Verify that "Servers" table shows all joined nodes
• [x] Verify that "Connect" button shows a list of available logins
• [x] Verify that "Hostname", "Address" and "Labels" columns show the current values
• [x] Verify that "Search" by hostname, address, labels works
• [x] Verify that terminal opens when clicking on one of the available logins
• [x] Verify that clicking on Add Server button renders dialogue set to Automatically view
  • [x] Verify clicking on Regenerate Script regenerates token value in the bash command
  • [x] Verify using the bash command successfully adds the server (refresh server list)
  • [x] Verify that clicking on Manually tab renders manual steps
  • [x] Verify that clicking back to Automatically tab renders bash command

Applications

• [x] Verify that clicking on Add Application button renders dialogue
  • [x] Verify input validation (prevent empty value and invalid url)
  • [x] Verify after input and clicking on Generate Script, bash command is rendered
  • [x] Verify clicking on Regenerate button regenerates token value in bash command

Databases

• [x] Verify that clicking on Add Database button renders dialogue for manual instructions:
  • [x] Verify selecting different options on Step 4 changes Step 5 commands

    Active Sessions

• [x] Verify that "empty" state is handled
• [x] Verify that it displays the session when session is active
• [x] Verify that "Description", "Session ID", "Users", "Nodes" and "Duration" columns show correct values
• [x] Verify that "OPTIONS" button allows to join a session

Audit log

• [x] Verify that time range button is shown and works
• [x] Verify that clicking on Session Ended event icon, takes user to session player
• [x] Verify event detail dialogue renders when clicking on events details button
• [x] Verify searching by type, description, created works

Users

• [x] Verify that users are shown
• [x] Verify that creating a new user works
• [x] Verify that editing user roles works
• [x] Verify that removing a user works
• [x] Verify resetting a user's password works
• [x] Verify search by username, roles, and type works

Auth Connectors

• [x] Verify that creating OIDC/SAML/GITHUB connectors works
• [x] Verify that editing OIDC/SAML/GITHUB connectors works
• [x] Verify that error is shown when saving an invalid YAML
• [x] Verify that correct hint text is shown on the right side
• [x] Verify that encrypted SAML assertions work with an identity provider that supports it (Azure).

Auth Connectors Card Icons

• [x] Verify that GITHUB card has github icon
• [x] Verify that SAML card has SAML icon
• [x] Verify that OIDC card has OIDC icon
• [x] Verify when there are no connectors, empty state renders

Roles

• [x] Verify that roles are shown
• [x] Verify that "Create New Role" dialog works
• [x] Verify that deleting and editing works
• [x] Verify that error is shown when saving an invalid YAML
• [x] Verify that correct hint text is shown on the right side

Managed Clusters

• [x] Verify that it displays a list of clusters (root + leaf)
• [x] Verify that every menu item works: nodes, apps, audit events, session recordings.

Help & Support

• [x] Verify that all URLs work and correct (no 404)

Access Requests

Creating Access Requests

1. Create a role with limited permissions (defined below as allow-roles). This role allows you to see the Role screen and ssh into all nodes.
2. Create another role with limited permissions (defined below as allow-users). This role session expires in 4 minutes, allows you to see Users screen, and denies access to all nodes.
3. Create another role with no permissions other than being able to create requests (defined below as default)
4. Create a user with role default assigned
5. Create a few requests under this user to test pending/approved/denied state.

   kind: role
   metadata:
   name: allow-roles
   spec:
   allow:
   logins:
   - root
   node_labels:
     '*': '*'
   rules:
   - resources:
     - role
     verbs:
     - list
     - read
   options:
   max_session_ttl: 8h0m0s
   version: v3

   kind: role
   metadata:
   name: allow-users
   spec:
   allow:
   rules:
   - resources:
     - user
     verbs:
     - list
     - read
   deny:
   node_labels:
     '*': '*'
   options:
   max_session_ttl: 4m0s
   version: v3

   kind: role
   metadata:
   name: default
   spec:
   allow:
   request:
     roles:
     - allow-roles
     - allow-users
     suggested_reviewers:
     - random-user-1
     - random-user-2
   options:
   max_session_ttl: 8h0m0s
   version: v3

   • [x] Verify that creating a new request works
   • [x] Verify that under requestable roles, only allow-roles and allow-users are listed
   • [x] Verify input validation requires at least one role to be selected
   • [x] Verify you can select/input/modify reviewers
   • [x] Verify after creating, requests are listed in pending states
   • [x] Verify you can't review own requests

Viewing & Approving/Denying Requests

Create a user with the role reviewer that allows you to review all requests, and delete them.

kind: role
version: v3
metadata:
  name: reviewer
spec:
  allow:
    review_requests:
      roles: ['*']

• [x] Verify you can view access request from request list
• [x] Verify there is list of reviewers you selected (empty list if none selected AND none wasn't defined in roles)
• [x] Verify threshold name is there (it will be default if thresholds weren't defined in role, or blank if not named)
• [x] Verify you can approve a request with message, and immediately see updated state with your review stamp (green checkmark) and message box
• [x] Verify you can deny a request, and immediately see updated state with your review stamp (red cross)
• [x] Verify deleting the denied request is removed from list

Assuming Approved Requests

• [x] Verify assume buttons are only present for approved request and for logged in user
• [x] Verify that assuming allow-roles allows you to see roles screen and ssh into nodes
• [x] Verify that after clicking on the assume button, it is disabled in both the list and in viewing
• [x] After assuming allow-roles, verify that assuming allow-users allows you to see users screen, and denies access to nodes
  • [x] Verify a switchback banner is rendered with roles assumed, and count down of when it expires
  • [x] Verify switching back goes back to your default static role
  • [x] Verify after re-assuming this role, the user is automatically logged out after the expiry is met (4 minutes)
• [x] Verify that after logging out (or getting logged out automatically) and relogging in, permissions are reset to default, and requests that are not expired and are approved are assumable again

Access Request Waiting Room

Strategy Reason

Create the following role:

kind: role
metadata:
  name: restrict
spec:
  allow:
    request:
      roles:
      - <some other role to assign user after approval>
  options:
    max_session_ttl: 8h0m0s
    request_access: reason
    request_prompt: <some custom prompt to show in reason dialogue>
version: v3

• [x] Verify after login, reason dialogue is rendered with prompt set to request_prompt setting
• [x] Verify after clicking send request, pending dialogue renders
• [x] Verify after approving a request, dashboard is rendered
• [x] Verify the correct role was assigned

Strategy Always

With the previous role you created from Strategy Reason, change request_access to always:

• [x] Verify after login, pending dialogue is rendered
• [x] Verify after approving a request, dashboard is rendered
• [x] Verify after denying a request, access denied dialogue is rendered
• [x] Verify a switchback banner is rendered with roles assumed, and count down of when it expires
• [x] Verify switchback button says Logout and clicking goes back to the login screen

Strategy Optional

With the previous role you created from Strategy Reason, change request_access to optional:

• [x] Verify after login, dashboard is rendered

Terminal

• [x] Verify that top nav has a user menu (Main and Logout)
• [x] Verify that switching between tabs works on alt+[1...9]

Node List Tab

• [x] Verify that Cluster selector works (URL should change too)
• [x] Verify that Quick launcher input works
• [x] Verify that Quick launcher input handles input errors
• [x] Verify that "Connect" button shows a list of available logins
• [x] Verify that "Hostname", "Address" and "Labels" columns show the current values
• [x] Verify that "Search" by hostname, address, labels work
• [x] Verify that new tab is created when starting a session

Session Tab

• [x] Verify that session and browser tabs both show the title with login and node name
• [x] Verify that terminal resize works
  • Install midnight commander on the node you ssh into: $ sudo apt-get install mc
  • Run the program: $ mc
  • Resize the terminal to see if panels resize with it
• [x] Verify that session tab shows/updates number of participants when a new user joins the session
• [x] Verify that tab automatically closes on "$ exit" command
• [x] Verify that SCP Upload works
• [x] Verify that SCP Upload handles invalid paths and network errors
• [x] Verify that SCP Download works
• [x] Verify that SCP Download handles invalid paths and network errors

Session Player

• [x] Verify that it can replay a session
• [x] Verify that when playing, scroller auto scrolls to bottom most content
• [x] Verify when resizing player to a small screen, scroller appears and is working
• [x] Verify that error message is displayed (enter a invalid SID in the URL)

Invite Form

• [x] Verify that input validates
• [x] Verify that invite works with 2FA disabled
• [x] Verify that invite works with OTP enabled
• [x] Verify that invite works with U2F enabled
• [x] Verify that invite works with WebAuthn enabled
• [x] Verify that error message is shown if an invite is expired/invalid

Login Form

• [x] Verify that input validates
• [x] Verify that login works with 2FA disabled
• [x] Verify that changing passwords works for 2FA disabled
• [x] Verify that login works with OTP enabled
• [x] Verify that changing passwords works for OTP enabled
• [x] Verify that login works with U2F enabled
• [x] Verify that changing passwords works for U2F enabled
• [x] Verify that login works with WebAuthn enabled
• [x] Verify that changing passwords works for WebAuthn enabled
• [x] Verify that login works for Github/SAML/OIDC
• [x] Verify that account is locked after several unsuccessful attempts
• [x] Verify that redirect to original URL works after successful login

Multi-factor Authentication (mfa)

Create/modify teleport.yaml and set the following authentication settings under auth_service

authentication:
  type: local
  second_factor: optional
  require_session_mfa: yes
  webauthn:
    rp_id: example.com

MFA invite, login, password reset, change password

• [x] Verify during invite/reset, second factor list all auth types: none, hardware key, and authenticator app
• [x] Verify registration works with all option types
• [x] Verify login with all option types
• [x] Verify changing password with all option types
• [x] Change second_factor type to on and verify that mfa is required (no option none in dropdown)

MFA require auth

Go to Account Settings > Two-Factor Devices and register a new device

Using the same user as above:

• [x] Verify logging in with registered WebAuthn key works
• [x] Verify connecting to a ssh node prompts you to tap your registered WebAuthn key
• [ ] Verify in the web terminal, you can scp upload/download files

MFA Management

• [x] Verify adding first device works without requiring re-authentication
• [x] Verify re-authenticating with a WebAuthn device works
• [x] Verify re-authenticating with a U2F device works
• [x] Verify re-authenticating with a OTP device works
• [x] Verify adding a WebAuthn device works
• [x] Verify adding a U2F device works
• [x] Verify adding an OTP device works
• [x] Verify removing a device works
• [x] Verify second_factor set to off disables adding devices

Cloud

Invite/Reset

• [x] Verify email as usernames, receives recovery codes
• [x] Verify non email usernames, receives no recovery codes

Recovery Code Management

• [x] Verify generating recovery codes for local accounts with email usernames works
• [x] Verify local accounts with non-email usernames are not able to generate recovery codes
• [x] Verify SSO accounts are not able to generate recovery codes

Recovery Flow: Add new mfa device

• [ ] Verify recovering (adding) a new hardware key device with password
• [x] Verify recovering (adding) a new otp device with password
• [x] Verify viewing and deleting any old device (but not the one just added)
• [x] Verify new recovery codes are created at the end of flow

Recovery Flow: Change password

• [x] Verify recovering password with any mfa device
• [x] Verify new recovery codes are created at the end of flow

Recovery Email

• [x] Verify receiving email for link to start recovery
• [x] Verify receiving email for successfully recovering
• [x] Verify receiving email for locked account when max attempts reached

RBAC

Create a role, with no allow.rules defined:

kind: role
metadata:
  name: test
spec:
  allow:
    app_labels:
      '*': '*'
    logins:
    - root
    node_labels:
      '*': '*'
  options:
    max_session_ttl: 8h0m0s
version: v3

• [x] Verify that a user has access only to: "Servers", "Applications", "Databases", "Kubernetes", "Active Sessions", "Access Requests" and "Manage Clusters"
• [x] Verify there is no Add Server, Application, Databases, Kubernetes button in each respective view
• [x] Verify only Nodes, Apps, Databases, and Kubernetes are listed under options button in Manage Clusters

Note: User has read/create access_request access to their own requests, despite resource settings

Add the following under spec.allow.rules to enable read access to the audit log:

  - resources:
      - event
      verbs:
      - list

• [x] Verify that the Audit Log and Session Recordings is accessible
• [x] Verify that playing a recorded session is denied

Add the following to enable read access to recorded sessions

  - resources:
      - session
      verbs:
      - read

• [x] Verify that a user can re-play a session (session.end)

Add the following to enable read access to the roles

- resources:
      - role
      verbs:
      - list
      - read

• [x] Verify that a user can see the roles
• [x] Verify that a user cannot create/delete/update a role

Add the following to enable read access to the auth connectors

- resources:
      - auth_connector
      verbs:
      - list
      - read

• [x] Verify that a user can see the list of auth connectors.
• [x] Verify that a user cannot create/delete/update the connectors

Add the following to enable read access to users

  - resources:
      - user
      verbs:
      - list
      - read

• [x] Verify that a user can access the "Users" screen
• [x] Verify that a user cannot reset password and create/delete/update a user

Add the following to enable read access to trusted clusters

  - resources:
      - trusted_cluster
      verbs:
      - list
      - read

• [x] Verify that a user can access the "Trust" screen
• [x] Verify that a user cannot create/delete/update a trusted cluster.

Performance/Soak Test @fspmarshall @rosstimothy

Using tsh bench tool, perform the soak tests and benchmark tests on the following configurations:

• Cluster with 10K nodes in normal (non-IOT) node mode with ETCD

• Cluster with 10K nodes in normal (non-IOT) mode with DynamoDB

• Cluster with 1K IOT nodes with ETCD

• Cluster with 1K IOT nodes with DynamoDB

• Cluster with 500 trusted clusters with ETCD

• Cluster with 500 trusted clusters with DynamoDB

Soak Tests

Run 4hour soak test with a mix of interactive/non-interactive sessions:

tsh bench --duration=4h user@teleport-monster-6757d7b487-x226b ls
tsh bench -i --duration=4h user@teleport-monster-6757d7b487-x226b ps uax

Observe prometheus metrics for goroutines, open files, RAM, CPU, Timers and make sure there are no leaks

• [ ] Verify that prometheus metrics are accurate.

Breaking load tests

Load system with tsh bench to the capacity and publish maximum numbers of concurrent sessions with interactive and non interactive tsh bench loads.

Teleport with Cloud Providers

AWS @timothyb89

• [x] Deploy Teleport to AWS. Using DynamoDB & S3
• [x] Deploy Teleport Enterprise to AWS. Using HA Setup https://gravitational.com/teleport/docs/aws-terraform-guide/

GCP @xacrimon

• [x] Deploy Teleport to GCP. Using Cloud Firestore & Cloud Storage
• [x] Deploy Teleport to GKE. Google Kubernetes engine.
• [x] Deploy Teleport Enterprise to GCP.

IBM @xacrimon

• [x] Deploy Teleport to IBM Cloud. Using IBM Database for etcd & IBM Object Store
• [x] Deploy Teleport to IBM Cloud Kubernetes.
• [x] Deploy Teleport Enterprise to IBM Cloud.

Application Access @r0mant @smallinsky

• [x] Run an application within local cluster.
  • [x] Verify the debug application debug_app: true works.
  • [x] Verify an application can be configured with command line flags.
  • [x] Verify an application can be configured from file configuration.
  • [x] Verify that applications are available at auto-generated addresses name.rootProxyPublicAddr and well as publicAddr.
• [x] Run an application within a trusted cluster.
  • [x] Verify that applications are available at auto-generated addresses name.rootProxyPublicAddr.
• [x] Verify Audit Records.
  • [x] app.session.start and app.session.chunk events are created in the Audit Log.
  • [x] app.session.chunk points to a 5 minute session archive with multiple app.session.request events inside.
  • [x] tsh play <chunk-id> can fetch and print a session chunk archive.
• [x] Verify JWT using verify-jwt.go.
• [x] Verify RBAC.
• [x] Verify CLI access with tsh app login.
• [x] Verify AWS console access.
  • [x] Can log into AWS web console through the web UI.
  • [x] Can interact with AWS using tsh aws commands.
• [x] Verify dynamic registration.
  • [x] Can register a new app using tctl create.
  • [x] Can update registered app using tctl create -f.
  • [x] Can delete registered app using tctl rm.
• [x] Test Applications screen in the web UI (tab is located on left side nav on dashboard):
  • [x] Verify that all apps registered are shown
  • [x] Verify that clicking on the app icon takes you to another tab
  • [x] Verify using the bash command produced from Add Application dialogue works (refresh app screen to see it registered)

Database Access @r0mant @smallinsky

• [x] Connect to a database within a local cluster.
  • [x] Self-hosted Postgres.
  • [x] Self-hosted MySQL.
  • [x] Self-hosted MongoDB.
  • [x] Self-hosted CockroachDB.
  • [x] AWS Aurora Postgres.
  • [x] AWS Aurora MySQL.
  • [x] AWS Redshift.
  • [x] GCP Cloud SQL Postgres.
  • [x] GCP Cloud SQL MySQL.
• [x] Connect to a database within a remote cluster via a trusted cluster.
  • [x] Self-hosted Postgres.
  • [x] Self-hosted MySQL.
  • [x] Self-hosted MongoDB.
  • [x] Self-hosted CockroachDB.
  • [x] AWS Aurora Postgres.
  • [x] AWS Aurora MySQL.
  • [x] AWS Redshift.
  • [x] GCP Cloud SQL Postgres.
  • [x] GCP Cloud SQL MySQL.
• [x] Verify audit events.
  • [x] db.session.start is emitted when you connect.
  • [x] db.session.end is emitted when you disconnect.
  • [x] db.session.query is emitted when you execute a SQL query.
• [x] Verify RBAC.
  • [x] tsh db ls shows only databases matching role's db_labels.
  • [x] Can only connect as users from db_users.
  • [x] (Postgres only) Can only connect to databases from db_names.
  • [x] db.session.start is emitted when connection attempt is denied.
  • [x] (MongoDB only) Can only execute commands in databases from db_names.
  • [x] db.session.query is emitted when command fails due to permissions.
  • [x] Can configure per-session MFA.
  • [x] MFA tap is required on each tsh db connect.
• [x] Verify dynamic registration.
  • [x] Can register a new database using tctl create.
  • [x] Can update registered database using tctl create -f.
  • [x] Can delete registered database using tctl rm.
• [x] Verify discovery.
  • [x] Can detect and register RDS instances and Aurora clusters.
• [x] Test Databases screen in the web UI (tab is located on left side nav on dashboard):
  • [x] Verify that all dbs registered are shown with correct name, description, type, and labels
  • [x] Verify that clicking on a rows connect button renders a dialogue on manual instructions with Step 2 login value matching the rows name column
  • [x] Verify searching for all columns in the search bar works
  • [x] Verify you can sort by all columns except labels

Desktop Access @zmb3

• [x] Connect to statically defined desktop
• [x] Connect to desktop discovered via LDAP
• [x] User lock terminates desktop session
• [x] Session terminates after client idle timeout
• [x] Audit logs show session start/end
• [x] Hosts labeled appropriately via regexp
• [x] RBAC denies access due to labels
• [x] RBAC denies access due to Windows OS Login
• [x] Multiple sessions as different users on the same desktop are allowed
• [ ] Can connect multiple windows_desktop_services to the same Teleport cluster and connect to desktops on different AD domains
• [x] Typing abcdefghijklmnopqrstuvwxyz1234567890-=!@#$%^&*()_+[]\{}|;':",./<>?~`, backspace, return and moving a cursor around with arrow keys works as expected with all supported browsers. Update: all other keys should now work. A known bug is that F11 (show desktop shortcut) doesn't work if you're on MacOS, as the OS seems to capture it and doesn't forward it to the browser. A useful tool for testing which keys are being registered on a windows machine is: https://dennisbabkin.com/kbdkeyinfo/
• [x] Left click and right clicks register as Windows clicks (right click on the desktop should show a Windows menu, not a Chrome one) on all supported browsers.

TLS Routing @r0mant

• [x] Verify that teleport proxy v2 configuration starts only a single listener.

  version: v2
  teleport:
  proxy_service:
    enabled: "yes"
    public_addr: ['root.example.com']
    web_listen_addr: 0.0.0.0:3080

• [x] Run Teleport Proxy in multiplex mode auth_service.proxy_listener_mode: "multiplex"
  • [x] Trusted cluster
  • [x] Setup trusted clusters using single port setup web_proxy_addr == tunnel_addr

    kind: trusted_cluster
    spec:
    ...
    web_proxy_addr: root.example.com:443
    tunnel_addr: root.example.com:443
    ...

• [x] Database Access
  • [x] Verify that tsh db connect works through proxy running in multiplex mode
  • [x] Postgres
  • [x] MySQL
  • [x] MongoDB
  • [x] CockroachDB
  • [x] Verify connecting to a database through TLS ALPN SNI local proxy tsh db proxy with a GUI client.
• [x] Application Access
  • [x] Verify app access through proxy running in multiplex mode
• [x] SSH Access
  • [x] Connect to a OpenSSH server through a local ssh proxy ssh -o "ForwardAgent yes" -o "ProxyCommand tsh proxy ssh %r@%h:%p" user@host.example.com
  • [x] Connect to a OpenSSH server on leaf-cluster through a local ssh proxyssh -o "ForwardAgent yes" -o "ProxyCommand tsh proxy ssh --cluster=leaf-cluster %r@%h:%p" user@node.foo.com
  • [x] Verify tsh ssh access through proxy running in multiplex mode
• [x] Kubernetes access:
  • [x] Verify kubernetes access through proxy running in multiplex mode

[zmb3]

Desktop Access: Seeing the arrow keys map incorrectly on macOS client. Same behavior in Chrome, Safari, and Firefox. This is using the built-in MacBook keyboard.

• left arrow is mapping to 4
• right arrow is mapping to 6
• up arrow is mapping to 8
• down arrow is mapping to 2

This is addressed with https://github.com/gravitational/teleport/pull/8791 which still needs to be backported (https://github.com/gravitational/teleport/pull/8813).

[r0mant]

Potential K8s access issue when used via TLS routing. I get:

➜  ~ kubectl get nodes
Unable to connect to the server: x509: certificate is valid for mbp, root.gravitational.io, host.minikube.internal, localhost, remote.kube.proxy.teleport.cluster.local, host.minikube.internal, *.teleport.cluster.local, teleport.cluster.local, *.root.gravitational.io, *.host.minikube.internal, not kube.

I'm using kubernetes_service with kubeconfig_file (if that matters). cc @smallinsky

[zmb3]

Desktop Access: disconnects due to client idle timeout fail to emit a disconnect event to the audit log (but otherwise disconnect correctly and emit a desktop session end event correctly). This is due to attempting to use the desktop ID as the event's "server ID."

Rejecting audit event client.disconnect("") from "3b581a11-b94a-4274-860b-8266868ca42e": 
server "3b581a11-b94a-4274-860b-8266868ca42e" can't emit event with 
server ID "foo-example-com". 

The server ID must be the Windows Desktop Service's HostUUID, otherwise the ValidateServerMetadata check will fail.

Fixed in #8828, which needs merge + backport.

[rosstimothy]

etcd Load Tests

3 auth 2 proxy 1 node non-IoT

tsh bench --duration=30m root@loadtest-7fbf6bcbfc-b5l7l ls

* Requests originated: 17988
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1289 ms
50         1301 ms
75         1319 ms
90         1374 ms
95         1600 ms
99         1648 ms
100        2053 ms

tsh bench --interactive --duration=30m root@loadtest-7fbf6bcbfc-b5l7l ps aux

* Requests originated: 17987
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1301 ms
50         1314 ms
75         1330 ms
90         1352 ms
95         1388 ms
99         1724 ms
100        3553 ms

3 auth 2 proxy 1 node IoT

tsh bench --duration=30m root@loadtest-7fbf6bcbfc-ww7h8 ls

* Requests originated: 17987
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1314 ms
50         1326 ms
75         1340 ms
90         1359 ms
95         1381 ms
99         1561 ms
100        3329 ms

tsh bench --interactive --duration=30m root@loadtest-7fbf6bcbfc-ww7h8 ps aux

* Requests originated: 17987
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1302 ms
50         1314 ms
75         1328 ms
90         1344 ms
95         1363 ms
99         1457 ms
100        3479 ms

10k node non-IoT

Soak tests:

tsh bench --duration=30m root@loadtest-7fbf6bcbfc-52zdr ls 

* Requests originated: 17982
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1481 ms
50         1505 ms
75         1541 ms
90         1623 ms
95         1714 ms
99         1920 ms
100        3335 ms

tsh bench --interactive --duration=30m root@loadtest-7fbf6bcbfc-klws7 ps aux

* Requests originated: 17985
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1496 ms
50         1515 ms
75         1543 ms
90         1624 ms
95         1720 ms
99         1978 ms
100        4647 ms

10k nodes IoT

Soak tests:

tsh bench --duration=30m root@loadtest-7fbf6bcbfc-47hh9 ls

* Requests originated: 17986
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1499 ms
50         1530 ms
75         1642 ms
90         1834 ms
95         1990 ms
99         2257 ms
100        4259 ms

tsh bench --interactive --duration=30m root@loadtest-7fbf6bcbfc-6gjnv ps aux

* Requests originated: 17982
* Requests failed: 0

Histogram

Percentile Response Duration
---------- -----------------
25         1527 ms
50         1558 ms
75         1682 ms
90         1897 ms
95         2035 ms
99         2249 ms
100        3143 ms

500 Trusted Clusters

[timothyb89]

Tested out several of the AWS deployment examples. Both Terraform examples worked fine with the 8.0.0-beta.2 AMIs (tested the simple example with the OSS AMIs, and the HA example with Enterprise).

The CloudFormation example seems very broken. I'm told it hasn't been maintained in quite a while and am not sure if we care to validate it here. Some of its issues:

• Node processes needed to be manually restarted to pick up the CA pin change after fetching Let's Encrypt certs
• The proxy load balancer doesn't forward port 3023 so tsh login hangs

• With a load balancer rule manually added to expose 3023, tsh login still fails:

  Original Error: *trace.ConnectionProblemError Get "https://teleport.cluster.local/v2/authorities/host?load_keys=false": ssh: rejected: connect failed (Cannot open new SSH session on reverse tunnel. Are you connecting to the right port?)

  (I gave up debugging here, I'm not sure if we care to get this example working again)

[zmb3]

Desktop access: heartbeats for multiple Windows hosts discovered via LDAP all report the same host.

Issue #8846 Fixed in #8847

[russjones]

Desktop Access: Investigate required libraries.

Issue https://github.com/gravitational/teleport/issues/8765

[timothyb89]

Just filed https://github.com/gravitational/teleport/issues/8860 for my issues with the CloudFormation example, though given https://github.com/gravitational/teleport/issues/8665#issuecomment-961368649, I wonder if I hit a similar issue.

[russjones]

Aggregate last 3 releases.

Backend	Cluster Size	Mode	PTY	6.2	7.0	8.0
etcd	10k	Regular	No	~49183 ms~	~56383 ms~ 4475 ms	3335 ms
etcd	10k	Regular	Yes	~59423 ms~	~61215 ms~ 4507 ms	4647 ms
etcd	10k	Tunnel	No	~65439 ms~	~53759 ms~ 4451 ms	4259 ms
etcd	10k	Tunnel	Yes	~64924 ms~	~48223 ms~ 4435 ms	3143 ms
DynamoDB	10k	Regular	No
DynamoDB	10k	Regular	Yes
DynamoDB	10k	Tunnel	No
DynamoDB	10k	Tunnel	Yes
DynamoDB	1	Regular	No		2471 ms	1824 ms
DynamoDB	1	Regular	Yes		2081 ms	1483 ms
DynamoDB	1	Tunnel	No		826 ms	2125 ms
DynamoDB	1	Tunnel	Yes		518 ms	2002 ms

[tcsc]

Re: Check agent forwarding is correct based on role and proxy mode.

• WebUI allows Agent Forwarding when allowed by role
• WebUI disallows Agent Forwarding when disallowed by role
• WebUI allows agent forwarding in session recording modes off, proxy, node.

rec mode	fwd allowed by role	fwd not allowed by role
off	allowed	disallowed
proxy	allowed	conn failed
node	allowed	disallowed

Caveats:

I'm still not sure what Proxy Mode means in this context, but I've interpreted to to mean the session_recording mode. Even so, I am still not sure what the correct Agent Forwarding behaviour is for the different recording modes, or even how they should be expected to affect the Agent Forwarding modes.

Fully aware that I may have been testing the wrong thing, I considered something that looks like this to be "allowed":

..and something that looks like this was considered "disallowed":

[smallinsky]

ALPN Proxy + Reverse Tunnel fails when ACME is used:

Issue https://github.com/gravitational/teleport/issues/8665#issuecomment-961368649 Fixed in https://github.com/gravitational/teleport/pull/8869

[fspmarshall]

Dynamo (IoT)

10k Scaling

Soak

tsh bench --duration=30m root@ip-172-31-11-250-us-west-2-compute-internal ls

* Requests originated: 17999
* Requests failed: 0

Histogram

Percentile Response Duration 
---------- ----------------- 
25         152 ms            
50         166 ms            
75         181 ms            
90         194 ms            
95         208 ms            
99         283 ms            
100        2125 ms

tsh bench --interactive --duration=30m root@ip-172-31-11-250-us-west-2-compute-internal ps aux

* Requests originated: 17999
* Requests failed: 0

Histogram

Percentile Response Duration 
---------- ----------------- 
25         159 ms            
50         169 ms            
75         179 ms            
90         192 ms            
95         207 ms            
99         284 ms            
100        2002 ms

Dynamo (non-IoT)

10k Scaling

Soak

tsh bench --duration=30m root@ip-172-31-11-250-us-west-2-compute-internal ls

* Requests originated: 17999
* Requests failed: 0

Histogram

Percentile Response Duration 
---------- ----------------- 
25         145 ms            
50         154 ms            
75         164 ms            
90         176 ms            
95         188 ms            
99         237 ms            
100        1824 ms

tsh bench --interactive --duration=30m root@ip-172-31-11-250-us-west-2-compute-internal ps aux

* Requests originated: 17999
* Requests failed: 0

Histogram

Percentile Response Duration 
---------- ----------------- 
25         150 ms            
50         161 ms            
75         174 ms            
90         186 ms            
95         195 ms            
99         238 ms            
100        1483 ms

Dynamo (500 Trusted Cluster)

---

Notes

• 500tc memory usage high due to failure to clean up sqlite databases for remote clusters: #8916
• All soak command were run from bastion within the same network as the auth/proxy servers, and the target node. Previous tests were not necessarily done in this manner. Improved numbers are no representative of a real performance improvement.