Explore: prepare a proposal for our CLI changes

[gcapizzi]

The bulk of our authentication support work will happen in the cf CLI. The idea is to change the CLI so that, when pointed at a cf-k8s foundation, it leverages the user's $KUBECONFIG and possibly code from client-go to authenticate exactly in the same way as kubectl would.

In #62 we have proved that we can support $KUBECONFIG and reuse a lot of client-go functionality, while implementing the same interface the CLI already uses for UAA-based authentication.

Now we need to get a more thorough understanding of all changes needed to implement our strategy. Some questions that come to mind:

• In which part of the codebase are we going to recognise cf-k8s and instantiate our ConnectionWrapper implementation? Can those two things even happen in the same place with the current architecture?
• Is implementing ConnectionWrapper enough to abstract all authentication aspects?
  • We know this is not the case:
    • some packages try to parse the token to get the username out of it;
    • some packages try to refresh the token.
  • Does this mean we need to introduce new abstractions before we can proceed with adding new behaviour?
• How are we going to proceed? Which stories and related PRs do we envision?
  • We already have some CLI related stories in the backlog.

The result might look a bit like a Kubernetes Enhancement Proposal although probably not so detailed!

Deliverable

Let's start this on a Google Doc, and then eventually open a GItHub issue on the CLI repository once we reach agreement in the team.

[gcapizzi]

I have taken a quick look at the CLI codebase, and in particular at the cf api command.

The command delegates to Actor.SetTarget, which, in order:

1. initialises the Cloud Controller client;
2. calls the CF root endpoint;
3. stores the results in the configuration.

Config gets loaded in the CommandParser and then passed to each command via the Setup method. It gets then used to instantiate the wrapped clients, so it is available when the wrappers are wired in. This means that we could instantiate a different ConnectionWrapper based on the Config:

• ApiCommand tells the Actor to fetch information about the API and to store them into the Config;
• Any subsequent command (they all embed BaseCommand) receives the Config and injects it into its clients, which in turn get injected into the Actor;
• GetNewClientsAndConnectToCF can then use the Config to decide which ConnectionWrappers to apply!

[gcapizzi]

Today I systematically searched for all the code paths that used a UAAClient. Here is the list of Actor methods that do, alongside the commands they get invoked from:

• Authenticate(credentials map[string]string, origin string, grantType uaa.GrantType) error
  • auth
  • login
• CreateOrgRole(roleType constant.RoleType, orgGUID string, userNameOrGUID string, userOrigin string, isClient bool (v7action.Warnings, error)
  • set-org-role
  • create-space
  • set-space-role
• CreateSpaceRole(roleType constant.RoleType, orgGUID string, spaceGUID string, userNameOrGUID string, userOrigin string, isClient bool) (v7action.Warnings, error)
  • create-space
  • set-space-role
• CreateUser(username string, password string, origin string) (resources.User, v7action.Warnings, error)
  • create-user
• DeleteUser(userGuid string) (v7action.Warnings, error)
  • delete-user
• GetLoginPrompts() (map[string]coreconfig.AuthPrompt, error)
  • login
• GetSSHPasscode() (string, error)
  • ssh-code
• GetSecureShellConfigurationByApplicationNameSpaceProcessTypeAndIndex(appName string, spaceGUID string, processType string, processIndex uint) (v7action.SSHAuthentication, v7action.Warn ings, error)
  • ssh
• GetUAAAPIVersion() (string, error)
  • auth
• GetUser(username, origin string) (resources.User, error)
  • create-user
  • delete-user
• RefreshAccessToken() (string, error)
  • logs
  • oauth-token
• RevokeAccessAndRefreshTokens() error
  • logout
• ScheduleTokenRefresh(func(time.Duration) <-chan time.Time, chan struct{}, chan struct{}) (<-chan error, error)
  • logs
• UpdateUserPassword(userGUID string, oldPassword string, newPassword string) error
  • passwd

Some observations:

• We probably want these methods to belong to a separate interface that we can implement differently for cf-k8s.
• Some of these methods can be left empty in our implementation, as we don't plan to support the corresponding commands:
  • Authenticate, GetLoginPrompts, GetUAAAPIVersion, RevokeAccessAndRefreshTokens, UpdateUserPassword are all used by the auth flow commands (login, auth, logout, passwd).
  • CreateUser, DeleteUser and GetUser are used in the user management commands (create-user, delete-user).
  • GetSSHPasscode and GetSecureShellConfigurationByApplicationNameSpaceProcessTypeAndIndex are used in the ssh* commands.
• The token refreshing methods (RefreshAccessToken and ScheduleTokenRefresh) are needed by log to keep the access token fresh while streaming logs.
  • Note that this has nothing to do with the CC client and the ConnectionWrapper, as the logs are streamed from Log Cache via a LogCacheClient, which is wired to use the Config.AccessToken method as its token-generating function.
  • RefreshAccessToken mutates the Config which then returns the new token to LogCacheClient: Config is effectively used as shared mutable state.
  • Provided we will have a Log Cache compatible API that can authenticate in the same way, the tools provided by client-go make it trivial to generate an authenticated http.Client which can then be injected in LogCacheClient.
• CreateOrgRole and CreateSpaceRole are interesting: we definitely want to implement these, but I'm still not sure how.

[gcapizzi]

@emalm had an interesting question on our authentication doc: can we leverage cf login to allow users to choose one of the AuthInfos stored in $KUBECONFIG vs trying to detect it automatically? The answer is yes!

This is what cf login does:

• LoginCommand calls Actor.GetLoginPrompts to get a list of prompts to submit to the user via the UI.
• Each [AuthPrompt]() has a Type, currently either TEXT or PASSWORD. Different UI methods are called based on this.
• The resulting credentials are then passed to Actor.Authenticate.

Here is how we could modify it:

• By providing different implementations of both GetLoginPrompts and Authenticate, we could prompt the user for the AuthInfo they want to use, and store it in the Config.
• Given AuthInfos tend to have long and hard-to-type names, we could introduce a new AuthPrompt.Type called MENU and use UI.DisplayTextMenu for it.
• The list of menu entries could be stored in AuthPrompt.DisplayName (e.g. separated by commas) or in a new Entries field.

This flow looks more intuitive and more similar to what CF users already do. It also does not need the Kubernetes cluster URL, which simplifies the shim implementation, as the root endpoint could just return a "kubernetes": true flag.

[gcapizzi]

Methods trying to parse the JWT token:

• Actor
  • RefreshAccessToken() (string, error)
  • ScheduleTokenRefresh(func(time.Duration) <-chan time.Time, chan struct{}, chan struct{}) (<-chan error, error)
  • ParseAccessToken(string) (jwt.JWT, error)
• UAAAuthentication.Make(*cloudcontroller.Request, *cloudcontroller.Response) error
• Config
  • CurrentUser() (User, error)
  • CurrentUserName() (string, error)

The Actor methods are only invoked in cf logs and cf oauth-token, which we can ignore for now. We're going to provide an alternative ConnectionWrapper implementation to UAAAuthentication, so we don't care about that either.

Unfortunately the Config methods are called literally everywhere. I have identified three reasons:

1. to print user info on screen;
2. to check if the user is logged in;
3. in CreateSpaceCommand, the CLI assigns the roles of Space Manager and Space Developer to the user creating the space.

The username from $KUBECONFIG is usually not a user name, but a cluster name. It could be fine for 1 and 2, but not for 3. I can think of two strategies to handle this:

1. Implement our own CurrentUser to return the real username, using the techniques described here to introduce a /whoami endpoint.
2. Add some conditional logic in CreateSpaceRole which ignores the userNameOrGUID argument if it's equal to the current output of CurrentUser. The shim could then default the username to the one in the token if it's missing.

[gcapizzi]

Today I'm looking at a couple of other interfaces that we might have to (partially) reimplement:

• command.SharedActor has an IsLoggedIn method. The current implementation checks if tokens are present in the config. We could replace it with checking if Kubernetes user is present in the config.
• command.Config has quite a few auth-related (and UAA-related methods), including the CurrentUser* methods mentioned in my previous comment. The implementation of this interface gets wired in very early in the program lifecycle, much before the code to recognise a cf-k8s foundation could run. We might have a way to replace/decorate it later on, I'm on it.

[danail-branekov]

We started hacking the cli and try to integrate it with cfshim and we scoredcertain progress.

Setting up the dev environment:

• Checkout the auth-explore
• Checkout the wip-auth brach of cf-crd-explorations
• Checkout the wip-k8s branch of our cli fork
• Create a kind cluster with dex and cfshim CRs via running the deploy.sh script
• Run the cfshim with run.sh (I got bored and frustrated trying to make it run on the kind cluster)
• Get yourself a token for alice@vcap.me and put it in the kubeconfig, see this README for details. No need to login with github, just use the email login on the dex page
• Build the cli fork

Behold!

• target

❯ ./cli api http://localhost:9000
Setting API endpoint to http://localhost:9000...
Warning: Insecure http API endpoint detected: secure https API endpoints are recommended
OK

API endpoint:   http://localhost:9000
API version:

Not logged in. Use 'cli login' or 'cli login --sso' to log in.

• login

❯ ./cli login
API endpoint: http://localhost:9000
Warning: Insecure http API endpoint detected: secure https API endpoints are recommended
Warning: unable to determine whether targeted API's version meets minimum supported.

1. alice
2. gke_cff-eirini-peace-pods_europe-west1-b_cf4k8s4a8e
3. gke_cff-eirini-peace-pods_europe-west1-b_dex-pinniped
4. gke_cff-eirini-peace-pods_europe-west1-b_pinniped
5. kind-cross-org-1
6. kind-cross-org-2
7. kind-dex-play

Choose your Kubernetes user (enter to skip): 1

Authenticating...
OK

Targeted org foo.

Targeted space bar.

API endpoint:   http://localhost:9000
API version:
user:           alice
org:            foo
space:          bar

The cli would show a menu with the entries from kubeconfig, note that alice is there! After being chosen, the UI reports that you are logged in as alice

• apps

❯ ./cli apps
Getting apps in org foo / space bar as alice...

Unexpected Response
Response Code: 500
Code: 0, Title: , Detail: {"errors":[{"detail":"error fetching app: apps.apps.cloudfoundry.org is forbidden: User \"oidc:alice@vcap.me\" cannot list resource \"apps\" in API group \"apps.cloudfoundry.org\" at the cluster scope","title":"ServerError","code":10001}]}

FAILED

Good, we are hitting the shim with alice's JWT token so K8S knows who we are. alice is not allowed to do that though... yet

❯ k apply -f ~/workspace/auth-explore/run-cfshim/alice-viewer-rolebinding.yml
clusterrolebinding.rbac.authorization.k8s.io/alice-viewer created
clusterrole.rbac.authorization.k8s.io/app-viewer created
clusterrolebinding.rbac.authorization.k8s.io/alice-apps-viewer created

❯ ./cli apps
Getting apps in org foo / space bar as alice...

No apps found

Let's create an app and try to list it:

❯ k apply -f ~/workspace/cf-crd-explorations/config/samples/cf-crds/app.yaml
app.apps.cloudfoundry.org/my-app-guid created

~/workspace/cli wip-k8s
❯ ./cli apps
Getting apps in org foo / space bar as alice...

Error unmarshalling the following into a cloud controller error: 404 page not found

FAILED

This error is caused by the shim not implementing the /processes endpoint. However, this is irrelevant to the spike

[mnitchev]

Blocking until the rest of the team can take a look at the draft proposal here

[emalm]

@mnitchev, could you please grant global comment-level access to that proposal doc? Thanks!

[kieron-dev]

@emalm - global comment-level enabled.