search

vmware-tanzu / tanzu-framework

Tanzu Framework provides a set of building blocks to build atop of the Tanzu platform and leverages Carvel packaging and plugins to provide users with a much stronger, more integrated experience than the loose coupling and stand-alone commands of the previous generation of tools.
Apache License 2.0
195 stars 192 forks source link

Umbrella: API-driven discovery of CLI plugins #74

Open iancoffey opened 3 years ago

iancoffey commented 3 years ago

Constituent Issues

Describe the feature request In moving toward packaging things as imgpkg, the clearest way of packaging up a CLI plugin will be to make an API for it.

extensions.tanzu.vmware.com

// CLIPlugin is the Schema for the plugins API
type CLIPlugin struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   CLIPluginSpec   `json:"spec,omitempty"`
    Status CLIPluginStatus `json:"status,omitempty"`
}

// CLIPluginSpec defines the desired state of Plugin
type CLIPluginSpec struct {
    // Description of the plugin
    Description string `json:"description"`

    // Version of the plugin
    Version string `json:"version"`

    // Artifacts for the plugin
    Artifacts Artifacts `json:"artifacts"`
}

// Artifacts  for the plugin
type Artifacts struct {
        // LinuxAMD64 binary in imgpkg
        LinuxAMD64  string `json:"linuxAMD64"`

        // DarwinAMD64 binary in imgpkg
        DarwinAMD64  string `json:"darwinAMD64"`
        ...
}

This file could be placed in an imgpkg and however that package is applied (*may be UI), the CLI plugin will exist server side which the Framework CLI can then query to learn about plugins it should have installed.

This also supports air gapped environments because imgpkg can overwrite the artifact images with a localized registry.

The CLI then upon login could check what plugins are installed in that cluster, and ask the user if they wish to download those locally. We could also provide a tanzu plugin sync command which would invoke this logic.

When installing a package tanzu package install serverless the CLI should apply the plugin to the cluster and then install the appropriate binary locally before exiting.

Describe alternatives you've considered

Affected product area (please put an X in all that apply)

[ ] APIs [ ] Addons [x] CLI [ ] Docs [ ] Installation [x] Plugin [ ] Security [ ] Test and Release [ ] User Experience

Additional context

Collated Context

Context from 2021-04-15 15:47:11 User: vuil Thanks, I like the idea, but I would like to understand more the airgap scenario we are talking about. And I am not clear what having a plugin installed on a cluster means. Is the proposal supposed to apply to all plugins? If so is there a chick-egg issue with handling a plugin like management-cluster needed to stand up a Tanzu cluster in the first place?

Context from 2021-04-15 16:07:22 User: pbarker So let me back up a bit and explain the product end of it a bit. This largely comes out of me working with packaging up the CLI plugins within imgpkg and how that should look.

A couple of things this aims to solve:

Ability to install packages from the UI Customers will often be installing packages from the UI, in this instance if the CLI plugin is in the package, how does it get installed?

By making the CLI plugin an API, it would simply get applied to the cluster. Then when a customer logs in to a cluster or runs tanzu plugin sync, the CLI will look to see what plugins are available in the cluster and install them locally.

Ability to see and interact with the packages on a given cluster As we're seeing the ecosystem grow, its becoming apparent there will be lots of plugins/packages and likely very different packages for any given cluster.

If a user has to manually install all the plugins for a given cluster, this could become very burdensome.

By making them an API, when I log into a cluster I am now able to simply interact with the packages using the plugins provided. It greatly simplifies workload/managment/saas interactions as well.

Clean airgapped and install story imgpkg finds images listed in your config files and rewrites the registry path to a local registry. Having the plugin as an API with each arch binary as its own imgpkg will play well with this tooling, while not providing significant overhead of installing a plugin for a given architecture.

The previous idea was to put all the plugin binaries for every arch in an imgpkg, however that meant every time you needed to install a given plugin you would have to download an image with all the architectures in it.

In this model, the CLI would be able to download a specific image for that architecture, keeping things light and simple.

On bootstrapping

For bootstrap plugins like the management cluster which only exist client side, I think we just place these plugin files locally in ~/.config/tanzu/plugins. The CLI will look in this path or whatever server its pointed at to know what plugins it needs.

Context from 2021-04-16 16:45:03 User: cppforlife from what i understood here... package author that writing Kubernetes configuration will include a CLIPlugin CR and appropriate CLI binaries within package.

package consumer (platform operator?) at some point decides to install a package to a single kubernetes cluster. once the install completes if user runs tanzu plugin sync, tanzu cli will search for CLIPlugin resources on cluster and decide what to download locally onto users computer.

assuming i got above right:

Context from 2021-04-17 13:15:48 User: pbarker

user that runs tanzu plugin sync -- is it platform operator, app dev, someone else, doesnt matter?

Doesn't matter, it just syncs your local plugins to whats available in the cluster

this may make sense for single k8s cluster scenario. what happens for LCP managed clusters?

The roughly just marries CLI functionality to API functionality. So whatever package I'm installing on LCP would have an API and this provides the corresponding CLI.

are CLIPlugins cluster scoped or namespace scoped?

This one is a bit hard, I want to say cluster scoped, but I could see a scenario in which we have different potentially breaking plugin versions in different namespaces. This part needs more thought

should this metadata be on Package CR or in cluster after package install?

In the cluster I think, but I haven't looked much into the metadata on the Package CR, let me know if you see another path there

authenticating to registry from tanzu CLI to download binaries?

I would think it would be the same registry you download the packages from unless I'm missing a piece here

Context from 2021-04-18 22:40:19 User: cppforlife

Doesn't matter, it just syncs your local plugins to whats available in the cluster

i think it matters not in terms of technical impl but possibly in terms of user experience. for devs for example, do they need CLIs for packages that were installed by platform operators on all clusters? i think figuring out multiple concrete examples of plugins and who/why needs them would nail this down.

So whatever package I'm installing on LCP would have an API and this provides the corresponding CLI.

question about LCP was in a context of which CLIs one would want when interacting with LCP clusters. CLIs for packages that were installed on management cluster itself or CLIs on workload clusters that may aid administration.

In the cluster I think, but I haven't looked much into the metadata on the Package CR, let me know if you see another path there

since Package CR represents what's available, we might want to present some info around CLI plugins there. CLIPlugin CR that gets created as part of installation also makes sense. possibilities here might be -- does package author specify it as part of kubernetes config they are authoring, or is this a packaging system concept. not sure (there is definitely an appeal for it being decoupled). i think figuring out e2e UI/CLI experiences with plugin examples will shed more light here.

I would think it would be the same registry you download the packages from unless I'm missing a piece here

yeah it would be, but network access and authentication info is still a question in my mind. going back to use cases, will devs machines have access to registry that's accessible to clusters. how would CLI find credentials to auth to a registry. potentially solvable problems but definitely somewhat tricky imho.

Context from 2021-04-19 22:32:23 User: pbarker

for devs for example, do they need CLIs for packages that were installed by platform operators on all clusters?

We've mostly agreed to not split up functionality by personas, there is too much overlap. But yeah I agree we need to walk through use cases

CLIs for packages that were installed on management cluster itself or CLIs on workload clusters that may aid administration.

What we're roughly trying to get at is avoiding implicit logins across domains, and making plugins first class citizens which are tied to APIs. If LCP wants to provide an API that does operations against workload clusters it manages then that would be within their capability.

since Package CR represents what's available, we might want to present some info around CLI plugins there

Yeah I would like to dig more into this, it may aid in plugins which are solely client side too.

will devs machines have access to registry that's accessible to clusters

Yeah this would be something to validate

Context from 2021-04-21 18:59:59 User: pbarker Building on this we should also make the root command compositional and move toward first principles.

A DynCmd is an extension on top of cobra commands that allows any cobra command to become dynamic and load subcommands from plugins.

// DynCmd is the Schema for the dyncmds API
type DynCmd struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   DynCmdSpec   `json:"spec,omitempty"`
    Status DynCmdStatus `json:"status,omitempty"`
}

// DynCmdSpec defines the desired state of DynCmd
type DynCmdSpec struct {
    // Sources of plugins
    Sources []PluginSource `json:"sources"`

    // Views available for the plugin
    Views []View `json:"views"`
}

// PluginSource is a source of plugins
type PluginSource struct {
    // Name of the plugin source
    Name string `json:"name"`

    // PluginReconciler is an imgpkg which finds plugins definitions.
    PluginReconciler Artifacts `json:"pluginReconciler`
}

// View for the dynamic command
type View struct {
    // Template to render the view.
    Template string `json:"template"`

    // Data for the view.
    Data   map[string]interface{}
}

// DynCmdStatus defines the observed state of DynCmd
type DynCmdStatus struct {
    // Catalog is a list of plugin executables
    Catalog []PluginExecutable `json:"catalog"`

    // CurrentView is the current view
    CurrentView string `json:"view"`
}

Context from 2021-04-22 16:00:18 User: iancoffey The DynCmd concept is 💯, particularly if it just works with our linting and doc generation and allows us to gradually migrate commands . My 2 cents is we should focus on the dynCmd API as first steps.

Context from 2021-04-22 22:06:00 User: pbarker Created https://github.com/vmware-tanzu-private/dyncmd for us to begin iterating on this component. I think we should take a note out of your book @iancoffey and define the API for this in https://cuelang.org. It would allow it to be usable from more API surfaces

UPDATE:

The new proposed API is as below


// ArtifactList contains an Artifact object for every supported platform of a version.
type ArtifactList []Artifact

// OCIImage is a fully qualified OCI image of the plugin binary.
type OCIImage string

// AssetURI is a URI of the plugin binary. This can be a fully qualified HTTP path or a local path.
type AssetURI string

// Artifact points to an individual plugin binary specific to a version and platform.
type Artifact struct {
    // Image is a fully qualified OCI image for the plugin binary.
    Image OCIImage `json:"image,omitempty"`
    // AssetURI is a URI of the plugin binary.
    URI AssetURI `json:"uri,omitempty"`
    // SHA256 hash of the plugin binary.
    Digest string `json:"digest"`
    // Type of the binary artifact. Valid values are S3, GCP, OCIImage.
    Type string `json:"type"`
    // OS of the plugin binary in `GOOS` format.
    OS string `json:"os"`
    // Arch is CPU architecture of the plugin binary in `GOARCH` format.
    Arch string `json:"arch"`
}

// CLIPluginSpec defines the desired state of CLIPlugin.
type CLIPluginSpec struct {
    // Description is the plugin's description.
    Description string `json:"description"`
    // Recommended version that Tanzu CLI should use if available.
    // The value should be a valid semantic version as defined in
    // https://semver.org/. E.g., 2.0.1
    RecommendedVersion string `json:"recommendedVersion"`
    // Artifacts contains an artifact list for every supported version.
    Artifacts map[string]ArtifactList `json:"artifacts"`
    // Optional specifies whether the plugin is mandatory or optional
    // If optional, the plugin will not get auto-downloaded as part of
    // `tanzu login` or `tanzu plugin sync` command
    // To view the list of plugin, user can use `tanzu plugin list` and
    // to download a specific plugin run, `tanzu plugin install <plugin-name>`
    Optional bool `json:"optional"`
}

//+kubebuilder:object:root=true

// CLIPlugin denotes a Tanzu cli plugin.
type CLIPlugin struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata"`
    Spec              CLIPluginSpec `json:"spec"`
}
scothis commented 3 years ago

In moving toward packaging things as imgpkg, the clearest way of packaging up a CLI plugin will be to make an API for it.

I would avoid conflating use of an imgpkg image to distribute files, with an k8s-style resource to configure CLI plugins. They are both independently useful ideas and in no way coupled to each other. imgpkg doesn't care about the content of the files in the image, whatever you put into it will come back out on the other side.

zjs commented 3 years ago

Summarizing my current thinking on this issue as I think the top-level comment is kind of hard to read at this point:

We have a variety of ways to extend Tanzu. Currently, these work in independent ways. You can add a management package to the management cluster. You can add a plugin to the Tanzu CLI.

The independence between these things is important, and provides flexibility. It's good that these separate extensibility points exist.

However, it will be common that someone wanting to extend the management cluster will also want to introduce a corresponding CLI plugin. When the management package is installed, it would be useful to make that corresponding CLI plugin available to all users of a management cluster.

To address this, we can introduce an API on the management cluster (i.e., a new framework management package) that allows others (including other management packages) to register CLI plugins.

When the Tanzu CLI connects to the management cluster, it can use this API to determine if there are any additional plugins which should be installed, and proceed appropriately. (Exact UX needs to be defined.)

This architecture has added benefits.

For example, it may help with the situation where a user is working with multiple heterogeneous management clusters. Having the management cluster report what plugins are registered can allow the CLI to tailor its behavior to the cluster it's being used with.

This can also provide value in the case of environments with restricted connectivity. By maintaining this information within each management cluster, users of that management cluster can be instructed to get the plugin from the same container registry as the management package was retrieved from.

The main outputs of this work will include:

  1. A core management package which defines and implements this new API
  2. A core CLI plugin which acts as a client for this API
  3. Tooling to help CLI plugin authors define the custom resource to register their plugin with a management cluster
  4. Tooling to help management package authors leverage (3)

Key considerations will include:

  1. User experience (including around associating discovery with the login/context-switching workflow)
  2. Ensuring that the available set of plugins can be tailored to the endpoint (e.g., management cluster) being used, without leading to conflicts (e.g., if two endpoints expect different versions of a plugin)
  3. Security (e.g., a signing pattern to ensure that the plugin we're downloading is what we expect as well as some client-side controls, such as allow-listing registries.)
zjs commented 3 years ago

I would avoid conflating use of an imgpkg image to distribute files, with an k8s-style resource to configure CLI plugins. They are both independently useful ideas and in no way coupled to each other. imgpkg doesn't care about the content of the files in the image, whatever you put into it will come back out on the other side.

To achieve the above, we just need a way for a CRD to reference a CLI plugin. There are a variety of ways to do that, but using imgpkg would be aligned with other technical decisions.

And, looking at this from the other direction, defining a way to distribute CLI plugins (such as by using imgpkg) does not require an API; that's independently useful.

pbarker commented 3 years ago

Also worth noting that imgpkg somewhat does require an API in this use case. You wouldn't be able to place the binaries directly in the main bundle because they need to be host OS specific.

If you look at:

// Artifacts  for the plugin
type Artifacts struct {
        // LinuxAMD64 binary in imgpkg
        LinuxAMD64  string `json:"linuxAMD64"`

        // DarwinAMD64 binary in imgpkg
        DarwinAMD64  string `json:"darwinAMD64"`
        ...
}

This provides us pointers to individual bundles for each arch, which can then be airgapped/rewritten with the imgpkg copy

Unless we can find another sane way of doing this, they are coupled a bit

anujc25 commented 3 years ago

The proposed API is as below:


// ArtifactList contains an Artifact object for every supported platform of a version.
type ArtifactList []Artifact

// OCIImage is a fully qualified OCI image of the plugin binary.
type OCIImage string

// AssetURI is a URI of the plugin binary. This can be a fully qualified HTTP path or a local path.
type AssetURI string

// Artifact points to an individual plugin binary specific to a version and platform.
type Artifact struct {
    // Image is a fully qualified OCI image for the plugin binary.
    Image OCIImage `json:"image,omitempty"`
    // AssetURI is a URI of the plugin binary.
    URI AssetURI `json:"uri,omitempty"`
    // SHA256 hash of the plugin binary.
    Digest string `json:"digest"`
    // Type of the binary artifact. Valid values are S3, GCP, OCIImage.
    Type string `json:"type"`
    // OS of the plugin binary in `GOOS` format.
    OS string `json:"os"`
    // Arch is CPU architecture of the plugin binary in `GOARCH` format.
    Arch string `json:"arch"`
}

// CLIPluginSpec defines the desired state of CLIPlugin.
type CLIPluginSpec struct {
    // Description is the plugin's description.
    Description string `json:"description"`
    // Recommended version that Tanzu CLI should use if available.
    // The value should be a valid semantic version as defined in
    // https://semver.org/. E.g., 2.0.1
    RecommendedVersion string `json:"recommendedVersion"`
    // Artifacts contains an artifact list for every supported version.
    Artifacts map[string]ArtifactList `json:"artifacts"`
    // Optional specifies whether the plugin is mandatory or optional
    // If optional, the plugin will not get auto-downloaded as part of
    // `tanzu login` or `tanzu plugin sync` command
    // To view the list of plugin, user can use `tanzu plugin list` and
    // to download a specific plugin run, `tanzu plugin install <plugin-name>`
    Optional bool `json:"optional"`
}

//+kubebuilder:object:root=true

// CLIPlugin denotes a Tanzu cli plugin.
type CLIPlugin struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata"`
    Spec              CLIPluginSpec `json:"spec"`
}
joshrosso commented 3 years ago

I'm not sure where in the struct this belongs, but the bootstrap process need to inject, somewhere, the edition that initially bootstrapped the cluster. This should enable us to discover which edition the cluster belongs to.

Related to: https://github.com/vmware-tanzu/community-edition/issues/2056