Introduce `Resource`, refine `Kind` interfaces, define standard Grafana shape <> K8s shape transforms

[sdboyer]

This PR introduces:

• Refinements on existing interfaces (Kind, Core, Custom) - a set of interfaces that are basically analogous to "class" in conventional OOP thinking
• New key base interfaces for kindsys (Resource, TypedCore, TypedCustom). Resource is basically like "instance of class" aka "object" in conventional OOP thinking
• A more concrete implementation of the kind schema format by way of a decoder (encoding/KubernetesDecoder) that
  • Transforms from Kubernetes shape (where we expect certain data to appear in annotations) to Grafana shape (where we expect that data to appear somewhere within the metadata subresource)
  • Provides a general type, kindsys.Unstructured, which can be used to work dynamically with resources within our apiserver. This is exactly the same role as unstructured.Unstructured in conventional k8s

All that said, this PR is very messy. The unmarshaling and remarshaling and stages? They just didn't come out at all the way i'd originally imagined.

The main goal is to get us to a minimally working state that at least sorta captures this k8s shape->grafana shape transform, which has been really confusing for folks and the source of noteworthy divergence between what's in core/the apiserver vs. what's happening in app sdk/kindsys. The reason to stop here, in this minimally working state, is that it can unblock iterative progress on several threads of potentially concurrent work:

• Removing the GrafanaResource type (which is awkwardly in between Grafana shape and k8s shape and can never produce JSON that will be valid according to the agreed upon schema format) and replacing it with a standard thing here.
• Splitting thema types off the main kindsys interfaces and onto an optional interface, to support prototyping experiments with an alternative schema system implementation
• Introducing a test framework in kindsys that, at minimum, exercises the codejen pipelines of downstream projects (grafana, grok, app-sdk)
• Implementing conversion webhooks in our apiserver
• Proper progress towards kind providers
• Injection of Go functions for migrations at kind-bind time
• Converging [Core] and [Custom] into a single kind, and bringing naming back into alignment #25
• App SDK moving towards this common format as well, so that we can actually start bringing both the client and server side together
• grok generating Go types that can be serialized to a form our apiserver will actually accept

and i'm forgetting...at least one. So yes, this is a mess, and i'm now pretty convinced that we need to revisit some of the assumptions in the kind schema format. But it works with core, for dashboards, and we can move iteratively to fix these things from here.

[sdboyer]

whoops, i missed grafana/grafana#70287 - updating accordingly

[sdboyer]

Actually...hmm, not sure if that's a good idea. That PR isn't against main - would be good to merge it back in. Kindsys shouldn't reflect what's there until it's in main.

[IfSentient]

@IfSentient does it even make sense to pull this in now, then we can iteratively work on combining what's here with what's there?

I think pulling in the resource.Object interface and its associated types should be fine, the only thing that gives me pause is that there's a possibility that it's still in flux. I was already looking at how to reasonably separate out the resource package, or at least its constructs (go submodules proved to have too many pain points) so that it can be imported without the associated kubernetes dependencies from the app-sdk's k8s package, and the grafana-plugin-sdk-go's dependencies that we need in the plugin package.

Ultimately, I'm still also trying to determine where the border between kindsys and the app-sdk is, because it's a tricky question to draw a hard border. Right now, kindsys is only used in the SDK for codegen purposes, so is it less ergonomic for a user to have to import both the SDK and kindsys when working with kinds, or does it make sense to try to slice up the SDK into smaller pieces that exist independently of each other (submodules is I guess the only real solution here aside from multiple repos, which re-introduces the same pain points as using kindsys).

[sdboyer]

After some chatting, and seeing the needed work in grafana/grafana#70728, i'm going to copy over the resource subpackage from app sdk, as well as this file, which contains the logic for going back and forth between the object form we schematize in our kinds, and k8s native object form.

[sdboyer]

This has uncovered a significant divergence between what's being done in grafana, and what's in app-sdk/kindsys. Fortunately, i think we can nail this one pretty dead-on, and add a whole lot of clarity while we do.

[sdboyer]

i've pushed up a handwavy sketch of what i think the key, top-level interfaces for kinds are:

 ┌----> Core -------> TypedCore
 |
Kind -> Custom -----> TypedCustom
 |
 └----> Composable -> TypedComposable (TODO)

This doesn't compile yet, and there are methods not yet even written on them. But i think the docs are there, which in itself is a big deal given all the confusion we've had. The only thing that's missing is a treatment of the K8s shape <> Grafana shape transform relation.

i've also starting pulling over the relevant types from app sdk for defining Resource.

i'll be continuing to iterate on this through the day!

[sdboyer]

Fun note - these interfaces are where we're going to (finally!) see runtime composition. i made a separate commit with what i expect those methods will look like. I think it's suggestive of the expected use pattern here - whether a given core kind supports composition or not, it should fit tidily into the system we have, and be easy to manage generically.

[sdboyer]

Tests are failing in the current state, and i have to refactor the encoder to mirror changes in the decoder. And there's lots of half-baked method implementations that panic. So, definitely not ready to merge.

Buuuut...this all fundamentally works: saving dashboards passes through the new kind.Validate() method introduced in this PR, and works.

[sdboyer]

This is basically ready for review.

i need to rewrite the issue description (goal has shifted considerably since i first wrote it) and probably could stand to do another pass at just cleaning up what's in the code. But given that this PR has evolved to the goal of just trying to get to an intermediate state that unblocks several other paths of work we can work on concurrently, it's gonna be...messy.

[sdboyer]

Re: kind schema format - I'd really like to just have us specify new subresources, instead of doing this trickery with annotations. It will be cleaner to implement, simplify what the decoders have to do, and most critically, the relation between what app devs write in schema vs. what the fields on their generated types will just be obvious. Unstructured could become something more like:

// THIS IS HANDWAVING, PLZ READ IT AS A STRONG OPINION LOOSELY HELD
type UnstructuredResource struct {
    APIVersion string
    Kind string

    // just the standard k8s metadata fields
    Meta K8sMetadata `json:"metadata"`
    // kindsys specified fields, univeral to all resource kinds/GRDs
    GrafanaMeta GrafanaMetadata `json:"grafanaMetadata"`
    // kind-specific metadata
    KindMeta map[string]any `json:"customMetadata"`

    Spec   map[string]any `json:"spec,omitempty"`
    Status map[string]any `json:"status,omitempty"`
}

Now, CRDs don't allow additional subresources, which means we'll need to keep some shim in place to make all this work in cloud (where we still have to operate using a CRD representation with all those constraints). But since we're settled now on having our own aggregated apiserver, we could decide this is our target format and keep the CRD representation an implementation detail, rather than having to answer questions like "do we EVER want to serialize a grafana resource to a grafana shape, or do we only ever use k8s shape?"