Documentation request: Cluster-Stack Hosting Format

[mxmxchere]

The cluster-stack-operator has the task to render and apply helmcharts from a GitHub Release assets into a kubernetes cluster. I was not able to find information about the format that the operator expects the GitHub Release to be in (which orga, which repo, which release-name, which release assets it expects and in which format). I have an example CRD to describe my question:

apiVersion: clusterstack.x-k8s.io/v1alpha1
kind: ClusterStack
metadata:
  name: clusterstack-openstack
  namespace: scs-cs
spec:
  provider: openstack # this influences the path where the controller searches for releases
  name: scs           # same here
  kubernetesVersion: "1.27"
  channel: stable
  autoSubscribe: false
  noProvider: true    # do not require dedicated image controller 
  versions:
    - v1

GitHub Orga and Repo

is provided via an env variable to the operator (needs to be documented).

-> also this leads to the issue that it is not possible to consume clusterstacks from two different Repositories, it would be better to reference the github-release-source via the clusterstack CRD

GitHub Release Name

The cluster-stack-operator does not track a git-repo but GitHub Releases (Files downloadable via HTTP and accessible via a JSON-API https://docs.github.com/en/rest/releases?apiVersion=2022-11-28). As far as i found out the release has to have the name "{{.spec.provider}}-{{.spec.name}}-{{.spec.kubernetesVersion}}-{{.spec.versions[x]}}" where x is a valid index of the spec.versions list. Also the dot in the kubernetesVersion has to be replaed with a dash.

GitHub Release assets

Now my findings include a good portion of guessing:

metadata.yaml

Each GitHub release needs a metadata.yaml, below is one that is working

apiVersion: metadata.clusterstack.x-k8s.io/v1alpha1
versions:
  clusterStack: v1
  kubernetes: v1.27.3
  components:
    clusterAddon: v1
    nodeImage: v1

It is not clear to me what this file does, nor why it is needed. But if it is missing the cluster-stack operator throws an error. If a clusterstack wants to bundle different assets (addons and clusterclass for example) would it not be enough to do so via the release as containing structure?

{{.spec.provider}}-{{.spec.name}}-{{.spec.kubernetesVersion}}-cluster-class-vx.tgz

where x is 1 or 2 in the provided examples but i really do not know which vx references which vx. In the example that i use, i used 1 everywhere that works, but it would be great to find out which vx references which. Also i am a bit confused because i found this in the sourcecode: https://github.com/SovereignCloudStack/cluster-stack-operator/blob/main/internal/controller/clusterstackrelease_controller.go#L220 It seems that the operator looks up 4 hardcoded release assets: "metadata.yaml", "cluster-addon-values.yaml", "cluster-addon", "cluster-class". Does that work too? From my point of view that would be an even better solution because all the bundling and versioning already took place on the layer of the release, so i see no need to version it again.

So this issue is half-documentation and and half-guessing, so a lot of it may be wrong, which is why we need correct documentation (if we dont have that already).

[janiskemper]

Hi @mxmxchere , thanks for the issue! You have valid questions and I can see that you spent some time with the cluster stack operator and with cluster stacks.

Your questions are expected.

However, we have not done any documentation on building cluster stacks, which is what your questions are about.

Therefore, I would suggest this issue to be moved to the cluster-stacks repo, as the documentation about building and releasing cluster stacks should go there (correct me if I'm wrong @batistein @jschoone ).

Second, I would say, since you are the first "outside" person to try this, that you get in contact with @batistein to get answers to your questions.

We are currently in the process of making things easier and it is not worth documenting the existing state if it would change a lot later IMO. This also is up to @jschoone of course.

The goal is definitely to have good documentation on how to build and release cluster stacks. The question is just how to approach it and how to move fast in your case.

[jschoone]

Hi @janiskemper and @mxmxchere,

to make sure I understand things here, this is a a chicken or egg problem? So the guesswork is done using the existing repository with some reverse engineering because of lack of a documentation. I agree with it's not ideal but also with it's probably senseless to document the current state as it probably contains not needed stuff for "real" providers, since the Cluster Stacks are a whole new software product where references don't exist. This shouldn't be an excuse for a not existing documentation, but since it's the current state, I would like to do this on-the-fly while we work on the Openstack integration. This must not be in a perfect format, I think issues like this are a good start.

And yes: I'd also see this issue in the Cluster Stacks repo. Should I move it?

[mxmxchere]

Regarding the place where the documentation "lives": Since the documentation only describes how the operator works or which scheme in GitHub Releases it expects the documentation has to document the operator and the documentation belongs to the operator and not in a repository where potential data lives that the operator consumes (cluster-stacks).

An analogy: helm's behaviour is also documented in a helm domain and not in a helm-chart repo (if, it would even be the question which, there are thousands of repos)

What would also be possible (but i would rather see this as an additional help-resource and not as something that can replace proper documentation) would be a repository with a skeleton and maybe a few scripts that automatically create the correct releases.

[janiskemper]

I think I agree with your point that the operator could document what releases it expects.

However, this would not solve your problem of understanding how to obtain these release assets.

All questions about how to build cluster stacks e.g. for OpenStack should not be in this repo IMO.

[mxmxchere]

I can move the issue elsewhere, no problem. Would you prefer it to be in https://github.com/SovereignCloudStack/cluster-stacks?

[janiskemper]

I would say that makes sense, yes!

[mxmxchere]

Is there documentation yet? Building clusterstacks still involves some guessing / trial&error / reading the source code.

[janiskemper]

There is no documentation planned for building cluster stacks for now. We work on a tool that helps you to get the release assets from a cluster stack. The cluster stack should look pretty much like the existing ones.

Main things to consider are the following:

• Helm Chart which creates a functioning ClusterClass is needed
• Helm Chart containing all required Cluster Addons is needed
• A flow for node images should be established, but that is highly provider-dependent

[mxmxchere]

I just stumbled upon some docs that help regarding this. They are meant to explain how to use the CSO in local mode, but at least they give some hints about the format.

[janiskemper]

there are some docs in csctl about building cluster stacks as well now!