Re-architecture to better fit the modular nature of the project

[kurktchiev]

The project's current state allows for a great showcase of the general idea behind The BACK Stack. However, it poses challenges regarding the lifecycle management of components and the modularity of said components.

To alleviate some of these challenges, I am proposing a re-architecture that will ultimately enhance the original Hub/Spoke idea and allow us to better:

• manage individual hub components
• lifecycle management of those components
• control installation/or not of components
• include new packages as needed
• break out the core code from example implementations

This new design makes a few assumptions:

1. Hub clusters are a single tenant that houses the BACK stack HUB components, nothing else
2. We are basing the install/upgrade/uninstall process through porter
3. Each component is installed through helm
4. Break out the cluster/app claims to their own repository

It also introduces a new concept to the Hub/Spoke, called a Bead. Beads are 3rd party packages that allow an end user to extend the platform to their liking.

Below is the diagram describing the breakouts and decisions around the re-architecture:

Also, the following diagram is my current sketch of how I envision Beads working:

cc @JimBugwadia @patelrit @morey-tech for comments

[morey-tech]

Love the new design! Something to consider:

We are basing the install/upgrade/uninstall process through porter

It's common for Argo CD to manage itself after the initial bootstrapping, and it could also manage the other components of the hub. I wonder if Porter should be solely responsible for the install/uninstall but not the upgrade and ongoing operation of the hub components.

This would reduce the complexity we need in Porter. It could simply create the Hub cluster and bootstrap it with Argo CD. Then, uninstallation is simply deleting the hub cluster. In the long term, we could integrate Porter with Terraform (OpenTofu) to create the hub cluster on cloud providers and clean them up after uninstalling them.

Upgrading would be managed through GitOps, with changes to the manifests in the hub cluster repo. Argo CD can do the deployment. If there's any special logic we need to implement for those upgrades, it should be done using resource hooks in the Applications.

This answers how Beads are managed; Argo CD would handle them just like the other hub components.

[kurktchiev]

I am on board with using Argo as a the source of truth for things, take a look over at https://github.com/back-stack/bead-gke I am working through building out what adding GKE support can look like. I think overall things can be moved into Argo except the backstage stuff which is wonky and I am not sure what the best way to handle the wiring into that is honestly. It requires either manual interventions of a configmap potentially or whole new images for backstage, neither option is super appealing.

[morey-tech]

except the backstage stuff which is wonky and I am not sure what the best way to handle the wiring into that is honestly. It requires either manual interventions of a configmap potentially or whole new images for backstage, neither option is super appealing.

1. What is the official recommendation from the Backstage project for people adding in their customizations?
2. Maybe we should treat Backstage as a bead (e.g., if a user wants to customize it, they have to bring their own Backstage).

[kurktchiev]

There is no official way to specifically enhance it once the image has been built outside of rebuilding. There are ways to extend it to a point (the gke example leans onto that by using a catalog that can be imported to add in functionality), but if there is need for say new plugins to be added etc, it requires a new image altogether.

I am not against backstage being a Bead tbh, it is a nice interface into the system for those that want it but I think the heart of things should be git+argo/porter

[JimBugwadia]

Have we looked at Backstage templates (blog, docs) for extensions?

I feel we need to define a higher level abstraction to add a Service to a BACK Stack deployment, which allows declarative configuration of:

1. Backstage Template
2. ArgoCD Application
3. Crossplane Composite
4. Kyverno Policy Set

Perhaps the minimal requirement is the ArgoCD Application and the rest are optional?

I am still trying to understand what exactly a Bead is....did you have something similar to the above in mind?

This can be formalized as a new CRD.

[kurktchiev]

Yes a Bead is exactly what you described and in the linked repo I am building out what the specific needs for each of those components looks like.

And yes the backstage templates is what I am using for extending but as I said there are limitations with the running image and requirements a template can have.

-B On Apr 21, 2024 at 15:12 -0400, Jim Bugwadia @.***>, wrote:

Have we looked at Backstage templates (blog, docs) for extensions? I feel we need to define a higher level abstraction to add a Service to a BACK Stack deployment, which allows declarative configuration of:

1. Backstage Template
2. ArgoCD Application
3. Crossplane Composite
4. Kyverno Policy Set

Perhaps the minimal requirement is the ArgoCD Application and the rest are optional? I am still trying to understand what exactly a Bead is....did you have something similar to the above in mind? This can be formalized as a new CRD. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were assigned.Message ID: @.***>

[JimBugwadia]

Yes a Bead is exactly what you described and in the linked repo I am building out what the specific needs for each of those components looks like.

Naming aside, is this just a set of 1-4 K8s resources in a Helm chart or do we need a CRD and some lifecycle management via a Golang controller?

And yes the backstage templates is what I am using for extending but as I said there are limitations with the running image and requirements a template can have.

Is this a bug that is being addressed, or a more permanent limitation?

The blog post makes it seem like templates should simply work via declarative configuration. Is that not the case?

[kurktchiev]

Naming aside, is this just a set of 1-4 K8s resources in a Helm chart or do we need a CRD and some lifecycle management via a Golang controller?

For the most part yes, it is a few k8s resources, that I think in general as @morey-tech suggested can be Argo controlled, so I don’t know that we need a specialty controller for. As example defining the Back stack Crossplane CRDs https://github.com/back-stack/bead-gke/tree/main/crossplane (which requires building an OCI image, check the supplied action) and then configuring and applying them to the cluster https://github.com/back-stack/bead-gke/tree/main/porter/manifests/crossplane

Is this a bug that is being addressed, or a more permanent limitation? The blog post makes it seem like templates should simply work via declarative configuration. Is that not the case?

Yeah so the blog shows you how to add them locally to the image, which the process does not work the same way on a live running instance, and that is a current design limitation of backstage. I am using templates (https://github.com/back-stack/bead-gke/tree/main/backstage) and the overall process described in the blogs and docs to extend, however, there are a couple of intermediary steps that are less clean when it comes to making it work in a live container. It is also design limited around plugins, I have not found a way to add new plugins to backstage to a running instance and the only way to do it is to rebuild an image. With that said, there are some things we can do if I had a bit more React experience that can make some of this a bit better.

[JimBugwadia]

Yeah so the blog shows you how to add them locally to the image

What do you mean by image? The catalog?

This article lists a few ways to add templates, or other components, to the catalog:

https://dev.to/mccricardo/backstage-catalog-393c

[kurktchiev]

The catalog is one piece; that is how I am extending Backstage to support GKE https://github.com/back-stack/bead-gke/tree/main/backstage. However, in order to extend Backstage beyond what the catalog supports, you need to do full image rebuilds. As an example, if we wanted to install and support, say, Google Authorization on the front page, a number of components would have to be edited that cannot be controlled with templates, resources, and components.

[JimBugwadia]

Creating a custom image does not seem right to me.

I would suggest focusing on the extensions, and not the initial installs as we want flexibility in how each component is setup.

Lets discuss the requirements and the desired flow.

[kurktchiev]

That is unfortunately how backstage is designed and expected to operate at the moment :/, they have some new gen type work to address the very problem we are discussing but it is nowhere near baked or operational

-B On Apr 23, 2024 at 15:41 -0400, Jim Bugwadia @.***>, wrote:

Creating a custom image does not seem right to me. I would suggest focusing on the extensions, and not the initial installs as we want flexibility in how each component is setup. Lets discuss the requirements and the desired flow. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were assigned.Message ID: @.***>