Developer Platform MVP aka Demoable Golden Path

[puja108]

User Story

As a Developer I want to be able to create a new software project (written in golang) I want to get everything from pipeline setup through deployment to production up to introspection and alerting at runtime out-of-the-box with as minimal intervention from my side as needed I want my entire workflow to be as self-service as possible, especially without having to involve a platform team or other areas of my organization.

Details, Background

The goal of this MVP is a minimal golden path example that is demoable to customers and other stakeholders.

We want to be able to use it to test out some of our assumptions on the customer/user side by showing a full end-to-end usable golden path.

As a side goal we want to also test our assumptions on the technology side, especially seeing how far we can get by relying as much as possible on standard open source components and glueing those together.

### Horizon
- [ ] https://github.com/giantswarm/giantswarm/issues/27924
- [ ] https://github.com/giantswarm/giantswarm/issues/27086
- [ ] Get a project with a customer and define a timeline
- [ ] Talk to the teams and get their epics in here

### Honeybadger
- [ ] https://github.com/giantswarm/roadmap/issues/2774
- [ ] Create a project including its source code repo
- [ ] Deploy a project
- [ ] Setup OCI registry for images and charts
- [ ] View status of my projects and deployments

### Atlas
- [ ] Get out-of-the-box metrics for deployed projects
- [ ] Get logs for deployed projects
- [ ] Get basic alerts for deployed projects
- [ ] Have a generic Dashboard for deployed projects
- [ ] Get custom monitoring and alerting based on SLO
- [ ] Allow to configure custom metrics and a grafana dashboard

### Shield
- [ ] Scan for vulnerabilities at build time
- [ ] Scan for vulnerabilities at run time
- [ ] Scan for anomalies at run time
- [ ] Verify signatures in cluster (with Kyverno?)

### Cabbage
- [ ] Have Ingress setup out-of-the-box
- [ ] Get DNS and TLD config ootb
- [ ] Blackbox monitoring of network
- [ ] Blackbox monitoring through service mesh
- [ ] Service Identity / mTLS for services

[puja108]

Added the tasks we gathered in the miro to tasklists per team.

@weatherhog (for honeybadger) I only added the high level ones that were marked for the MVP. Can you and @piontec detail out the high-level issues with the detailed tasks (most of which we already have in the phases in the board)?

@stone-z @TheoBrigitte @weatherhog (for Cabbage) I added some ideas that we thought of in the sprint this week. You can add more where you see fit. For context the recording of today's Platform Sync will be shared soon. Let's go through the items for your teams together and see which ones and in which scope can/should be done within the scope of the MVP and which ones we should rather move to future releases.

I'll also be creating further issues, where we can for now park features we want to build but will not be needed for this stage of the MVP.

[puja108]

Here the link to the recording https://drive.google.com/file/d/1mcOJ3mguKsxyfVS1yhAD_X883vqT1nYd/view

[teemow]

@puja108 this is the epic we are currently working on

[uvegla]

We had a nice discussion in HoneyBadger about the current state and I would like to collect my thoughts about the current state based on my learnings building a similar system for some years.

Template repositories

I think they are good for quick start / demo purposes but in my experience they are very difficult to maintain in the long run:

• The Platform Team is developing the template repositories with new features / changes as the platform evolves, but it is difficult to propagate those changes back to the already generated repositories / project. There is no mechanism or solution out of the box for this problem, once you generate a repository it is living its own life and special snowflakes / divergences will come up. Maybe most repos will remain very similar, but you still have a migration on your hand to propagate the changes from the template. A usual problem here is that who is responsible for the migration: the platform team (why should they know all the services and their special needs, quirks) or the individual dev teams (is not the platform there for them "just use" and to build on it. As a platform engineer I think not, but I agree there are some fine lines here depending on the nature of the platform change)
  • There are tools to help with this, e.g. our own devctl + some other repos does something like this, but what usually lacking here is tracking the progress on the opened PRs. Some will fail and need manual intervention + teams to review and merge them. A migration is only good if fully done. Without overview you will have projects in all sorts of different states. I have some experience using Sourcegraph Batch Changes here that I think it has a solution for this problem in a good way, but Sourcegraph also does a lot more and a paid product so probably would not fit all needs here. Bringing them up as good example for this particular problem.
• Every company / team / branch use case is different. Building a platform that suits all / very generic will make it very complex, which kind of defeats the purpose or if very opinionated then often users need to find quirks and workarounds to build around the platform which is not optimal either and a nightmare to maintain and evolve the platform from a platform developer perspective.

Thoughts on Phase 1

In my experience it is okay if we think of this as using it for demo purposes. However if we aim for fully supporting this phase a projects lifecycle we lack - at the moment - thinking about some crucial parts, as creating a project is one thing and the easiest one, but:

• Developing, testing and debugging is where users spend their times on projects. It sounds great that you can bootstrap a project and run it somewhere in a short period of time (managers love it) but if it is a pain to develop, test and debug you quickly loose all the benefits you gained in the first 5 minutes and find yourself in a murky swamp for the rest of the 99.999% of the project's lifetime.
• Of course how you do these highly depend on the language and framework and many times the given company's needs and culture, but I try to collect some generic problems here:
  • Local development: tools are needed locally, if you really want to support this nicely, the local toolchain developers have is ideally the same. You can imagine this is a lot of work and also a cultural thing. Thinking of python - which is the one of the most used language in recent years - is super bad here, it would deserve a post of its own. But you could think of JVM version and type or SSL libraries, GNU tools, etc.
  • Actually testing a micro service beyond unit tests is very hard. You most probably want to have multiple other services running that the given service interacts with to test a flow and (!) you want to have proper data for those services as well. Running 10,20,30+ micro services locally is ranging from a nightmare to impossible due to all the manual work it would take in a bad platform and the limits of the local machine at (maybe not even so) extreme cases.
  • This makes it best in my experience to work with remote developer clusters. The plural here is intentional, just think of multiple developers working on the same project + each of them probably needs different test data too. Another issue here is whether you want to persistent or ephemeral / disposable clusters here each bringing its own stack of problems (cost, warmup time, scaling, one cluster with tenants or many actual clusters, persistence for databases, mimicking cloud services like S3, caches or Kafka for example, etc.)
  • Feedback loop: running through a single local change - imagine you had a typo - through the whole CI/CD pipeline to see the change results in super slow and annoying feedback loops making the development super slow. There are tools like Skaffold I have some experience with from years ago but setupt again depends on the language.
  • Debugging is tightly coupled here when you have your stuff running in a remote cluster. For JVM based stuff it can be really nice, for e.g. python, not so much. I don't know have experience in this regards with go, but again, highly depends on the language. Plus you probably want to build the container images remotely too which is another major layer for this problem space.

Projet settings file

Highly depends on what we want to store here. In my experience this is nice approach but can turn bad quickly if some commonly principles are not laid down and followed:

• Start in mind that there are different kind of projects, one wont fit all.
• Have schema, versions and validations for it.
• If multiple platform / SRE teams add things to the project descriptor and start cross (and sometimes miss-) using them in many different tools, you will have a bad time making changes in the future.

• Storing deployment configuration in them may not be a good idea. Imagine you have an issue that needs to be solved ASAP or the service just needs to be scaled fast or some some new configuration value is needed you would need to rebuild the whole service through all the CI/CD pipeline (yea, never seen issues with CI, they just work) to propagate that change easily taking up to one hour maybe. Then imagine you realise some other values need to change too or you updated the wrong number...

  Phase 0

I think it is fine we don't want to figure it all out right away, but it would have a huge impact on our MC and our current plans with them. Thinking of tenants vs admin access vs current MAPI / RBAC here.

Phase 2

I think we are already the strongest here. Crossplane can be a good bet, I see the value here. I personally had some bad experience with in the start but they are changing fast and from what I hear for the better. Thus my fears probably from my bad experience and lack of knowledge on this topic.

[stone-z]

Is the intent that we edit the items directly in the list or should there be some proposal/discussion first?

[puja108]

I guess it might need a bit of discussion, so a proposal by each team would be good, so we can edit the list based on that

[stone-z]

Shield intends to focus first on the platform management experience for the platform/security teams, with the developer features being primarily feedback that the platform teams don't need to manage.

These features include (roughly in order of when we want to do them):

• Automated adoption of new security policies
  • Platform team decides to enforce a new policy, our platform handles applying it without disturbing existing workloads
• Automated feedback to component owners
  • Open issues or automated PRs for teams to resolve issues in their workloads
  • Used during the above automated adoption rollout to notify teams of the required change, and for management tracking
• Streamlined gitops-based exception management
  • Each org has different exception policies. Enforcing them is a bottleneck. Policy API allows platform teams to design a workflow that works for the org. We are building toward a rough interface of team requests exception --> platform handles automated approvals --> reviewers handle remaining manual approvals --> platform manages the lifecycle of that exception
• Support for more types of policies
  • Only PSS is currently supported. We'd like to allow platform teams to slowly roll out e.g. more restrictive network policies, less privileged RBAC, service mesh adoption, etc.
• Provide a shared configuration interface for security notification / alert routing
  • For example, the platform team configures in one place the Slack or Splunk/Elastic endpoint where all (anomaly, vulnerability, policy, etc.) reports should be sent

All of the above include exposing metrics. Reports from the underlying tools are already available and Policy API CRs will include more data in a tool-independent format.

Due to the difficulty of managing CI/CD pipelines Giant Swarm doesn't own, we don't plan to do much (if any) pipeline work with the resources we currently have. However, we might explore "out of band" options for supporting these features in CI. For example, maybe we don't provide a Jenkins config for all the scanners we use, but we might instead provide the policies in a CLI-friendly format to be used, or we expose an endpoint configured with the customers' current policies against which a dry run can be made from CI to validate the workload being built.

The features currently listed in the main issue are mostly already available:

• Scan for vulnerabilities at build time - we won't do this, except as described above
• Scan for vulnerabilities at run time - we do this already, and will improve the interface for customers to consume the data
• Scan for anomalies at run time we do this already, and will improve the interface as well as potentially offer response or automation for certain anomalies (long term goal)
• Verify signatures in cluster - this can already be enabled. We will improve it by signing Giant Swarm resources and including supply chain policies as optional managed policies. The limitation here is mostly that it can be difficult for customers to adopt signature workflows in build pipelines.

[puja108]

Moving this to waiting, as we lowered priority on this to first work on the foundation that can enable a real golden path.

We will definitely revive this once we feel the foundation is laid.

Putting this on hold does not mean we stop discovering and co-innovating with customers around this topic and how we can already help them towards such a goal.