Documenting gateways in CCSM

[akeller]

After a discussion with @korthout we realized we might need more documentation on gateways in CCSM (CCSaaS too, but that's a separate issue). We realize this is a more involved conversation that requires product input to determine the documentation strategy.

Please keep in mind, with the docs vision we will document CCSaaS first and have a separate, dedicated section for CCSM. In this case I think it will be helpful given how little we'll discuss gateways from a CCSaaS perspective.

• How do Gateways work in CCSM? Running your own gateway? How do I deploy it? How do I configure it to work together with my cluster?
• What is the difference between a broker and a gateway?
• How do you add gateways for load balancing?
• What about gateways with HA?

We have config file templates, but should recommend an initial path for beginners to CCSM and offer educational opportunities to help developers understand where are the thresholds or what are the reasons you might move from one gateway configuration to another. What’s the preferred path for working with gateways?

Defaults are single items, likely not close enough to a realistic production scenario. Should we offer recommended settings? What are those settings?

@korthout's thoughts - Start with an embedded gateway. When they notice they need to run the gateway separate from the brokers, push them in that direction.

[akeller]

@berndruecker @felix-mueller @menski @meyerdan would like your thoughts on this topic.

[berndruecker]

I do see the need for some more explanation around that topic - but would question the priority. Do we already have concrete users asking for this information (so that this piece of information would provide immediate value)?

As for the content, I think our own cloud operations should know the best practices best ;-)

My personal answer to somebody asking me is:

• Run an embedded gateway for every test/eval scenario (despite load tests) - to keep it as simple as possible.
• Run two gateways (for HA) in prod scenarios - separated from the broker.

We might also find more arguments for a separate gateway, like for example security (brokers can only be reached via the gateway).

In our own load tests we did not see that gateways were bottlenecks, so more might not be necessary. But maybe that knowledge is outdated and has changed since... I try to do a quick fact check with @falko later today

[berndruecker]

I did discuss this with Falko briefly. We came up with the following best practices to run gateways

• Run an embedded gateway for test/evaluation/play-around scenarios to keep them as simple as possible. Exception: Load tests, as they need to have a close-to-prod environment.
• Run a dedicated gateway separated from the broker for prod scenarios. This makes sure that any load on the broker cannot disturb the gateway from taking requests. Additionally, the brokers and gateways are isolated from each other (meaning smaller config files each, possibility to scale them independently, crashed don't affect the others, ...). Also security is slightly increased, as you can't access brokers from the outside directly (https://en.wikipedia.org/wiki/DMZ_(computing)) - all traffic goes via the gateway. : SSL only till the gateway - broker in DMZ
• Run two instances of the gateways if you need high availability. Load distribution and failover are handled by Kubernetes out-of-the-box (Kubernetes Service with Load Balancer for the gateway). As the gateway is stateless, it is really easy to scale out. However, in our own load tests, we seldom see the gateway being a bottleneck, so you might not need to run more than two instances.

The helm charts (https://github.com/camunda-community-hub/zeebe-helm) can provision all of this.

[npepinpe]

One small thing, there's already a page about health probes for the gateway: https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/gateway-health-probes/

The port used is 8080, which is not the default port, so it would be great if instead we used the default port in the docs. I mention it here as I assume we will consolidate this information together in the same section.

[Zelldon]

I will use this issue to work on KR: Configuration and best practices for the Zeebe standalone gateway are documented

Last summary https://camunda.slack.com/archives/C03DFQQHPM0/p1651841920186529?thread_ts=1651212570.917299&cid=C03DFQQHPM0 states what we want to archive here:

Let me try to summarize and define what we want to archive with the KR for this quarter.

• We want to document the gateway in general in a better way for the user. What is the gateway and why do we have it? Maybe what is the task/problems it solves? How it does do it?
  • Might be useful to point out that it is recommended to use a standalone gateway, which is the default in self-managed, but it can also be run as embedded (which is useful for testing). Plus: Point out the advantages of running it standalone vs embedded.
• We want to focus on self-managed for now, since this is the most interesting part for the user, especially in regards to configuration
• I will add a component section here https://docs.camunda.io/docs/self-managed/overview/, since this is then similar to the other components
  • Especially it is similar to Identity which is also explained here as a self-managed component https://docs.camunda.io/docs/self-managed/identity/what-is-identity/

Based on that I will define for myself the following checklist:

• [x] Create a new section in self-managed documenting the Gateway
  • [x] Document what is a gateway, and what it does (what is the difference to a broker)
  • [x] What problem does it solve; Why we have it
  • [x] Potentially we move parts later so we have a component section and some parts in self-managed, we will see.
  • [x] Document differences between an embedded and standalone gateway. When to use which.
  • [x] Add some recommendations and reasonings
  • [x] How can it be configured, which should be configured in certain cases
  • [x] Kubernetes/Helm Configuration ?