Improve rebalance and add pause/resume exporters in camunda.mk

[chDame]

Add to make target to pause/resume exporters that will be useful for dual region failOver. And make the rebalance a oneline without sleep or kill. Cleaner

[upgradingdave]

I tested and was able to rebalance brokers and pause/resume exporters, looks good 😎

[falko]

@chDame @upgradingdave This change doesn't work for me:

$ make rebalance-leaders 
kubectl exec $(kubectl get pod --namespace camunda --selector="app=camunda-platform,app.kubernetes.io/component=zeebe-gateway,app.kubernetes.io/instance=camunda,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=zeebe-gateway,app.kubernetes.io/part-of=camunda-platform" --output jsonpath='{.items[0].metadata.name}') --namespace camunda -c zeebe-gateway -- curl -i localhost:9600/actuator/rebalance -XPOST
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "ae2dd0a850568d097a5157e487fcf4d2125a69e165cefbacc8c43607bb72e8d3": OCI runtime exec failed: exec failed: unable to start container process: exec: "curl": executable file not found in $PATH: unknown
make: *** [../../../../camunda-8-helm-profiles/include/camunda.mk:63: rebalance-leaders] Error 1

[upgradingdave]

Hi @falko , I have a working GCP cluster and just tested and seems to work fine for me.

The path to include/camunda.mk is calculated using a $(root) makefile variable defined in the Makefile here

The error message you pasted shows this as the path: ../../../../camunda-8-helm-profiles/include/camunda.mk:63

But I think the path should be this: ../../../../include/camunda.mk:63

So it seems in your environment that there's an extra camunda-8-helm-profiles/ directory in the $(root) path? Can you double check that you don't have any local changes that might be causing confusion?

[falko]

@upgradingdave it has nothing to do with the local files. curl doesn't seem to be available in the container: "curl": executable file not found in $PATH. I just also got the same problem with zbctl. Maybe the latest Zeebe images have been cleaned up to reduce attack surface?

[falko]

8.2.10 is the last image with zbctl after https://github.com/camunda/zeebe/pull/13706/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557L125

And due to the new base image curl is gone too. We should run curlimages/curl as a job. I already looked into it two weeks ago and found this example: https://theithollow.com/2019/12/16/kubernetes-jobs-and-cronjobs/

[upgradingdave]

Ah ha, sorry, I should have read the error message more carefully 😅

I just saw a comment about this https://nixery.dev on our slack recently ... maybe it could help here?

[falko]

Well, since there is an official curl image, I'd see that as the natural choice. Nixery would be a solution for zbctl: https://search.nixos.org/packages?channel=23.05&show=zbctl&from=0&size=50&sort=relevance&type=packages&query=zbctl

[falko]

46 already has the link to a rebalancing cronjob example