Use k8s Service to create fake avalanche targets for Prometheus

[arajkumar]

Currently we use k8s StatefulSet and native replica to create multiple Avalanche instances to generate series load to Prometheus. However this is less optimal and occupies real CPU & memory in a benchmarking environment.

This fix creates fake Service to mimic existence of multiple avalanche with less demanding resources.

A new helm value fakePrometheusTargetCount has been introduced to declare the number of fake prometheus targets to be created for avalanche.

Signed-off-by: Arunprasad Rajkumar ar.arunprasad@gmail.com

What this PR does / why we need it

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer

Checklist

• [X] Chart Version bumped
• [X] CLA signed

[niksajakovljevic]

@paulfantom I wonder why we use STS for Avalanche? Can you please clarify what is negative impact of individual pods?

I don't see why we need to switch from using STS to using static Pod. Such switch has negative impact on a scenario where avalanche is sending data via remote_write.

[paulfantom]

We are using STS as we need stable network identifiers and STS is providing them out of the box. We don't want to use static pods as it is not a canonical way of managing any application in kubernetes and it should be used only in very small subset of specific cases.

[arajkumar]

@paulfantom Adding multiple service monitor's not creating multiple targets, instead I created multiple services pointing to same pod which indeed creates multiple prometheus targets.

[arajkumar]

@paulfantom CI failure is due the following line where the grep exits with 1 when only avalanche charts are modified.

changed=$(ct list-changed --target-branch ${{ github.event.repository.default_branch }} | grep promscale)

I fixed it in the latest commit, PTAL.