Feat/Improvements on AWS Cloudwatch Elasticsearch monitoring

slopezz commented 3 years ago

Adds AWS CW ES FreeStorage Sum metric (example configuration configmap and grafana dashboard). A part from having individual node free space, it is good the overall free space (sometimes nodes gets unbalanced, this metric helps to detect it)
Update AWS CW ES grafana SingleStats panels to show instant value (so last one), it is more efficient in terms of promql performance, and in case of multiple values because pods have been recreated so multiple timeseries with the same metric, it just shows last one. It is possible that this change will be aplied to all dashboards SingleStats in future PRs (avoids the error "Multiple timeseries only can be shown one" (or something like that), when pod is recreated.
Adds more examples of AWS CW ElasticSearch alerts (prometheus rules)
Operator Release v0.3.4

End to end ests are not passing on GitHub action, there is a problem creating the kind cluster on GitHub Action CI, giving a timeout after 5minutes:

 • Waiting ≤ 5m0s for control-plane = Ready ⏳  ...
 ✗ Waiting ≤ 5m0s for control-plane = Ready ⏳
 • WARNING: Timed out waiting for Ready ⚠️

Something similar happens on marin3r repo https://github.com/3scale-ops/marin3r, since a few weeks ago tests are nor passing due to a problem between kind and GH action.

Testing locally, e2e tests woks:

$ make test-e2e
/home/slopez/work/es-prom-exporter/prometheus-exporter-operator/bin/kind-v0.10.0 create cluster --wait 5m
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Waiting ≤ 5m0s for control-plane = Ready ⏳ 
 • Ready after 55s 💚
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Thanks for using kind! 😊
docker build -t quay.io/3scale/prometheus-exporter-operator:v0.3.4 .
Sending build context to Docker daemon 129.6 MB
Step 1/6 : FROM quay.io/operator-framework/ansible-operator:v1.5.0
 ---> 5d900455b2e5
Step 2/6 : COPY requirements.yml ${HOME}/requirements.yml
 ---> Using cache
 ---> ee9eaba506f6
Step 3/6 : RUN ansible-galaxy collection install -r ${HOME}/requirements.yml  && chmod -R ug+rwx ${HOME}/.ansible
 ---> Using cache
 ---> c432f94978b3
Step 4/6 : COPY watches.yaml ${HOME}/watches.yaml
 ---> Using cache
 ---> 1f9c8ccaead2
Step 5/6 : COPY roles/ ${HOME}/roles/
 ---> c0eb588f7144
Removing intermediate container 56fb144d3f48
Step 6/6 : COPY playbooks/ ${HOME}/playbooks/
 ---> d9aec8c94fd9
Removing intermediate container 7ef4adcaae10
Successfully built d9aec8c94fd9
/home/slopez/work/es-prom-exporter/prometheus-exporter-operator/bin/kind-v0.10.0 load docker-image quay.io/3scale/prometheus-exporter-operator:v0.3.4
Image: "quay.io/3scale/prometheus-exporter-operator:v0.3.4" with ID "sha256:d9aec8c94fd987b5ade95eb078a4c90cb5f9e6dc32d1d51d8091f59201bcee0c" not yet present on node "kind-control-plane", loading...
cd config/manager && /home/slopez/bin/kustomize edit set image controller=quay.io/3scale/prometheus-exporter-operator:v0.3.4
/home/slopez/bin/kustomize build config/testing | kubectl apply -f -
namespace/prometheus-exporter-operator-system created
customresourcedefinition.apiextensions.k8s.io/prometheusexporters.monitoring.3scale.net created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/grafanadashboards.integreatly.org created
serviceaccount/prometheus-exporter-operator-controller-manager created
role.rbac.authorization.k8s.io/prometheus-exporter-operator-leader-election-role created
role.rbac.authorization.k8s.io/prometheus-exporter-operator-manager-role created
rolebinding.rbac.authorization.k8s.io/prometheus-exporter-operator-leader-election-rolebinding created
rolebinding.rbac.authorization.k8s.io/prometheus-exporter-operator-manager-rolebinding created
service/prometheus-exporter-operator-controller-manager-metrics-service created
deployment.apps/prometheus-exporter-operator-controller-manager created
mkdir -p /home/slopez/work/es-prom-exporter/prometheus-exporter-operator/bin
curl -sL -o /home/slopez/work/es-prom-exporter/prometheus-exporter-operator/bin/kuttl-v0.9.0 https://github.com/kudobuilder/kuttl/releases/download/v0.9.0/kubectl-kuttl_0.9.0_linux_x86_64
chmod +x /home/slopez/work/es-prom-exporter/prometheus-exporter-operator/bin/kuttl-v0.9.0
/home/slopez/work/es-prom-exporter/prometheus-exporter-operator/bin/kuttl-v0.9.0 test
=== RUN   kuttl
    harness.go:457: starting setup
    harness.go:248: running tests using configured kubeconfig.
    harness.go:285: Successful connection to cluster at: https://127.0.0.1:40277
    harness.go:353: running tests
    harness.go:74: going to run test suite with timeout of 120 seconds for each step
    harness.go:365: testsuite: ./test/e2e/ has 2 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/operator
=== PAUSE kuttl/harness/operator
=== RUN   kuttl/harness/prometheusexporter
=== PAUSE kuttl/harness/prometheusexporter
=== CONT  kuttl/harness/operator
    logger.go:42: 14:22:54 | operator | Creating namespace: kuttl-test-assuring-redfish
=== CONT  kuttl/harness/prometheusexporter
    logger.go:42: 14:22:54 | prometheusexporter | Creating namespace: kuttl-test-fit-ant
=== CONT  kuttl/harness/operator
    logger.go:42: 14:22:54 | operator/0- | starting test step 0-
=== CONT  kuttl/harness/prometheusexporter
    logger.go:42: 14:22:54 | prometheusexporter/1-prometheusexporter | starting test step 1-prometheusexporter
=== CONT  kuttl/harness/operator
    logger.go:42: 14:22:55 | operator/0- | test step completed 0-
=== CONT  kuttl/harness/prometheusexporter
    logger.go:42: 14:22:55 | prometheusexporter/1-prometheusexporter | PrometheusExporter:default/example-memcached created
=== CONT  kuttl/harness/operator
    logger.go:42: 14:22:55 | operator | operator events from ns kuttl-test-assuring-redfish:
    logger.go:42: 14:22:55 | operator | Deleting namespace: kuttl-test-assuring-redfish
=== CONT  kuttl/harness/prometheusexporter
    logger.go:42: 14:23:26 | prometheusexporter/1-prometheusexporter | test step completed 1-prometheusexporter
    logger.go:42: 14:23:26 | prometheusexporter | prometheusexporter events from ns kuttl-test-fit-ant:
    logger.go:42: 14:23:26 | prometheusexporter | Deleting namespace: kuttl-test-fit-ant
=== CONT  kuttl
    harness.go:399: run tests finished
    harness.go:508: cleaning up
    harness.go:563: removing temp folder: ""
--- PASS: kuttl (33.74s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/operator (1.23s)
        --- PASS: kuttl/harness/prometheusexporter (31.92s)
PASS

/kind feature /priority important-soon /assign

3scale-robot commented 3 years ago

LGTM label has been added.

Git tree hash: 8146d190c961cdcc6c0062344dd995814d86c7f4

slopezz commented 3 years ago

/approve

3scale-robot commented 3 years ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slopezz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/3scale-ops/prometheus-exporter-operator/blob/main/OWNERS)~~ [slopezz] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

raelga commented 3 years ago

/retest

3scale-ops / prometheus-exporter-operator

Feat/Improvements on AWS Cloudwatch Elasticsearch monitoring #30