Add Prometheus tests

[witekest]

Check if API responds correctly on /-/healthy and/or /-/ready endpoints.

Implements SUSE/spacewalk#24546

[witekest]

This currently doesn't work with docker, I can reproduce a bunch of failures locally by running:

CONTAINER_RUNTIME=docker tox -e prometheus -- -n auto

I don't understand why it doesn't work. Do you have a hint?

[dcermak]

EDIT: we have healtchecks in the images now, so this should no longer be necessary. Rest of the comment left for historical purposes.

I don't understand why it doesn't work. Do you have a hint?

I think docker starts the containers too fast and they are not yet ready when you launch the tests. There's two ways, you can change the requests to be repeated via tenacity as follows:

diff --git a/tests/test_prometheus.py b/tests/test_prometheus.py
index def8c5b..aca1724 100644
--- a/tests/test_prometheus.py
+++ b/tests/test_prometheus.py
@@ -1,5 +1,6 @@
 """Tests for the Prometheus containers."""

+import tenacity
 import pytest
 import requests
 from pytest_container.container import ContainerData
@@ -15,6 +16,13 @@ PROMETHEUS_AND_BLACKBOX_CONTAINERS = (
 )

+@tenacity.retry
+def _fetch_prometheus_status(port: int, route: str) -> requests.Response:
+    resp = requests.get(f"http://localhost:{port}/-/{route}")
+    resp.raise_for_status()
+    return resp
+
+
 @pytest.mark.parametrize(
     "container", PROMETHEUS_STACK_CONTAINERS, indirect=True
 )
@@ -22,7 +30,7 @@ def test_prometheus_ready(container: ContainerData) -> None:
     """Simple smoke test verifying that Prometheus is ready."""

     port = container.forwarded_ports[0].host_port
-    resp = requests.get(f"http://localhost:{port}/-/ready", timeout=2)
+    resp = _fetch_prometheus_status(port, "ready")
     assert resp.status_code == 200
     assert resp.text in ["Prometheus Server is Ready.\n", "OK"]

@@ -34,6 +42,6 @@ def test_prometheus_healthy(container: ContainerData) -> None:
     """Simple smoke test verifying that Prometheus is healthy."""

     port = container.forwarded_ports[0].host_port
-    resp = requests.get(f"http://localhost:{port}/-/healthy", timeout=2)
+    resp = resp = _fetch_prometheus_status(port, "healthy")
     assert resp.status_code == 200
     assert resp.text in ["Prometheus Server is Healthy.\n", "OK", "Healthy"]

Alternatively you can also consider adding a HEALTHCHECK to the container images (would be preferred, but that'll take longer)

[dcermak]

Sadly, https://build.suse.de/request/show/337271 cannot go in, as the grafana stack is in packagehub (and thereby cannot be made part of SLE). Could you then exclude the containers, that you added in this PR, from the test_no_orphaned_packages test in tests/test_all please?

[dcermak]

Thanks a lot for working on this!