search

runwhen-contrib / rw-public-codecollection

RunWhen Public Codecollection Repository - Open Source troubleshooting runbook library for Kubernetes and cloud infrastructure components.
Apache License 2.0
39 stars 5 forks source link

Libraries/RW/Prometheus - Optional Headers are now truly optional #139

Closed Hashfyre closed 5 months ago

Hashfyre commented 5 months ago

Objective: Run RW.Prometheus based codebundle against a bare unauthenticated Prometheus endpoint Codebundle / Robot Script: Infracloudio/ifc-rw-codecollection/.../runbook.robot

Test Procedure:

Documentation:
Performs a query against the prometheus instant API for metrics with a single data point.

Start / End / Elapsed: 20240125 12:38:53.207 / 20240125 12:38:53.208 / 00:00:00.001

12:38:53.208 TRACE Arguments: [ api_url='http://af87d61af0d814b8d8e99d37987b291d-483479890.us-west-2.elb.amazonaws.com/prometheus/api/v1' | query='aws_rds_database_connections_average{dimension_DBInstanceIdentifier="robotshopmysql"} > 1' | step='30' | target_service=Service(url='https://curl.sandbox.runwhen.com') ]

12:38:53.208 FAIL AttributeError: 'NoneType' object has no attribute 'value'

12:38:53.208 DEBUG Traceback (most recent call last): File "/app/codecollection/libraries/RW/Prometheus/Prometheus.py", line 127, in query_instant return self._query(api_url, target_service=target_service, optional_headers=optional_headers, params=params) File "/app/codecollection/libraries/RW/Prometheus/Prometheus.py", line 25, in _query rsp = self._query_with_service( File "/app/codecollection/libraries/RW/Prometheus/Prometheus.py", line 83, in _query_with_service optional_headers = self._secret_to_curl_headers(optional_headers=optional_headers) File "/app/codecollection/libraries/RW/Prometheus/Prometheus.py", line 54, in _secret_to_curl_headers headers.update(json.loads(optional_headers.value)) AttributeError: 'NoneType' object has no attribute 'value'


On further investigation we found that, 
- in [query_with_service(...)](https://github.com/runwhen-contrib/rw-public-codecollection/pull/139/files#diff-e4e34ff85734f3e772bdc4e72bb3a0c26d32dde1c214f5f320c2a49734fd83cbL74), `optional_headers` arg was not marked as optional, 
- And handling of `if optional_headers: ... else:..` was missing in a few key functions down the call-chain: [create_curl(...)](https://github.com/runwhen-contrib/rw-public-codecollection/pull/139/files#diff-e4e34ff85734f3e772bdc4e72bb3a0c26d32dde1c214f5f320c2a49734fd83cbL60)

This PR intends to fix these issues and makes Prometheus integration compatible with unauthenticated endpoints too (since prometheus endpoints are usually only exposed within the cluster, this case may be encountered in the wild.)

Post patch we have ran the same test-case and it has passed:

${rsp} = RW.Prometheus . Query Instant api_url=${ENV_PROMETHEUS_HOST} query=${ENV_QUERY} step=${STEP} target_service=${CURL_SERVICE}

Documentation:
Performs a query against the prometheus instant API for metrics with a single data point.

Start / End / Elapsed: 20240125 13:57:50.744 / 20240125 13:57:51.248 / 00:00:00.504

13:57:50.746 TRACE Arguments: [ api_url='http://af87d61af0d814b8d8e99d37987b291d-483479890.us-west-2.elb.amazonaws.com/prometheus/api/v1' | query='aws_rds_database_connections_average{dimension_DBInstanceIdentifier="robotshopmysql"} > 1' | step='30' | target_service=Service(url='https://curl.sandbox.runwhen.com') ]

13:57:50.748 DEBUG Starting new HTTPS connection (1): curl.sandbox.runwhen.com:443

13:57:51.245 DEBUG https://curl.sandbox.runwhen.com:443 "POST /api/v1/cmd HTTP/1.1" 200 1631

13:57:51.246 DEBUG execute_shell_command with shell service requrest: ShellServiceRequest(cmd='eval $(echo "curl -X GET \'http://af87d61af0d814b8d8e99d37987b291d-483479890.us-west-2.elb.amazonaws.com/prometheus/api/v1/query?query=aws_rds_database_connections_average%7Bdimension_DBInstanceIdentifier%3D%22robotshopmysql%22%7D%20%3E%201&time=2024-01-25T08%3A27%3A50.745982Z&step=30\'")', request_secrets=None, env=None, files=None, timeout_seconds=60) and received response ShellServiceResponse(cmd='eval $(echo "curl -X GET \'http://af87d61af0d814b8d8e99d37987b291d-483479890.us-west-2.elb.amazonaws.com/prometheus/api/v1/query?query=aws_rds_database_connections_average%7Bdimension_DBInstanceIdentifier%3D%22robotshopmysql%22%7D%20%3E%201&time=2024-01-25T08%3A27%3A50.745982Z&step=30\'")', parsed_cmd=['rbash', '-c', 'eval $(echo "curl -X GET \'http://af87d61af0d814b8d8e99d37987b291d-483479890.us-west-2.elb.amazonaws.com/prometheus/api/v1/query?query=aws_rds_database_connections_average%7Bdimension_DBInstanceIdentifier%3D%22robotshopmysql%22%7D%20%3E%201&time=2024-01-25T08%3A27%3A50.745982Z&step=30\'")'], stdout='{"status":"success","data":{"resultType":"vector","result":[{"metric":{"name":"aws_rds_database_connections_average","account_id":"590183940259","container":"yet-another-cloudwatch-exporter","dimension_DBInstanceIdentifier":"robotshopmysql","endpoint":"http","instance":"192.168.164.136:5000","job":"yace","name":"arn:aws:rds:us-west-2:590183940259:db:robotshopmysql","namespace":"monitoring","pod":"yace-8b8bd5598-j2xps","region":"us-west-2","service":"yace"},"value":[1706171270.745,"30"]}]}}', stderr='% Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\n 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\n100 498 100 498 0 0 4698 0 --:--:-- --:--:-- --:--:-- 4698\n100 498 100 498 0 0 4698 0 --:--:-- --:--:-- --:--:-- 4654', returncode=0, status=200, body='', errors=[])

13:57:51.248 TRACE Return: {'status': 'success', 'data': {'resultType': 'vector', 'result': [{'metric': {'name': 'aws_rds_database_connections_average', 'account_id': '590183940259', 'container': 'yet-another-cloudwatch-exporter', 'dimension_DBInstanceIdentifier': 'robotshopmysql', 'endpoint': 'http', 'instance': '192.168.164.136:5000', 'job': 'yace', 'name': 'arn:aws:rds:us-west-2:590183940259:db:robotshopmysql', 'namespace': 'monitoring', 'pod': 'yace-8b8bd5598-j2xps', 'region': 'us-west-2', 'service': 'yace'}, 'value': [1706171270.745, '30']}]}}

13:57:51.249 INFO ${rsp} = {'status': 'success', 'data': {'resultType': 'vector', 'result': [{'metric': {'name': 'aws_rds_database_connections_average', 'account_id': '590183940259', 'container': 'yet-another-cloudwatch-exp...