Add Tls secret to the reader sidecar container

[Sohamdg081992]

PR Description

Test Cluster: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/1a3fd8b1-7a92-4730-8e47-dec9e67f49a9/resourceGroups/testrecalertswcussoham/providers/Microsoft.ContainerService/managedClusters/TestRecAlertsWcusSoham/overview

Openssl command used to generate the certs with specific IP SAN: openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout client-key.pem -out client-cert.pem -subj "/C=US/ST=WA/L=Seattle/O=Microsoft/CN=PrometheusClient" -addext "subjectAltName = IP:10.224.0.4"

This fix adds Tls secret to the reader sidecar container.

Testing with configmap:

1. Secret create: kubectl create secret generic ama-metrics-mtls-secret --from-file=client-cert.pem=client-cert.pem --from-file=client-key.pem=client-key.pem -n kube-system

2. Configmap used: https://github.com/Azure/prometheus-collector/blob/main/internal/referenceapp/linux-https-scrape-config.yaml

Delete secret and then create the secret to an invalid/corrupted cert -> pods restart -> metric flow stops due to invalid auth. I then deleted and created the secret again to correct cert using command: kubectl create secret generic ama-metrics-mtls-secret --from-file=client-cert.pem=client-cert.pem --from-file=client-key.pem=client-key.pem. Metric flow continues after pods restart.

Testing with CRD:

1. Then I deleted secret and created a secret for CRD with command: kubectl create secret generic ama-metrics-mtls-secret --from-file=secret_kube-system_ama-metrics-mtls-secret_client-cert.pem=secret_kube-system_ama-metrics-mtls-secret_client-cert.pem --from-file=secret_kube-system_ama-metrics-mtls-secret_client-key.pem=secret_kube-system_ama-metrics-mtls-secret_client-key.pem -n kube-system
2. I then deleted the configmap and created a podmonitor. File used: https://github.com/Azure/prometheus-collector/blob/main/otelcollector/deploy/example-custom-resources/pod-monitor/pod-monitor-reference-app-mtls.yaml

Metrics flow.

New Feature Checklist

• [ ] List telemetry added about the feature.
• [ ] Link to the one-pager about the feature.
• [ ] List any tasks necessary for release (3P docs, AKS RP chart changes, etc.) after merging the PR.
• [ ] Attach results of scale and perf testing.

Tests Checklist

• [ ] Have end-to-end Ginkgo tests been run on your cluster and passed? To bootstrap your cluster to run the tests, follow these instructions.
  • Labels used when running the tests on your cluster:
  • [ ] operator
  • [ ] windows
  • [ ] arm64
  • [ ] arc-extension
  • [ ] fips
• [ ] Have new tests been added? For features, have tests been added for this feature? For fixes, is there a test that could have caught this issue and could validate that the fix works?
  • [ ] Is a new scrape job needed?
  • [ ] The scrape job was added to the folder test-cluster-yamls in the correct configmap or as a CR.
  • [ ] Was a new test label added?
  • [ ] A string constant for the label was added to constants.go.
  • [ ] The label and description was added to the test README.
  • [ ] The label was added to this PR checklist.
  • [ ] The label was added as needed to testkube-test-crs.yaml.
  • [ ] Are additional API server permissions needed for the new tests?
  • [ ] These permissions have been added to api-server-permissions.yaml.
  • [ ] Was a new test suite (a new folder under /tests) added?
  • [ ] The new test suite is included in testkube-test-crs.yaml.

[bragi92]

Looks like a trivy failure @Sohamdg081992 if you merge from main I believe it should pass.

[Sohamdg081992]

Looks like a trivy failure @Sohamdg081992 if you merge from main I believe it should pass.

Thanks Kaveesh! Just merged from main.