`[o11y]` Using `"*"` in MetricsEndpointProvider targets do not require exposing port via Service

Right now all CKF charms, which implements the metrics-endpoint, are using ServicePort and KubernetesServicePatch to expose metrics port. However, the MetricsEndpointProvider is using the unit IP address if "*" was provided.

For example:

self.metrics_endpoint_provider = MetricsEndpointProvider(
    self, jobs=[{"static_configs": [{"targets": [f"*:{METRICS_PORT}"]}]}]
)

will be translated to something like this:

endpoint": "http://10.1.23.223:8080/metrics",

at Grafana agent targets.

This is why I believe that the exposing metrics port via service is not needed, and it could be dropped. When using "*", we only need to think about setting any additional event, which can cause unit IP to change, via refresh_event parameter.

Another option would be to use that service instead of "*" to define the target. e.g. f"istiod.{self.model.name}.svc:{METRICS_PORT}" from istio-pilot.

Option one has another benefit, when we do not need to hide metrics behind rbac-proxy, since the port do not need to be exposed.

Charms using "*":

Charms using service name:

Charms using pod IP:

knative-operator

canonical / bundle-kubeflow

`[o11y]` Using `"*"` in MetricsEndpointProvider targets do not require exposing port via Service #1013