[prometheus] Default prometheus scrape config does not respect `prefixURL`

[godinja]

Describe the bug a clear and concise description of what the bug is.

The default prometheus scrape job is down due to an Nginx 404 Not Found response when server.prefixURL is set.

What's your helm version?

version.BuildInfo{Version:"v3.12.2", GitCommit:"1e210a2c8cc5117d1055bfaa5d40f51bbc2e345e", GitTreeState:"clean", GoVersion:"go1.20.6"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.`26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:32:21Z", GoVersion:"go1.20.1", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:08:36Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/arm64"}

Which chart?

prometheus

What's the chart version?

25.2.0

What happened?

I am serving Prometheus through a Traefik IngressRoute at /prometheus. This prevents Prometheus being able to scrape itself.

Prometheus metrics are located at /prometheus/metrics, which I am able to retrieve. The default scrape config in the Helm chart is:

serverFiles:
  prometheus.yml:
    scrape_configs:
      - job_name: prometheus
        static_configs:
          - targets:
            - localhost:9090
     # other scrape jobs omitted

If I overwrite serverFiles."prometheus\.yml".scrape_configs[0] and include metrics_path: /prometheus/metrics, Prometheus is able to scrape itself but the other default configuration is lost.

What you expected to happen?

Setting server.prefixURL would not interfere with Prometheus being able to scrape itself.

How to reproduce it?

No response

Enter the changed values of values.yaml?

global:
  scrape_interval: 10s

serverFiles:
  alerting_rules.yml:
{{ readFile "../prometheus/alerting-rules.yaml" | nindent 8 }}

configmapReload:
  reloadUrl: http://prometheus-server.monitoring.svc.cluster.local/prometheus/-/reload

extraScrapeConfigs: |
  - job_name: bull-exporter
    static_configs:
      - targets:
        - bull-exporter.monitoring.svc.cluster.local:9538
  - job_name: node
    static_configs:
      - targets:
        - prometheus-prometheus-node-exporter.monitoring.svc.cluster.local:9100

server:
  prefixURL: /prometheus
  baseURL: /prometheus

  extraArgs:
    log.level: debug

  alertmanagers:
    - static_configs:
      - targets:
        - prometheus-alertmanager.monitoring.svc.cluster.local:9093
      path_prefix: /alertmanager

Enter the command that you execute and failing/misfunctioning.

helm install prometheus prometheus-community/prometheus --version 25.2.0 --values values.yaml

Anything else we need to know?

This bug I've described has lots of discussion (https://github.com/helm/charts/issues/21841, https://github.com/helm/charts/issues/22511, https://github.com/prometheus-community/helm-charts/pull/806) with no apparent solution.

Traefik IngressRoute

Name:         prometheus-extra-prometheus-resources-prometheus-server-ingress-route
Namespace:    monitoring
Labels:       app.kubernetes.io/instance=prometheus
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=extra-prometheus-resources
              app.kubernetes.io/version=1.0.0
              helm.sh/chart=extra-prometheus-resources-1.0.0
Annotations:  meta.helm.sh/release-name: prometheus
              meta.helm.sh/release-namespace: monitoring
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Metadata:
  Creation Timestamp:  2023-10-27T15:20:46Z
  Generation:          1
  Managed Fields:
    API Version:  traefik.containo.us/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
          f:app.kubernetes.io/name:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
      f:spec:
        .:
        f:entryPoints:
        f:routes:
    Manager:         helm
    Operation:       Update
    Time:            2023-10-27T15:20:46Z
  Resource Version:  4949928
  UID:               982ed326-b028-4bbf-b32d-75f076b86863
Spec:
  Entry Points:
    web
  Routes:
    Kind:   Rule
    Match:  PathPrefix(`/prometheus`)
    Services:
      Name:       prometheus-server
      Namespace:  monitoring
      Port:       80
Events:           <none>

[mlindhout]

Same here