Ingress to VM Single server is not working

[hrbapna]

I am using victoria-metrics-k8s-stack and configuring ingress to divert requests at /vicm to the VM's main page. Unable to reach to VMSingle using http://localhost/vicm In the same setup, I also have prometheus running (for some comparison purpose). The same ingress manifest works on it. http://localhost/prometheus works. Can someone provide me pointers on what am I doing wrong?

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app.kubernetes.io/component: victoria-metrics-k8s-stack-vmsingle
    app.kubernetes.io/instance: victoriametrics
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-k8s-stack
    app.kubernetes.io/version: 1.81.1
    argocd.argoproj.io/instance: victoriametrics
    helm.sh/chart: victoria-metrics-k8s-stack-0.12.1
  name: vmsingle-vm
  namespace: monitoring
spec:
  rules:
    - host: localhost
      http:
        paths:
          - backend:
              service:
                name: vmsingle-vm
                port:
                  number: 8429
            path: /vicm
            pathType: Prefix

The below config works.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: prometheus-prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: prometheus
    app.kubernetes.io/version: 34.0.0
    argocd.argoproj.io/instance: prometheus
    chart: kube-prometheus-stack-34.0.0
    heritage: Helm
    release: prometheus
  name: prometheus-prometheus
  namespace: monitoring
spec:
  rules:
    - host: localhost
      http:
        paths:
          - backend:
              service:
                name: prometheus-prometheus
                port:
                  number: 9090
            path: /prometheus
            pathType: ImplementationSpecific

[hrbapna]

Is this fixed by https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858 ? In my setup, I am not even able to reach to the main page. I can understand that the above fix will make the relative paths for the child pages work.

[hrbapna]

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858 is the solution to this issue.

[hrbapna]

But this results into two new issues:

1. Continuous error logs in VM single container. Requests are originating from grafana VM dashboard. Resulting in these errors.

   2022-09-08T18:56:05.930Z    warn    VictoriaMetrics/lib/httpserver/httpserver.go:253    remoteAddr: "10.244.0.18:39644"; requestURI: /api/v1/query?query=max+by+%28cluster%2C+namespace%2C+workload%2C+pod%29+%28%0A++label_replace%28%0A++++kube_pod_owner%7Bjob%3D%22kube-state-metrics%22%2C+owner_kind%3D%22StatefulSet%22%7D%2C%0A++++%22workload%22%2C+%22%241%22%2C+%22owner_name%22%2C+%22%28.%2A%29%22%0A++%29%0A%29&step=15s&time=1662663360; missing -http.pathPrefix="/vicm" in the requested path "/api/v1/query"
   2022-09-08T18:56:05.930Z    warn    VictoriaMetrics/lib/httpserver/httpserver.go:253    remoteAddr: "10.244.0.18:39644"; requestURI: /api/v1/query?query=max+by+%28cluster%2C+namespace%2C+workload%2C+pod%29+%28%0A++label_replace%28%0A++++kube_pod_owner%7Bjob%3D%22kube-state-metrics%22%2C+owner_kind%3D%22Job%22%7D%2C%0A++++%22workload%22%2C+%22%241%22%2C+%22owner_name%22%2C+%22%28.%2A%29%22%0A++%29%0A%29&step=15s&time=1662663360; missing -http.pathPrefix="/vicm" in the requested path "/api/v1/query"
   2022-09-08T18:56:05.976Z    warn    VictoriaMetrics/lib/httpserver/httpserver.go:253    remoteAddr: "10.244.0.18:39644"; requestURI: /api/v1/query?query=count+by+%28cluster%29+%28count+by+%28git_version%2C+cluster%29+%28label_replace%28kubernetes_build_info%7Bjob%21~%22kube-dns%7Ccoredns%22%7D%2C%22git_version%22%2C%22%241%22%2C%22git_version%22%2C%22%28v%5B0-9%5D%2A.%5B0-9%5D%2A%29.%2A%22%29%29%29+%3E+1&step=15s&time=1662663360; missing -http.pathPrefix="/vicm" in the requested path "/api/v1/query"
   2022-09-08T18:56:05.977Z    warn    VictoriaMetrics/lib/httpserver/httpserver.go:253    remoteAddr: "10.244.0.18:39644"; requestURI: /api/v1/query?query=%28sum%28rate%28rest_client_requests_total%7Bcode%3D~%225..%22%7D%5B5m%5D%29%29+by+%28cluster%2C+instance%2C+job%2C+namespace%29%0A++%2F%0Asum%28rate%28rest_client_requests_total%5B5m%5D%29%29+by+%28cluster%2C+instance%2C+job%2C+namespace%29%29%0A%3E+0.01&step=15s&time=1662663360; missing -http.pathPrefix="/vicm" in the requested path "/api/v1/query"

2. This also messes up the Grafana dashboard. Datasource URL http://vmsingle-vm.monitoring.svc:8429/ doesn't work tried with few combinations such as http://vmsingle-vm.monitoring.svc:8429/vicm but getting this error on testing the datasource connection Error reading Prometheus: bad_response: readObjectStart: expect { or n, but found r, error found in #1 byte of ...|remoteAddr:|..., bigger context ...|remoteAddr: "10.244.0.10:46886"; requestURI: /api/v|... Any pointers on how to fix this?

[hrbapna]

missing -http.pathPrefix="/vicm".

Add helm config to add http.pathPrefix vmsingle: spec: extraArgs: http.pathPrefix: "/vicm"

[chasain]

https://github.com/VictoriaMetrics/helm-charts/pull/383