[kube-prometheus-stack] support --web.enable-remote-write-receiver

[daveoy]

Describe the bug a clear and concise description of what the bug is.

i cannot find a way to pass the command line arg --web.enable-remote-write-receiver=true to prometheus via the kube-prometheus-stack helm chart. i have added it to my prometheus.prometheusSpec.enableRemoteWriteReceiver value but it seems to not be respected, despite being listed in the prometheus-operator prometheus API: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#prometheusspec

What's your helm version?

3 via fluxcd

What's your kubectl version?

1.23.3

Which chart?

kube-prometheus-stack

What's the chart version?

34.6.0

What happened?

i cannot set enableRemoteWriteReceiver and have the --web.enable-remote-write-receiver=true flag set on my prometheus

What you expected to happen?

i set enableRemoteWriteReceiver and have the --web.enable-remote-write-receiver=true flag set on my prometheus

How to reproduce it?

set enableRemoteWriteReceiver

Enter the changed values of values.yaml?

prometheusSpec: enableRemoteWriteReceiver: true

Enter the command that you execute and failing/misfunctioning.

chart applies without modifying the Prometheus resource with the given value

Anything else we need to know?

i also cant edit the prometheus resource to assign this value, even though its listed as supported in that field here: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#prometheusspec

[daveoy]

i can confirm that deleting the prometheus operator deployment after a successful deployment of kube-prometheus-stack and editing the statefulset for prometheus to add the flag will enable the receiver and metrics flow in just fine.

is there another way to get a command line flag added to the statefulset without modifying first the prometheus crd to accept the field that isnt being respected?

[daveoy]

i can confirm that using the code referenced in pr #1963 we the correct flag isn't being set as per the prometheus-operator docs which claim it to be. this PR is still relevant and will set us up for the ability to have the prometheus CR set up properly for the operator to pick up. i will be opening an issue with that team too.

[daveoy]

looks like this is known in the prometheus operator community https://github.com/prometheus-operator/prometheus-operator/pull/4633

even though the docs say the feature flag isnt supported in 2.34, it does work with this fix: https://github.com/prometheus-operator/prometheus-operator/issues/4613#issuecomment-1060659255 so that's an acceptable workaround for me (setting prometheus.prometheusSpec.enableFeatures - remote-write-receiver).

must not have made it to release yet in prom-operator.

we should still merge #1963 in preparation for that

[GirPhi]

Hi daveoy, I am pretty much in the same boat as you trying to get the command prompt to take with either of the feature flag or the new --web.enable-remote-write-receiver. When you say this fix works for you, can you please tell me exactly what and where do you set this prometheus.prometheus.Spec.enableFeatures? I am trying in helm with "--set prometheus.enable-feature=remote-write-receiver" with no luck. Thank you for any hints.

[daveoy]

yes sure @GirPhi, in your values:

     prometheus:
       prometheusSpec:
         enableRemoteWriteReceiver: true
         enableFeatures:
         - remote-write-receiver

taken from here

[daveoy]

I am trying in helm with "--set prometheus.enable-feature=remote-write-receiver" with no luck.

If you're running helm on the cli, your --set arg is close, I think you want something like "--set prometheus.prometheusSpec.enable-feature=['remote-write-receiver']". Note the key/path in your comment is wrong and that key expects a list value.

[GirPhi]

Thanks for the reply... but still no go with either of your values: "remote write receiver needs to be enabled with --enable-feature=remote-write-receiver\n"

With yours, if you go in the prometheus web query page, under status, command-line flags, do you have anything showing for "-enable feature" ? mine is blank.

[daveoy]

yes i do.

you may want to check your Prometheus object to see that the values have provided have properly made their way to your prometheus object. if they're present in your prometheus object and match the api described here then i would check that your configuration has been reloaded. if it has and prometheus doesn't show any errors in the logs, perhaps your version of prometheus doesn't support this feature? are you using current versions of the operator, prometheus and this chart?

[daveoy]

This was resolved by merging #1963

[omerfsen]

Is this remote-write-receiver DEPRECATED ?

 --enable-feature= ...      Comma separated feature names
                                 to enable. Valid options: agent,
                                 exemplar-storage, expand-external-labels,
                                 memory-snapshot-on-shutdown,
                                 promql-at-modifier, promql-negative-offset,
                                 promql-per-step-stats, remote-write-receiver
                                 (DEPRECATED), extra-scrape-metrics,
                                 new-service-discovery-manager, auto-gomaxprocs,
                                 no-default-scrape-port, native-histograms. See
                                 https://prometheus.io/docs/prometheus/latest/feature_flags/
                                 for more details.

prometheus --version   
prometheus, version 2.42.0 (branch: non-git, revision: non-git)
  build user:       brew@Ventura-arm64.local
  build date:       20230131-17:04:46
  go version:       go1.19.5
  platform:         darwin/arm64

[gjkamstra]

Is this remote-write-receiver DEPRECATED ?

 --enable-feature= ...      Comma separated feature names
                                 to enable. Valid options: agent,
                                 exemplar-storage, expand-external-labels,
                                 memory-snapshot-on-shutdown,
                                 promql-at-modifier, promql-negative-offset,
                                 promql-per-step-stats, remote-write-receiver
                                 (DEPRECATED), extra-scrape-metrics,
                                 new-service-discovery-manager, auto-gomaxprocs,
                                 no-default-scrape-port, native-histograms. See
                                 https://prometheus.io/docs/prometheus/latest/feature_flags/
                                 for more details.

prometheus --version   
prometheus, version 2.42.0 (branch: non-git, revision: non-git)
  build user:       brew@Ventura-arm64.local
  build date:       20230131-17:04:46
  go version:       go1.19.5
  platform:         darwin/arm64

The feature flag is deprecated. It can now be enabled differently. See https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver

[thisurawijesekera]

Is this remote-write-receiver DEPRECATED ?

 --enable-feature= ...      Comma separated feature names
                                 to enable. Valid options: agent,
                                 exemplar-storage, expand-external-labels,
                                 memory-snapshot-on-shutdown,
                                 promql-at-modifier, promql-negative-offset,
                                 promql-per-step-stats, remote-write-receiver
                                 (DEPRECATED), extra-scrape-metrics,
                                 new-service-discovery-manager, auto-gomaxprocs,
                                 no-default-scrape-port, native-histograms. See
                                 https://prometheus.io/docs/prometheus/latest/feature_flags/
                                 for more details.

prometheus --version   
prometheus, version 2.42.0 (branch: non-git, revision: non-git)
  build user:       brew@Ventura-arm64.local
  build date:       20230131-17:04:46
  go version:       go1.19.5
  platform:         darwin/arm64

The feature flag is deprecated. It can now be enabled differently. See https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver

Did you able to enable the remote-write-receiver

[daveoy]

Is this remote-write-receiver DEPRECATED ?

 --enable-feature= ...      Comma separated feature names
                                 to enable. Valid options: agent,
                                 exemplar-storage, expand-external-labels,
                                 memory-snapshot-on-shutdown,
                                 promql-at-modifier, promql-negative-offset,
                                 promql-per-step-stats, remote-write-receiver
                                 (DEPRECATED), extra-scrape-metrics,
                                 new-service-discovery-manager, auto-gomaxprocs,
                                 no-default-scrape-port, native-histograms. See
                                 https://prometheus.io/docs/prometheus/latest/feature_flags/
                                 for more details.

prometheus --version   
prometheus, version 2.42.0 (branch: non-git, revision: non-git)
  build user:       brew@Ventura-arm64.local
  build date:       20230131-17:04:46
  go version:       go1.19.5
  platform:         darwin/arm64

The feature flag is deprecated. It can now be enabled differently. See https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver

Did you able to enable the remote-write-receiver

this issue is closed. open a new one if you're having trouble. this feature flag is deprecated as the option is now no longer flag-gated.

[vibhavachar]

Anyone got it working for real?

https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml tried all possible configuration nothing worked.

enableFeatures:

# - remote-write-receiver
# - web.enable-remote-write-receiver
# - remote-write-receiver
# - --web.enable-remote-write-receiver  does not work
# - --enable-feature=remote-write-receiver does not work

enableRemoteWriteReceiver: true enableFeatures:

• remote-write-receiver

[prores-raypet]

Also having issues, but the weird thing is that I see the flag in the UI. But getting 404s when trying to interact with the api.

[zozidalom]

I also tried the mentioned values, but I am facing the same issue.

[zozidalom]

@vibhavachar @prores-raypet did any of you find a solution?

[prores-raypet]

Yes, we had a routeprefix of /prometheus (we proxy to /prometheus). Once I appended that, it worked just fine.

  prometheusSpec:
    enableRemoteWriteReceiver: true
    routePrefix: /prometheus

Then ship to /prometheus/ without 404s.

[zozidalom]

I tried with the routePrefix, but it still didn't work for me. (I don't have one btw)

curl -X POST http://localhost:22767/api/v1/write % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 82 100 82 0 0 569 0 --:--:-- --:--:-- --:--:-- 573remote write receiver needs to be enabled with --web.enable-remote-write-receiver

I get this error when I try to send a http post to it, though I have

    values:
      - prometheus:
          prometheusSpec:
            enableRemoteWriteReceiver: true

set in as my helm values.

And when I got to my "Configuration" page and Ctrl + F search for "remote" I get 0 results, nothing relased to remote write.

I use this chart, and chart version:

kube-prometheus-stack
version: ~62.1.0