Is your feature request related to a problem? Please describe.
We use a setup with multiple go-carbon servers where metrics are distributed across using carbon-relay-ng. Which means a metric is present on 1 out of all go-carbon instances. Ingest side is carbon-relay-ng -> (hash-ring) go-carbon, the query side is Grafana -> carbonapi -> (all) go-carbon.
The setup itself works fine, however we get a non-stop spam in the logs with 404 on all nodes. Which is caused by the setup itself. Changing empty-result-ok = true reduces the amount of entries, but also significantly increases query execution time.
Describe the solution you'd like
A simple switch to completely disable logs for 404 error, as those are part of the setup. The default may be to keep writing the logs (so the default behavior does not change).
Additional context
Having a large volume (1000s per minute) of unnecessary logs impedes any other troubleshooting efforts and adds strain on the disk flushing the logs. Also we have to have rather aggressive log rotation policy just to make sure it does not consume unnecessary disk space.
Is your feature request related to a problem? Please describe. We use a setup with multiple go-carbon servers where metrics are distributed across using carbon-relay-ng. Which means a metric is present on 1 out of all go-carbon instances. Ingest side is carbon-relay-ng -> (hash-ring) go-carbon, the query side is Grafana -> carbonapi -> (all) go-carbon.
The setup itself works fine, however we get a non-stop spam in the logs with 404 on all nodes. Which is caused by the setup itself. Changing
empty-result-ok = truereduces the amount of entries, but also significantly increases query execution time.Describe the solution you'd like A simple switch to completely disable logs for 404 error, as those are part of the setup. The default may be to keep writing the logs (so the default behavior does not change).
Additional context Having a large volume (1000s per minute) of unnecessary logs impedes any other troubleshooting efforts and adds strain on the disk flushing the logs. Also we have to have rather aggressive log rotation policy just to make sure it does not consume unnecessary disk space.