search

prometheus / compliance

A set of tests to check compliance with various Prometheus interfaces
Apache License 2.0
125 stars 41 forks source link

promql-compliance-tester fails on victoriametrics #96

Open monotek opened 1 year ago

monotek commented 1 year ago

I've just tried to run the promql compatibility test with victoriametrics 1.83.0 but run into the following error.

Tried thanos and prometheus as reference config. Both fail.

Result is also non deterministic as the error alwasy fails at a different step. See:

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
403 / 548 [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->___________________________________________________________________] 73.54% 307 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
385 / 548 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->___________________________________________________________________________] 70.26% 363 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
445 / 548 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->_______________________________________________] 81.20% 267 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
431 / 548 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->______________________________________________________] 78.65% 348 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"
anish-katukam commented 1 month ago

Were you ever able to find a resolution to this? Attempting to run the tool and encountering the same issues.

monotek commented 1 month ago

Nope. But we use VictoriaMetrics now since i tried it and we could not find any incompatablities while doing so.