Issue trying to Test Query (Rule)

[fglaeser]

After hit Test Query button, I get a JS error in the chrome console and I never get a result. How the rule query is tested?

POST http://localhost:8000/rule/6/test 500 (Internal Server Error)
send @ jquery.min.js:4
ajax @ jquery.min.js:4
r.<computed> @ jquery.min.js:4
(anonymous) @ promgen.js:25
dispatch @ jquery.min.js:3
q.handle @ jquery.min.js:3

Thanks

[kfdm]

Internally, Promgen will try to query the upstream Prometheus servers to test the query. Can you confirm you have some shards configured with 'proxy' checked? /admin/promgen/shard/

[fglaeser]

[fglaeser]

Maybe this could help you.

promgen_1     | Internal Server Error: /rule/6/test
promgen_1     | Traceback (most recent call last):
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
promgen_1     |     response = get_response(request)
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
promgen_1     |     response = self.process_exception_by_middleware(e, request)
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
promgen_1     |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
promgen_1     |     return view_func(*args, **kwargs)
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 71, in view
promgen_1     |     return self.dispatch(request, *args, **kwargs)
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/mixins.py", line 52, in dispatch
promgen_1     |     return super().dispatch(request, *args, **kwargs)
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/django/views/generic/base.py", line 97, in dispatch
promgen_1     |     return handler(request, *args, **kwargs)
promgen_1     |   File "/usr/src/app/promgen/views.py", line 1303, in post
promgen_1     |     result = util.get(url, {'query': query}).json()
promgen_1     |   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 897, in json
promgen_1     |     return complexjson.loads(self.text, **kwargs)
promgen_1     |   File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
promgen_1     |     return _default_decoder.decode(s)
promgen_1     |   File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
promgen_1     |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
promgen_1     |   File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
promgen_1     |     raise JSONDecodeError("Expecting value", s, err.value) from None
promgen_1     | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[kfdm]

Can you use your browser's inspector, to see what's being returned in that ajax request ? It seems like the request isn't returning json properly.

[fglaeser]

How can I see this log in the console? logger.debug('Querying %s with %s', url, query)

I've set DEBUG: 1 in the docker-compose with no luck.

class RuleTest(LoginRequiredMixin, View):
    def post(self, request, pk):
        if pk == 0:
            rule = models.Rule()
            rule.set_object(request.POST['content_type'], request.POST['object_id'])
        else:
            rule = get_object_or_404(models.Rule, id=pk)

        query = macro.rulemacro(rule, request.POST['query'])
        # Since our rules affect all servers we use Promgen's proxy-query to test our rule
        # against all the servers at once
        url = resolve_domain('proxy-query')

        logger.debug('Querying %s with %s', url, query)
        start = time.time()
        result = util.get(url, {'query': query}).json()
        duration = datetime.timedelta(seconds=(time.time() - start))

Do you know what result = util.get(url, {'query': query}).json() tries to do? Because the date in the POST seems to be ok, but then a util.get is called just with the query.

btn.data('href'): "/rule/0/test"
btn.data(): Object
content_type: "service"
href: "/rule/0/test"
object_id: 4
query: "up > 1"
source: "#id_clause"
target: "#ajax-clause-check"

[kfdm]

The test command roughly works this way

1. Promgen's Javascript calls /rule/0/test
2. Promgen's /rule/0/test formats the query and then calls Promgen's /api/v1/query to proxy to Prometheus
3. /api/v1/query will take the query and send it to all configured Prometheus

I suspect the call from /api/v1/query -> Prometheus is not going correctly.

Can you confirm what version of Promgen you're using and how you installed it? (Docker, virtualenv, etc)

[fglaeser]

Hi, Promgen version is 0.48, running on Docker.

[kfdm]

Can you open up this URL, and see if it gives you an error ? This should help test just the proxy URL

/api/v1/query?query=count(up)

[kfdm]

One other thing to check, would be the configured domain name here /admin/sites/site/

[fglaeser]

Can you open up this URL, and see if it gives you an error ? This should help test just the proxy URL

/api/v1/query?query=count(up)

This is the return value:

{"status": "success", "data": {"resultType": "vector", "result": [{"metric": {}, "value": [1600432500.475, "10"]}]}}

The domain name is example.com.

[kfdm]

ok try updating the domain to match your promgen install and try the test rule again