Label metrics with jsonrpc method

[xunto]

Hey, our team wants to label metrics with jsonrpc method.

I'm not even sure how to approach this problem. Can you suggest something? Do I need lua for this? Do I need to fork nginx-vts?

[u5surf]

@xunto Thanks! Unfortunately, it is hard to use jsonrpc method with this module itself. It's possibly to use combined lua-resty-jsonrpc to serve the jsonrpc request from client and to proxy to the anthor server which has the status endpoint returns the json format with vhost_traffic_status_display_format json; and then it might modify the jsonrpc response from json in the proxy. We're not sure whether it works or not. We could not answer completely such that individual cases.

[xunto]

@xunto Thanks! Unfortunately, it is hard to use jsonrpc method with this module itself. It's possibly to use combined lua-resty-jsonrpc to serve the jsonrpc request from client and to proxy to the anthor server which has the status endpoint returns the json format with vhost_traffic_status_display_format json; and then it might modify the jsonrpc response from json in the proxy. We're not sure whether it works or not. We could not answer completely such that individual cases.

Thank you for your response! But I feel like I wasn't clear enough. We already have a python application, that handles jsonrpc requests, which is behind the nginx with vts plugin installed.

What we want is to somehow parse jsonrpc requests in nginx, fetch jsonrpc method and add it as a label to vts metrics.

[u5surf]

@xunto I’ve been clear what you said. But I'm not sure that you said 'label'. Is that a [vhost_traffic_status_filer_by_set_key] (https://github.com/vozlt/nginx-module-vts#vhost_traffic_status_filter_by_set_key)?

e.g. you wanna get the status by country, it can obtain the data to set filer_by_set_key like this.

nginx.conf

server {
  server_name example.org;
  vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;

  ...
}

result

  ...
  "serverZones": {
  ...
  },
  "filterZones": {
      "country::example.org": {
          "KR": {
              "requestCounter":...,
              "inBytes":...,
              "outBytes":...,
              "responses":{
                  "1xx":...,
                  "2xx":...,
                  "3xx":...,
                  "4xx":...,
                  "5xx":...,
                  "miss":...,
                  "bypass":...,
                  "expired":...,
                  "stale":...,
                  "updating":...,
                  "revalidated":...,
                  "hit":...,
                  "scarce":...
              },
              "requestMsecCounter":...,
              "requestMsec":...,
              "requestMsecs":{
                  "times":[...],
                  "msecs":[...]
              },
          },
          "US": {
          ...
          },
          ...
      },
      ...
  },
  ...

If you use this variables, you cannot manipulate that because generally the fundamental of nginx configuration cannot be written dynamically on the fly, it needs to restart. Unless you can use lua and something key-value-store on memory, it probably doen't it.

[xunto]

But I'm not sure that you said 'label'. Is that a [vhost_traffic_status_filer_by_set_key] (https://github.com/vozlt/nginx-module-vts#vhost_traffic_status_filter_by_set_key)?

Yeah, they are called labels in the Prometheus terms. I forgot that the library supports more than just Prometheus. :)

If you use this variables, you cannot manipulate that because generally the fundamental of nginx configuration cannot be written dynamically on the fly, it needs to restart. Unless you can use lua and something key-value-store on memory, it probably doen't it.

Can you elaborate on how it would work? I'm open to using Lua, but I'm not sure how exactly to connect vts and lua.

I need to somehow set the variable in Lua, and then pass it to vhost_traffic_status_filter_by_set_key? How do I do that?

[SuperQ]

The VTS module has no visibility into your Python code. For this you need to use the Prometheus python library.

[u5surf]

@xunto

they are called labels in the Prometheus terms.

But I've confused the overview of your system and I might misunderstand several points. Which the system diagram does near for you? A.

| client | ---> | python-app | ---> | nginx with vts | 
                      +
                      + earn metrics
                      +
                | prometheus |

B.

| client | ---> | python-app | ---> | nginx with vts | 
                                              +
                                              + earn metrics
                                              +
                                       | prometheus |

C.

| client | ---> | nginx with vts | ---> | python-app | 
                      +
                      + earn metrics
                      +
                | prometheus |

If you choose the A, you should use the python library which @SuperQ said. Otherwise, it is prefer to use the nginx-vts-exporter. https://github.com/hnlq715/nginx-vts-exporter

[xunto]

@u5surf C is the closest one.

We also gather metrics from the Python app. We do it for both Python and Nginx, so we could see the difference if there is any (for example, in case nginx or interaction between Python and Nginx is laggy).

So kinda like this:

| client | ---> | nginx with vts | ---> | python-app | 
                      +                       +
                      + earn metrics          + earn metrics
                      +                       +
                | prometheus |          | prometheus |

We already added a custom label with JSON-RPC method in Python. What we want is to also have it in Nginx, so we could compare between Nginx and Python.

@SuperQ as you can see, we do it in Python too, but we also want to have it in Nginx. (Honestly, currently, I'm trying to understand if this is even possible).

[SuperQ]

We already added a custom label with JSON-RPC method in Python. What we want is to also have it in Nginx, so we could compare between Nginx and Python.

This is not supported.

[xunto]

@SuperQ By "not supported", do you mean "not easily possible"? Or that you are not interested in answering (you don't know/beyond the frame of the support/etc)?

[SuperQ]

It is not possible. As I said above, the vts module has no visibility into your code.

[xunto]

It is not possible. As I said above, the vts module has no visibility into your code.

@SuperQ I'm missing the point of how this is relevant (also this is kinda obvious, don't take me for an amateur, please).

I want a separate code inside Nginx that could handle that. And I'm looking for options for that.

[xunto]

The most straightforward way to do that is, of course, to fork vts, adding my custom logic inside it. But this is the way I won't go, so I'm asking if you know of an easier way to add such logic on the nginx level.

[u5surf]

@xunto

We also gather metrics from the Python app. We do it for both Python and Nginx, so we could see the difference if there is any (for example, in case nginx or interaction between Python and Nginx is laggy).

Why doesn’t it better each systems(thus nginx, python app) gather the metrics independently like you see the fig? If it do that, the prometheus python library and nginx vts exporter might help you. Moreover if you want to observe more precisely or one request level, it shoud be better to use the apprication performance monitoring with the destributed tracing tools on Datadog or Newrelic.

[xunto]

Why doesn’t it better each systems(thus nginx, python app) gather the metrics independently like you see the fig? If it do that, the prometheus python library and nginx vts exporter might help you.

I don't understand what you mean. They are doing it independently. It's just we want them to have similar labels on their data so that we could compare them more precisely.

For example, we already label metrics from Python with the JSON-RPC method of request. We want to parse requests in Nginx and label metrics with the JSON-RPC method, so that we could compare latency between Nginx and Python for a specific JSON-RPC method, not just latency in general.

[u5surf]

@xunto It might quick resolution which I came up with that fork it and change it you would around below. You can understand easily to use simply C's sprintf functions so that it could be rewrite some labels of already exist. (e.g. "nginx_vts_main_connections{status=\"accepted\"} %uA\n" to "foo_connections{status=\"accepted\"} %uA\n" ) value but it quite difficult to use some arbitrary value.

 buf = ngx_sprintf(buf, NGX_HTTP_VHOST_TRAFFIC_STATUS_PROMETHEUS_FMT_MAIN, &ngx_cycle->hostname, 
                   NGX_HTTP_VTS_MODULE_VERSION, NGINX_VERSION, 
                   (double) vtscf->start_msec / 1000, 
                   your_own_val1, your_own_val2, your_own_val3, your_own_val4, your_own_val5, your_own_val6, your_own_val7, 
                   shm_info->name, shm_info->max_size, 
                   shm_info->used_size, shm_info->used_node); 

https://github.com/vozlt/nginx-module-vts/blob/bd338835463811fe0221db858d865269952aaed9/src/ngx_http_vhost_traffic_status_display_prometheus.h#L11-L31

https://github.com/vozlt/nginx-module-vts/blob/bd338835463811fe0221db858d865269952aaed9/src/ngx_http_vhost_traffic_status_display_prometheus.c#L37-L42

Of course it cannot be supported us for the dependencies issue in case that you fork and modify your way.

BTW, how about you should use relabel_config or label_replace in prometheus feature below? I'm not sure those in detail...

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace

[xunto]

@u5surf Great, thank you for your insight! Very helpful!