How to have dynamic type and metric names from the record?

[misteryiz]

I am trying to generate metrics using fluend-prometheus plugin from the log files of the containers i.e, my containers are generaing metrics at regular inervals along with the application logs. I would like to use the filter prometheus plug in to generate the metricname, type and value of the metric from the logs (because I already have that info in the logs, don't want fluentd/prometheus instument seperately, rather than just populate the values of prometheus type from the container generated log metric values). Using this, https://github.com/fluent/fluent-plugin-prometheus/blob/master/README.md#prometheus-outputfilter-plugin.

It says,

Both output/filter plugins instrument metrics from records. Both plugins have no impact against values of each records, just read.

Does it mean, I won't be able to use the values of the keys that I would like to use? If so, is there any way to solve my problem of populating the values dynamically using the keys/values from the existing record?

I have the following configuration for my fluentd-prometheus plugin

<filter docker.var.log.containers.**>
  @type grep
  <regexp>
    key details
    pattern /metric/
  </regexp>
</filter>

<filter docker.var.log.container.**>
  @type prometheus
  <metric>
    name "$['details']['metrics'][0]['metric_name']"
    ###type ${if $['details']['metrics'][0].has_key?("log_type"); if $['details']['metrics'][0]["log_type"] == "BucketCounter"; histogram; elsif $['details']['metrics'][0]["log_type"] == "PERCENTILE"; summary; else $['details']['metrics'][0]['log_type']; end; else; "summary"; end }
###    type ${if $['details']['metrics'][0].has_key?("log_type");
###                if ($['details']['metrics'][0]["log_type"] == "BucketCounter");
###                     histogram;
###                elsif ($['details']['metrics'][0]["log_type"] == "PERCENTILE");
###                     summary;
###                else;
###                     $['details']['metrics'][0]['log_type'];
###                end;
###           else;
###                "summary";
###           end }
    type ${if $['details']['metrics'][0].has_key?("log_type"); if ($['details']['metrics'][0]["log_type"] == "BucketCounter"); histogram; elsif ($['details']['metrics'][0]["log_type"] == "PERCENTILE"); summary; else; $['details']['metrics'][0]['log_type']; end; else; "summary"; end }
    #type gauge
    desc ${if $['details']['metrics'][0].has_key?("description"); $['details']['metrics'][0]['description']; else; $['details']['metrics'][0]['metric_name']; end}
    key $['details']['metrics'][0]['value']
    <labels>
      tag ${tag}
    </labels>
  </metric>
</filter>

Sample Record

record = '{"stream":"stdout","msg":"New metric value","level":"INFO","ts":"2019-08-06T00:03:50.754248378+00:00","details":{"metrics":[{"metric_name":"http_request_sent_duration_ms_hist","log_type":"BucketCounter","rrdescription":"Duration of HTTP requests sent, separated into buckets","value":5.0,"tags":{"client":"","le":"500","operation":"GetConfig"}}]}}'

#record='{"stream":"stdout","msg":"New metric value","level":"info","ts":"2019-08-06T04:34:10.965038228+00:00","details":{"metrics":[{"metric_name":"http_responses_received_total","log_type":"Counter","description":"Number of HTTP responses received","value":17.0,"tags":{"client":"MCS","operation":"GetConfig","status_code":"200"}}]},}

I get the following messages when run the above config:

[root@test td-agent]# /usr/sbin/td-agent -c /etc/td-agent/td-agent.conf --dry-run
2019-08-06 22:43:03 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2019-08-06 22:43:03 +0000 [info]: starting fluentd-1.4.2 as dry run mode ruby="2.4.6"
2019-08-06 22:43:03 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="type option must be 'counter', 'gauge', 'summary' or 'histogram'"

When I set type to gauge (as a tmp value), I get

[root@test td-agent]# /usr/sbin/td-agent -c /etc/td-agent/td-agent.conf --dry-run
2019-08-06 22:59:59 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2019-08-06 22:59:59 +0000 [info]: starting fluentd-1.4.2 as dry run mode ruby="2.4.6"
2019-08-06 22:59:59 +0000 [error]: dry run failed: metric name must match /[a-zA-Z_:][a-zA-Z0-9_:]*/

[ganmacs]

metric name must match /[a-zA-Z:][a-zA-Z0-9:]*/

Please read the log. the name name "$['details']['metrics'][0]['metric_name']" is invalid. can you try to use other name?

[misteryiz]

I understand the log, But I would like to use the name that exists in the log already and I'm trying to assign the value of the key (which is $['details']['metrics'][0]['metric_name'] = 'http_responses_receivedtotal'. And, it matches with the "match /[a-zA-Z:][a-zA-Z0-9_:]*/". My question is can I use ruby-like expressions for accessing the values and then assigns them here?

[ganmacs]

My question is can I use ruby-like expressions for accessing the values and then assigns them here?

I think there is no way to do it now. Which metrics this plugin collects are determined at start time( and name is passed at the same time). it means this plugin must know specific metrics name at start time.

https://github.com/fluent/fluent-plugin-prometheus/blob/041f9f5f5c2f0b83e391728d2f355dbfbb47d33c/lib/fluent/plugin/prometheus.rb#L147

[misteryiz]

Yeah, that's where I stumbled. Would you really useful if (at all possible like record accessor) I can access the metric and it's value in from the logs using the filter plugin. I'm trying to avoid the unnecessary calculation of metrics where the metrics already available. @ganmacs Can you clarify one thing here, (https://github.com/fluent/fluent-plugin-prometheus#prometheus-outputfilter-plugin) - it says

Both output/filter plugins instrument metrics from records. Both plugins have no impact against values of each records, just read.

Does the plugin calculates the counter/guage/sumaary values from the provided key?