message_foo_counter example not working? What am I missing?

[shinebayar-g]

Hi, this is my config.

# fluent.conf
@include prometheus.conf
@include my.conf

<filter **>
  @type record_modifier
  char_encoding utf-8
</filter>

<match **>
  @type stdout
</match>

# prometheus.conf
<source>
  @type prometheus
  bind 0.0.0.0
  port 24231
  metrics_path /metrics
</source>

<source>
  @type prometheus_output_monitor
</source>

<filter **>
  @type prometheus
  <metric>
    name message_foo_counter
    type counter
    desc The total number of foo in message.
    key foo
  </metric>
</filter>

# my.conf
<source>
  @type tail
  @id in_tail_my_logs
  path /var/log/app/*.log
  pos_file /var/log/app/fluentd-containers.log.pos
  tag my-log.test
  read_from_head true
  <parse>
    @type none
  </parse>
</source>

To test prometheus metrics I've created test.log file in /var/log/app/ folder with following content.

foo: 1

When I curl localhost:24231/metrics endpoint I get:

# TYPE message_foo_counter counter
# HELP message_foo_counter The total number of foo in message.
# TYPE fluentd_output_status_buffer_queue_length gauge
# HELP fluentd_output_status_buffer_queue_length Current buffer queue length.
# TYPE fluentd_output_status_buffer_total_bytes gauge
# HELP fluentd_output_status_buffer_total_bytes Current total size of queued buffers.
# TYPE fluentd_output_status_retry_count gauge
# HELP fluentd_output_status_retry_count Current retry counts.
fluentd_output_status_retry_count{plugin_id="object:3f963045689c",type="stdout"} 0.0
# TYPE fluentd_output_status_num_errors gauge
# HELP fluentd_output_status_num_errors Current number of errors.
fluentd_output_status_num_errors{plugin_id="object:3f963045689c",type="stdout"} 0.0
# TYPE fluentd_output_status_emit_count gauge
# HELP fluentd_output_status_emit_count Current emit counts.
fluentd_output_status_emit_count{plugin_id="object:3f963045689c",type="stdout"} 7.0
# TYPE fluentd_output_status_emit_records gauge
# HELP fluentd_output_status_emit_records Current emit records.
fluentd_output_status_emit_records{plugin_id="object:3f963045689c",type="stdout"} 7.0
# TYPE fluentd_output_status_write_count gauge
# HELP fluentd_output_status_write_count Current write counts.
fluentd_output_status_write_count{plugin_id="object:3f963045689c",type="stdout"} 0.0
# TYPE fluentd_output_status_rollback_count gauge
# HELP fluentd_output_status_rollback_count Current rollback counts.
fluentd_output_status_rollback_count{plugin_id="object:3f963045689c",type="stdout"} 0.0
# TYPE fluentd_output_status_retry_wait gauge
# HELP fluentd_output_status_retry_wait Current retry wait
fluentd_output_status_retry_wait{plugin_id="object:3f963045689c",type="stdout"} 0.0

As you see message_foo_counter metric is not working. What am I could be missing? :sos:

[repeatedly]

Because you use none parser. It creates {"message":"foo: 1"} so prometheus filter ignores it.

[shinebayar-g]

I see, now I use json parser and also noticed value must be number. Actually prometheus metrics shows exact value of the key. I thought counter would go up by 1 whenever foo appears in log again. I guess this working as intended.

My log looks like this which is why I use none parser at the moment.

INFO - 2019-07-04 11:27:52 --> Something here ERROR - 2019-07-04 11:26:43 --> Error message here

I wanted to count how many INFO or ERRORs I'm getting in logs using this plugin. Is it possible to achieve? If so how should I structure my log format? If not please let me know, thank you.

[ganmacs]

How about counting these value by prometheus side? see https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors

like

fluentd_***_***{foo=~"ERROR - "}