prometheus_output_monitor crashes

tetramin commented 5 years ago

I want to use the RecordAccessor helper in a prometheus_output_monitor plugin, but I get the error:

/var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin/filter_record_transformer.rb:230:in `expand': undefined method `gsub' for #<Fluent::PluginHelper::RecordAccessor::Accessor:0x007f25a86283b8> (NoMethodError)
        from /var/lib/gems/2.3.0/gems/fluent-plugin-prometheus-1.3.0/lib/fluent/plugin/in_prometheus_output_monitor.rb:45:in `block in configure'
        from /var/lib/gems/2.3.0/gems/fluent-plugin-prometheus-1.3.0/lib/fluent/plugin/in_prometheus_output_monitor.rb:44:in `each'
        from /var/lib/gems/2.3.0/gems/fluent-plugin-prometheus-1.3.0/lib/fluent/plugin/in_prometheus_output_monitor.rb:44:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin.rb:164:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/root_agent.rb:282:in `add_source'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/root_agent.rb:122:in `block in configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/root_agent.rb:118:in `each'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/root_agent.rb:118:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/engine.rb:131:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/engine.rb:96:in `run_configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/supervisor.rb:795:in `run_configure'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/supervisor.rb:579:in `dry_run'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/supervisor.rb:597:in `supervise'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/supervisor.rb:502:in `run_supervisor'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.5/bin/fluentd:8:in `<top (required)>'
        from /usr/local/bin/fluentd:22:in `load'
        from /usr/local/bin/fluentd:22:in `<main>'

Here is part of my fluentd config:

<source>
  @id prometheus_output_monitor
  @type prometheus_output_monitor
  <labels>
    pod ${hostname}
    node $.kubernetes.host # Error because of this line
  </labels>
</source>

Perhaps in the file lib/fluent/plugin/in_prometheus_output_monitor.rb you need to make such changes.

38     def configure(conf)
39       super
40       hostname = Socket.gethostname
41       expander = Fluent::Plugin::Prometheus.placeholder_expander(log)
42       placeholders = expander.prepare_placeholders({'hostname' => hostname, 'worker_id' => fluentd_worker_id})
43       @base_labels = Fluent::Plugin::Prometheus.parse_labels_elements(conf)
44       @base_labels.each do |key, value|
+++        if value.is_a?(String)
45           @base_labels[key] = expander.expand(value, placeholders)
+++        end
46       end

tetramin commented 5 years ago

Problem in the prometheus_monitor.

Part of my fluentd config:

<source>
  @id prometheus_monitor
  @type prometheus_monitor
  <labels>
    pod ${hostname}
    node $.kubernetes.host # This does not crash the plugin, but it does not work correctly. 
  </labels>
</source>

Metrics in prometheum look like this:

fluentd_status_buffer_queue_length{endpoint="24231",exported_pod="fluentd-bmlnv",instance="10.44.11.97:24231",job="fluentd",namespace="logging",node="#<Fluent::PluginHelper::RecordAccessor::Accessor:0x007fa5619199a8>",plugin_category="output",plugin_id="output_all_timeout",pod="fluentd-bmlnv",service="fluentd",type="elasticsearch"}

node="#\<Fluent::PluginHelper::RecordAccessor::Accessor:0x007fa5619199a8>"

Maybe I'm doing something wrong?

kazegusuri commented 5 years ago

Yes, it works correctly with the change. But the accessor syntax is available in prometheus output/filter plugins, not in prometheus monitor plugin. So the label for node key is not works as you intended even with the change.

kazegusuri commented 5 years ago

I added validation for this. thank you for reporting!

fluent / fluent-plugin-prometheus

prometheus_output_monitor crashes #80