search

tagomoris / fluent-plugin-flowcounter

TODO: one-line summary of your gem
Other
53 stars 20 forks source link

Allow aggregating by an arbitrary field #22

Open jonmoter opened 6 years ago

jonmoter commented 6 years ago

I'm running fluentd as a daemonset in a Kubernetes cluster. For each application we deploy to the cluster, we add a project label, and then have some logic to apply that label to the log entries.

<filter kubernetes.pod.**.log>
  type kubernetes_metadata
</filter>

# Add metadata to each record
<filter kubernetes.pod.**.log>
  @type record_transformer
  enable_ruby
  <record>
    application #{record['kubernetes'].fetch('labels', {}).fetch('project', 'unknown')}
  </record>
</filter>

I'd like to generate counts of log entries generated in fluentd, aggregated by that application field. So I can know how many log messages each of the apps in my cluster are generating.

(From there, I'm using the dd plugin to send counts to Datadog.)

As far as I can tell, there's no way to aggregate counts with an arbitrary event field. But that would be a super-useful feature for us.

tagomoris commented 6 years ago

Patches are welcome :D It looks good idea to have another option :field in config_param :aggregate list, and a new parameter aggregate_field_name to specify the name of field.

SleepyBrett commented 5 years ago

Being able to count bytes per kubernetes namespace if a requirement I have. Unfortunate that this is so limited. Don't know enough ruby to work out a pull request.