takus / fluent-plugin-ec2-metadata

Fluentd output plugin to add Amazon EC2 metadata into messages
Other
44 stars 28 forks source link

Can this plugin be used to substitute the instance ID in the S3 matcher? #40

Closed et304383 closed 6 years ago

et304383 commented 6 years ago

Basically I want to do something like this:

<match httpd_access_log>
  @type s3

  s3_bucket my-bucket
  path {instance_id}/httpd_access_log/
  acl bucket-owner-full-control

  <buffer time>
    @type memory
    timekey 60s
    timekey_wait 10s
  </buffer>

  time_slice_format %Y%m%d%H%M
</match>

The above isn't substituting {instance_id}.

I don't know much about fluentd and filters/matchers, but do I have to pass through your logic first then send to the S3 matcher?

cosmo0920 commented 6 years ago

Your configuration aims to use fluent-plugin-s3 not fluent-plugin-ec2-metadata. fluent-plugin-s3 does not have {instance_id} placeholder and fluent-plugin-ec2-metadata cannot substitute {instance_id} placeholder its outside.

et304383 commented 6 years ago

I'm confused - what is the point of your plugin then? Can it not be used in a pipeline?

cosmo0920 commented 6 years ago
<match foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds

  output_tag ${instance_id}.${tag}
  <record>
    # ...
  </record>
</match>

can substitute ${instance_id} because this match section aims to use ec2_metadata plugin. But your configuration is not for ec2_metadata configuration.

Fluentd plugin cannot recognize other plugin settings in the pipeline.

et304383 commented 6 years ago

It seems the best I could do is pass through your plugin and add an instance ID field to the record, correct?

cosmo0920 commented 6 years ago

It seems the best I could do is pass through your plugin and add an instance ID field to the record, correct?

It depends on your Fluentd configuration. I have no enough your configuration information for advising. Thanks.

bmihaescu commented 4 years ago

@et304383 did you manage to pass the instanceID to S3 eventually ? I know it was a long time ago :)