Can this plugin be used to substitute the instance ID in the S3 matcher?

[et304383]

Basically I want to do something like this:

<match httpd_access_log>
  @type s3

  s3_bucket my-bucket
  path {instance_id}/httpd_access_log/
  acl bucket-owner-full-control

  <buffer time>
    @type memory
    timekey 60s
    timekey_wait 10s
  </buffer>

  time_slice_format %Y%m%d%H%M
</match>

The above isn't substituting {instance_id}.

I don't know much about fluentd and filters/matchers, but do I have to pass through your logic first then send to the S3 matcher?

[cosmo0920]

Your configuration aims to use fluent-plugin-s3 not fluent-plugin-ec2-metadata. fluent-plugin-s3 does not have {instance_id} placeholder and fluent-plugin-ec2-metadata cannot substitute {instance_id} placeholder its outside.

[et304383]

I'm confused - what is the point of your plugin then? Can it not be used in a pipeline?

[cosmo0920]

<match foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds

  output_tag ${instance_id}.${tag}
  <record>
    # ...
  </record>
</match>

can substitute ${instance_id} because this match section aims to use ec2_metadata plugin. But your configuration is not for ec2_metadata configuration.

Fluentd plugin cannot recognize other plugin settings in the pipeline.

[et304383]

It seems the best I could do is pass through your plugin and add an instance ID field to the record, correct?

[cosmo0920]

It seems the best I could do is pass through your plugin and add an instance ID field to the record, correct?

It depends on your Fluentd configuration. I have no enough your configuration information for advising. Thanks.

[bmihaescu]

@et304383 did you manage to pass the instanceID to S3 eventually ? I know it was a long time ago :)