search

giraffi / fluent-plugin-amqp

Use AMQP broker to send or receive messages via FluentD
MIT License
15 stars 31 forks source link

Reduce chunk size #46

Open cosmo0920 opened 7 years ago

cosmo0920 commented 7 years ago

In v0.14, tag can be included in chunk's metadata. It makes to reduce chunk size.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.05%) to 91.649% when pulling 97340738c54e75e52ba76a0ac86e7cbaf790e98a on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.05%) to 91.649% when pulling 97340738c54e75e52ba76a0ac86e7cbaf790e98a on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage decreased (-0.1%) to 91.453% when pulling 147a47fdbc6807a724d937c05eee7dbc9dc9c6b1 on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.1%) to 91.72% when pulling 4096f2641b5e939dfaa1f7596c6429f8d31f84fa on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.1%) to 91.72% when pulling 4096f2641b5e939dfaa1f7596c6429f8d31f84fa on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.8%) to 92.368% when pulling 603a3732d8ea45a3c4ca18ec7f8943297463efef on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.1%) to 91.72% when pulling 5b66de287835ab2a619443f17df3ebbaefcb6d7a on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.1%) to 91.72% when pulling 5b66de287835ab2a619443f17df3ebbaefcb6d7a on cosmo0920:reduce-chunk-size into b1f9355c1c2e3dceb250668b8011cac314890c1f on giraffi:master.

cosmo0920 commented 7 years ago

Yes, this PR does not break compatibility for Fluentd v0.14.

Note that chunk compatibility will be broken with this change. If users use this plugin with this patch, he/she should ensure that there is no unprocessed buffer. This breaking changes is operational issue same as migrating to use v0.14 from v0.12 series of Fluentd.

sawanoboly commented 7 years ago

Hmm, I'll notify again @warmfusion

warmfusion commented 7 years ago

Do both end's of the plugin need to be modified so the tag data in the metadata is recognised; ie should in_amqp also be modified?

Can the README be updated to account for the new configuration option and its behaviour?

warmfusion commented 7 years ago

Should the chunk_keys configuration even be exposed to end-users through configuration? Why not just use the tag value as currently defined? I assume you'd want to be able to set multiple chunk_key values (otherise the plural name is misleading) - but I see no mechanism for such a thing to be defined?

Should the implementation silently use the tag value as currently defined, or is there something fundamental i'm missing here?

cosmo0920 commented 7 years ago

Do both end's of the plugin need to be modified so the tag data in the metadata is recognised; ie should in_amqp also be modified?

No we don't need to this. Input plugins do not have chunk -- i.e. this optimization is effectively working with buffered output plugins, which handle chunk.

cosmo0920 commented 5 years ago

I think that this PR is too hard to understand metadata mechanism. It is early to add this reducing feature in 3rd party plugin.

I leave this PR as-is for now.