Many customers have encountered a problem (feature?) where it seems that it is not possible to perform any filtering, mutation, addition or removal of fields in the input stream when using the plain or lines codecs. It seems that the only codec which allows for such filtering is the json_lines codec.
The fact that filtering is restricted for any reason is counter-intutive and completely unexpected and quite unusual for logstash plugins. Additionally, the problem (feature?) is wholly undocumented. I am filing this documentation request after spending at least 4 hours tracking this down for a customer.
Is this intended behavior? If so, it would also be helpful if the intent of this behavior could be explained.
btw: I am only able to identify the behavior for the plain, lines and json_lines codecs; if there are other codecs, please document the behavior for those codecs as well.
Many customers have encountered a problem (feature?) where it seems that it is not possible to perform any filtering, mutation, addition or removal of fields in the input stream when using the plain or lines codecs. It seems that the only codec which allows for such filtering is the json_lines codec.
See https://discuss.elastic.co/t/question-about-s3-output-plugin/124321 and https://discuss.elastic.co/t/s3-output-plugin-codec-issue/25237 for public discussions of this problem.
The fact that filtering is restricted for any reason is counter-intutive and completely unexpected and quite unusual for logstash plugins. Additionally, the problem (feature?) is wholly undocumented. I am filing this documentation request after spending at least 4 hours tracking this down for a customer.
Is this intended behavior? If so, it would also be helpful if the intent of this behavior could be explained.
btw: I am only able to identify the behavior for the plain, lines and json_lines codecs; if there are other codecs, please document the behavior for those codecs as well.