search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.61k stars 1.55k forks source link

Integrate `codecs::Decoder` with sources #9404

Closed pablosichert closed 1 year ago

pablosichert commented 3 years ago

This is a tracking issue to integrate codecs::Decoder with relevant sources as consistently as possible.

Closely related to the work that has been done in #6958.

Source Issue PR Status Comment
aws_kinesis_firehose #9212 #9332 Merged
aws_s3 #9211 #9335 In review Deprecates multiline feature to reduce complexity of integration: https://github.com/vectordotdev/vector/pull/9335#issuecomment-932340534.
datadog #9429 #9430 Merged
docker_logs #9434 Missing specification Conflict in integrating codecs::Decoder due to multiline handling: https://github.com/vectordotdev/vector/issues/9434#issuecomment-933438704.
exec #9336 #9337 Merged
file #9210 Blocked Blocked by #9407.
fluent - - Not applicable Protocol encourages sending structured fields. Uses TcpSource and custom codec.
generator  #9401 #9402 Merged
heroku_logs #9431 #9432 Merged
http - #9157 Merged
internal_logs - - Not applicable Creates events based on the internal log format.
journald - - Not applicable Parses journalctl JSON output.
kafka - #9071 Merged
kubernetes_logs #9433 Blocked Blocked by #9407.
logstash - - Not applicable Protocol encourages sending structured fields. Uses TcpSource and custom codec.
socket - #9209 Merged
splunk_hec - - Not applicable Protocol encourages sending structured fields.
stdin - #9070 Merged
syslog - - Not applicable Uses codecs::Decoder, also see #4502.
vector - - Not applicable Parses events from proto.
nats #9340 #9400 Merged
jszwedko commented 3 years ago

We will deprecate multiline config on aws_s3 source in 0.17.0 in-lieu of reduce transform since it is difficult to integrate the multiline config with the decoder.

pablosichert commented 3 years ago

The docker_logs source implements quite complex logic to handle multiline events as well.

I'm not sure we can drop it there, since users were requesting it: https://github.com/vectordotdev/vector/issues/2137#issuecomment-646802454, https://github.com/vectordotdev/vector/issues/3307.

jszwedko commented 3 years ago

Thanks for creating this table! I agree, it seems like we need to leave the multi-line aggregation for the docker_logs source.

jszwedko commented 1 year ago

Closing this epic as complete.