vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.36k stars 1.51k forks source link

Integrate `codecs::Encoder` with sinks #9459

Closed binarylogic closed 1 year ago

binarylogic commented 2 years ago

Like #9404 but for encoding/sinks

Issues that will be resolved with the integration:

Native encoding issues:

Other issues:

Priority of sinks (check means "has open PR" or "marked as not applicable"):

Sink Issue PR Status Comment Legacy encoding
aws_cloudwatch_logs - #12175 Merged - EncodingConfig<StandardEncodings>
aws_kinesis_firehose - #12176 Merged - EncodingConfig<StandardEncodings>
aws_kinesis_streams - #12177 Merged - EncodingConfig<StandardEncodings>
aws_s3 - #12136 Merged - EncodingConfig<StandardEncodings>
aws_sqs - #12550 Merged - EncodingConfig<(Text,Json)>
azure_blob - #12179 Merged - EncodingConfig<StandardEncodings>
azure_monitor_logs - Blocked Needs improvements to Encoder to emit JSON value when serializer supports JSON. (Part of #12495.) EncodingConfigWithDefault<(Default)>
blackhole - - Not applicable - -
clickhouse - - Not applicable Has opinionated fixed encoding EncodingConfigWithDefault<(Default)>
console - #12181 In review EncodingConfig<StandardEncodings>
datadog_archives - - Not applicable Has opinionated fixed encoding -
datadog_events - - Not applicable Has opinionated fixed encoding -
datadog_logs - - Not applicable Has opinionated fixed encoding -
datadog_traces - - Not applicable Has opinionated fixed encoding -
elasticsearch - Needs clarification The Elasticsearch Bulk API expects entries in JSON format. EncodingConfigFixed<{doc_type,suppress_type_name}>
file - #12548 Merged - EncodingConfig<(Text,Ndjson)>
gcp_cloud_storage - #12488 Merged - EncodingConfig<StandardEncodings>
gcp_pubsub - #12718 In review EncodingConfigWithDefault<(Default)>
gcp_stackdriver_logs - #12719 Blocked Needs improvements to Encoder to emit JSON value when serializer supports JSON. (Part of #12495.) EncodingConfigWithDefault<(Default)>
honeycomb - #12764 Blocked Needs improvements to Encoder to emit JSON value when serializer supports JSON. (Part of #12495.) -
http #11579 #11647 Merged - EncodingConfig<(Text,Ndjson,Json)>
humio_logs - #12495 In review EncodingConfig<(Json,Text)>
influxdb - #12583 Merged - EncodingConfigWithDefault<(Default)>
kafka - #12133 Merged - EncodingConfig<StandardEncodings>
logdna - EncodingConfigWithDefault<(Default)>
loki - #12558 Merged - EncodingConfig<(Json,Text,Logfmt)>
nats - #12586 Merged - EncodingConfig<(Text,Json)>
new_relic - EncodingConfigFixed<(Default)>
new_relic_logs - EncodingConfigWithDefault<(Json)>
papertrail - #12589 Merged - EncodingConfig<(Text,Json)>
pulsar - #13139 In review EncodingConfig<(Text,Json,Avro)>
redis - #12596 Merged - EncodingConfig<(Text,Json)>
sematext - EncodingConfigFixed<{doc_type,suppress_type_name}>
socket - #10684 Merged - EncodingConfig<(Text,Json)>
splunk_hec_logs - #12495 In review EncodingConfig<(Json,Text)>
vector - - Not applicable - -
websocket - #13054 In review - EncodingConfig<StandardEncodings>
jszwedko commented 1 year ago

Closing this epic since it was complete. Remaining work is tracked separately.