Closed trennepohl closed 1 year ago
Thanks for opening this @trennepohl ! It seems like we need to:
Alternatively to preserve backwards compatibility we could remove the limits again and re-add them later. What version were you upgrading from where it worked?
Thanks for opening this @trennepohl !
No worries š
Set the max encoding size in the Vector sink to whatever the maximum configured batch size is
Yeah I tried to do that using max_decoding_message_size
on the proto_client
in Vector's sink, but didn't work š¤·
Not super familiar with Rust š
.
What version were you upgrading from where it worked?
0.29
Thanks for the additional context @trennepohl ! I think we can probably back out the limits for now and then re-add them later; tying them to the batch sizes.
@jszwedko Hi, I've updated vector to 0.32.0 and still get this error:
Some(Request { source: Status { code: OutOfRange, message: "Error, message length too large: found 4612000 bytes, the limit is: 4194304 bytes", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Fri, 18 Aug 2023 09:15:09 GMT", "content-length": "0"} }, source: None } })
My configuration:
[sinks.center_vector]
address = "${AGGREGATOR_ADDRESS}"
compression = true
healthcheck = false
inputs = ["log_files", "vector_logs"]
type = "vector"
[sinks.center_vector.batch]
max_events = 5000
timeout_secs = 1
[sinks.center_vector.buffer]
max_events = 6000
type = "memory"
when_full = "block"
Any thoughts about this?
Huh, interesting. I admittedly hadn't actually tested this š I just followed the recommendations. Let me try it with your config. I'll reopen this to track.
Hi @slgero ,
I'm having trouble reproducing this with 0.32.0 though I can with 0.31.0. Are you sure you are running v0.32.0 as the receiver (that is, the Vector instance with the vector
source)?
As I was setting up the reproduction I remembered I did actually test this change locally.
I was also able to repro this with 0.31.0 but not on 0.32.0. In the A/B cases, both sender and receiver were on the same version.
Hi, @jszwedko, As soon as I updated the version on the centre vector too, the error disappeared. I've only updated the version at the agent before. So you can close the issue. Thank you!
Great, thanks for confirming @slgero !
A note for the community
Problem
Vector sink fails to push messages bigger than 4MB.
Configuration
Version
0.31.0
Additional Context
After upgrading Vector to version 0.31.0 we started seeing this error.
After a bit of searching on the internet I bumped into this article. https://cprimozic.net/notes/posts/rust-tonic-request-response-size-limits/
Looks like from
tonic
0.9 default max size is 4mb.References