vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.58k stars 1.54k forks source link

Drop "connection closed before message completed" messages to `debug` #15196

Open jszwedko opened 1 year ago

jszwedko commented 1 year ago

A note for the community

Use Cases

A few users have reported that the HTTP warning messages "connection closed before message completed" are noisy since having connections terminate abruptly is not unusual and the requests are retried.

Some users also think these are hard errors that are resulting in dropped events because of the log level.

Attempted Solutions

No response

Proposal

Drop these log messages to the DEBUG level.

References

Version

vector 0.25.1

lordsherman commented 11 months ago

These messages were fairly concerning to me in my setup until I stumbled upon this posted issue. While I never saw lost data due to this it was making me think something was wrong.

I send out data to central Loki and Victoriametrics instances. Some of my regional vector instances that push this data out will see these errors way more than others. It depends on how many servers locally push to these regional instances. I am going to have to set log limits for my containers in the locations that have a lot of servers pushing to them or my instances will run out of storage space because of this.

I would love to see this change.