zalando / nakadi

A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
https://nakadi.io
MIT License
958 stars 292 forks source link

Reliably track request and response content length #1437

Closed a1exsh closed 2 years ago

a1exsh commented 2 years ago

Some user agents set the Content-Length header, but not all. Use request and response wrapper class together with counting input/output stream wrappers.

antban commented 2 years ago

the PR is transforming from counting actual values (compressed bytes) to non-compressed, is it the intention?

a1exsh commented 2 years ago

the PR is transforming from counting actual values (compressed bytes) to non-compressed, is it the intention?

It is not intentional, but I thought it might be useful to measure both compressed and original size of both request and response. Btw, it doesn't look like we support zstd compression for response — only for request.

antban commented 2 years ago

@a1exsh yes, zstd response compression is not supported

a1exsh commented 2 years ago

@antban Counting actual bytes on the uncompressed stream is the only reliable way to do it on this level, as the Content-Length header is only set by some of the user agents (and they are not required to set it anyway). If we want to track compressed sizes as well, we should be doing that on the compression filter level.

adyach commented 2 years ago

👍

antban commented 2 years ago

:+1:

a1exsh commented 2 years ago

:+1:

antban commented 2 years ago

:+1:

a1exsh commented 2 years ago

:+1:

antban commented 2 years ago

:+1: