cloudflare / pingora

A library for building fast, reliable and evolvable network services.
Apache License 2.0
20.28k stars 1.1k forks source link

Disable Chunk Encoding by Config #202

Open janbraunsdorff opened 2 months ago

janbraunsdorff commented 2 months ago

What is the problem your feature solves, or the need it fulfills?

Bevor Upstream the content gets parse base on the corrosponding headers. Some Clients still use http1 with Chunk encoding. Is there a way to disabel Chunk encoding parsing for all requests from Downstream?

Describe the solution you'd like

A server Config to default disabele Chunk Encoding in the yaml file.

Describe alternatives you've considered

Solve the issue client Side or use an nginx befor the proxy.

Additional context

AWS S3 clients and pyspark still uses Chunk Enconding on http1. During access throw the proxy, the request will fail because the md5-hash is incorrect. Therefore is there way to pass the payload throw without converting it?

Or am I completly wrong and miss something?

eaufavor commented 2 months ago

Do you mean that you want a feature to preserve the exact request chunked encoding through our proxy? Or do you what a feature to convert chunked encoding to something else (content-length)?

janbraunsdorff commented 2 months ago

I want to have a feature to pass throw the chunked encoding.

client --> Proxy --> Server. The Server needs the exact encoding from the client. No buffering or Parsing. So the proxy only reads the header, do something (e.g. Routing and Auth) an pass throw the body without parsing oder try to read all chunks

janbraunsdorff commented 2 months ago

Hi everyone

for more infromation, i try to build an Gateway for Authentification for an S3 Storage. Older Clients want to use chunked Encoding. And the Chunks will be reassembled by S3. So I need an option to pass throw the raw payload. Is this possible?

Thank you :)

eaufavor commented 2 months ago

It is possible for us to implement something that doesn't combine chunks internally in order to preserve. Because this is a relatively niche ask, it has not been worked on yet.