Chunked: Support Chunked Transfer Encoding for client requests (RFC 9112 6.1)

[robjr]

Hi guys, I'm facing this issue where unit returns 411 content-length required when the request doesn't specify the content-length, but it actually specifies Transfer-Encoding: Chunked.

    const formData = new FormData();
    formData.append('file', request.get('any.com/img.png'));
    await axios.post('svc.com/media', formData, { headers: formData.getHeaders() });

It is streaming a file from one end to another and it works locally, but whenever it goes through unit, it returns 411. Is there any configuration to enable transfer-encoding?

[VBart]

Unit doesn't support chunked transfer encoding in requests for the moment. So, the only options right now is to specify the Content-Length.

[mafin]

Good question - is there any configuration to change the Transfer-Encoding?

[VBart]

@mafin It needs to be changed on the client. There's no way to control client's request Transfer-Encoding from the server side.

[saper]

Ouch, we just ran into this issue with Axway API Gateway ...

[tippexs]

Hi @saper thanks for reaching out? What do you mean with that? Can you explain that? Thanks Timo

[saper]

Hi @saper thanks for reaching out? What do you mean with that? Can you explain that? Thanks Timo

We started using unit for some REST services and now someone put Axway API manager in front of them - the result - unit always sends 411 Length Required, because API manager ALWAYS transforms requests with Content-Length into chunked encoding.

We also could not easily tell unit to dump all requests and responses received, hard to trace the down (debug mode prints lots of syscall and thread info but no actual requests).

[lcrilly]

Thanks for the extra context - that's really helpful. In this case, I suspect the Axway behavior is due to it's transformation features for things like JSON/XML conversion. It's a shame that it does the same even when no transformation occurs.

We strive to be RFC compliant, and updated the issue title to reflect that. For now, this is on the backlog, but no ETA just yet.

[saper]

Why "Ice-Box"? Is something unclear about it?

I was thinking, isn't websocket code already doing something similar to chunked encoding?

[ac000]

RFC 7230 has been obsoleted by RFCs 9110 & 9112

[lcrilly]

Unit's HTTP implementation is against the 7000-series RFCs which is why I specified it that way. As the chunked encoding of responses follows RFC 7230 we should be very careful before switching to 9000-series specs.

[saper]

7230 or 9110, any chance for an implementation? There is a chunking-like feature in the websocket code, but I couldn't find any easy way to factor the code out to be shared between the WS and HTTP

[lcrilly]

A demonstration/prototype patch is now available (PR https://github.com/nginx/unit/pull/1262).

Please give it a try!

[callahad]

(Note the initial prototype has severe limitations at the moment: it "assumes that the chunked body and header are received all at once" -- but we're actively working toward a robust implementation.)