Currently request that require more than max_chunks_per_request (default 1000) or read/write more than max_request_size bytes (default 100mb) fail with a 413 error. The intent of these limits was to avoid out-of-memory errors on the SN node and/or too many requests being sent to the DN nodes.
Streaming support will get around these limits by continually reading from the input stream (for write requests) or continually writing to the output stream (for read requests) so that any given time there will be a limited amount of memory used and a limit to the number of inflight DN requests.
Larger requests will require longer to complete but at least the http headers will be received in a reasonable time.
jreadey
commented
2 years ago
Currently request that require more than max_chunks_per_request (default 1000) or read/write more than max_request_size bytes (default 100mb) fail with a 413 error. The intent of these limits was to avoid out-of-memory errors on the SN node and/or too many requests being sent to the DN nodes.
Streaming support will get around these limits by continually reading from the input stream (for write requests) or continually writing to the output stream (for read requests) so that any given time there will be a limited amount of memory used and a limit to the number of inflight DN requests.
Larger requests will require longer to complete but at least the http headers will be received in a reasonable time.