joakime
commented
1 year ago You have to finish reading from the HTTP stream. The ZipInputStream does not read to EOF. Just closing the request.getInputStream is insufficient in many cases, as it can leave data on the connection that is unread (eg: breaking HTTP/1 connection persistence in the process). What if the transfer-encoding was chunked? and the terminal chunk hasn't been read from the network buffers yet? What if the transfer-encoding final chunk hasn't even been sent by the client yet? What if the stream contained superflous data, unrelated to the Zip stream after the zip stream?
All your code is doing is iterating through the Zip directory/dictionary/index for entries, and reading the entry content. How this is done is by reading the entire stream until it finds the bytes indicating the start of directory section at the end of the stream, then reading each entry of the directory to know where in the previous parts of the stream the ZipEntry contents are located (and what size they are). So you are basically reading ALL of the file contents first, then slowly iterating through the dictionary, and then going backwards in the zip stream to each entry contents, once you reach an entry in the Zip directory section that cannot be represented by java ZipEntry it stops. You haven't actually read the whole stream yet if that last entry is something structural that cannot be represented by a java ZipEntry. This technique of ZipInputStream also means that large zip files can blow out your memory quite easily. Most organizations that deal with zip files will stream it all down to a temporary file on disk, then read/parse/iterate/validate from disk (which is far more memory friendly).
How do you fix this? You have to satisfy the basics of HTTP here. Even libraries that read / parse JSON and XML have this same issue. Every modern JSON and XML library have options to continue reading to EOF. They can even be configured to behave differently depending on what is found after the content. Eg: in many XML and JSON parsers there's an option to fail the document parse if any non-whitespace is discovered after the document.
See TODO in the code snippet below.
val in = request.getInputStream
val zip = new ZipInputStream(in)
var entry = zip.getNextEntry
while (entry != null)
{
val out = new CountingOutputStream(new NullOutputStream)
zip.transferTo(out)
out.close
System.err.println(entry.getName + ":" + out.bytesWritten)
entry = zip.getNextEntry
}
zip.close
// TODO: you need to finish reading from `in` here, the HTTP stream is not yet complete, just the ZIP content is.
// You have to choose a behavior here,
// eg: if there is content, do you fail the upload? or ignore it the extra content? warn the user?
// small amounts of content ok? large amounts of extra content show a problem?If you don't want to do this, you can send the Zip stream in a multipart/form-data, which puts the HTTP stream behavior requirement on the Server, not you.
gregw
sbordet
ghost
lachlan-roberts
lorban
Jetty version(s) 12.0.1
Jetty Environment ee10 / http/2
Java version/vendor
(use: java -version)OpenJDK Runtime Environment (Red_Hat-21.0.0.0.35-1.rolling.fc38) (build 21+35) OpenJDK 64-Bit Server VM (Red_Hat-21.0.0.0.35-1.rolling.fc38) (build 21+35, mixed mode, sharing)OS type/version Fedora 38 Workstation
Description This application sets up an embedded Jetty 12.0.1 instance with one servlet class TestServlet1. The doPost method simply tries to read-and-discard the content of the body before sending a short response. Posting a sufficiently large file to the application will result in a lockup. The curl client will block trying to send more data while the application blocks trying to read. The point where th size of the POST body is printed to stderr is then never reached.
A slight modification of the code makes it use TestServlet2 with a doPost that processes the input expecting a ZIP-formatted body by reading and discarding entries and printing the name:size of each entry to stderr. Posting a sufficiently large ZIP file to it shows that initial entries are processed corrrectly then it locks up the same way as with TestServlet1.
Small POST requests seem to work without issues. GET requests seem to work fine also and our application ported from Jetty 11 to Jetty 12 works mostly fine except for large POST requests.
How to reproduce? See the above description. I am sorry that this is Scala code instead of Java. I currently have no Java development environment setup.
Compile/runtime dependencies specified in build.sbt, some of which redundant for the test code
Scala code