Closed cccs-jh closed 3 years ago
What do you suggest? Limit Hachoir memory? Don't parse a JPEG file if it's truncated?
Oh, I noticed afterwards that you proposed PR #68 to fix it. I merged your fix, thanks.
It's also good to see that Hachoir is used ;-)
A truncated jpeg can have a JpegImageData field with no terminator, which is created without a known size. Because the size isn't known the corrupted JpegImageData must be parsed in full to calculate its size when the field is added to its parent JpegFile during JpegFile parsing. This forces simple operations that don't care about the JpegImageData, like checking if a field with a given name is in the JpegFile, to parse the corrupted JpegImageData fully. Parsing the corrupted section can blow up the memory use of the parser as it tries to parse the entire rest of the file in small chunks.
An example file that causes this issue can be found here: https://github.com/CybercentreCanada/assemblyline-service-characterize/issues/12. This jpeg truncated to 500 000 bytes consumes approximately 1 GB of memory parsing JpegHuffmanUnits until it reaches the end of the file and errors. This happens when extracting metadata, or whenever a checking for a field name in the jpeg that isn't there.