Jpylyzer crashes on very large images

[bitsgalore]

Jpylyzer loads the whole image into memory, which causes an exception if system memory is exceeded. No easy solution for this, as changing this behaviour would require fundamentally different way of parsing.

To do:

• Add exception handler so memory errors are handled gracefully
• Perhaps add output element that reports jpylyzer success/failure at analysing file

[bitsgalore]

From version https://github.com/openpreserve/jpylyzer/releases/tag/1.15.0 onward jpylyzer uses memory mapping. This doesn't solve this issue completely, as memory errors may still occur (also depending on hardware/OS setup).

[stweil]

Especially 32 bit operating systems (Linux / Windows in i386 mode) cannot mmap more than 4 GiB (minus system and program memory, so the real limit might be lower than 2 GiB, on Windows even lower than 1 GiB). For those systems, tests and code fixes to handle mmap errors gracefully will still be needed. These tests can be done with any kind of file (no JP2 needed), for example with DVD images or video files, because only the file size matters.

On 64 bit operating systems, even very large files (much larger than RAM size) should work thanks to mmap.

[bitsgalore]

On 64 bit operating systems, even very large files (much larger than RAM size) should work thanks to mmap.

I tried this, but I couldn't get a 6.7 GB image working under 64-biy Linux Mint with 4 GB RAM. See also the comments I made here:

http://jpylyzer.openpreservation.org/2015/12/03/Release-of-jpylyzer-1-15-1/

However my impression is that things only go wrong after the image data are read/mapped, as in the case of the 6.7 GB image jpylyzer would spit out an "unknown box" warning before the error occurred. That might also explain why a test with e.g. a large DVD image doesn't result in the error, because jpylyzer cannot really parse those file for much beyond the start of the file anyway (unlike a JP2).

[bitsgalore]

OK, so I think I found a way to make jpylyzer handle any memory errors gracefully without crashing. Coming up shortly!

[bitsgalore]

First stab here:

https://github.com/openpreserve/jpylyzer/tree/improvedExceptionHandling

• In checkOneFile I created a handler for any exceptions that may occur while reading/analysing a file
• Added a statusInfo element to the output which contains a Boolean success field, and (in case an exception occurred) a failureMessage with a user-friendly message of what went wrong
• In addition a warning is printed to screen for each exception.

This should prevent any crashes in case of memory and I/O errors. If multiple files are processed at once, jpylyzer will keep running in case of an exception.

Also updated the XSD schema accordingly a (using 1-1 version number, so previously created jpylyzer output will still validate); published schema at http://jpylyzer.openpreservation.org/jpylyzer-v-1-1.xsd

Two examples of output files here:

https://gist.github.com/bitsgalore/a76a64804f772a569299

This needs some further testing (also under Windows).

[bitsgalore]

Changes added to 1.16.0 release: https://github.com/openpreserve/jpylyzer/releases/tag/1.16.0