search

openpreserve / jpylyzer

JP2 (JPEG 2000 Part 1) validator and properties extractor. Jpylyzer was specifically created to check that a JP2 file really conforms to the format's specifications. Additionally jpylyzer is able to extract technical characteristics.
http://jpylyzer.openpreservation.org/
Other
69 stars 28 forks source link

Add explicit error on opening large file with 32-bit python #169

Closed VirtualTim closed 2 years ago

VirtualTim commented 4 years ago

Previously opening a file that's too large would just report the file as invalid. With this PR the user will see a message like:

User warning: The file is too large to open (5.9 GB). Try using 64-bit python.
Error: Opening file failed: mmap length is too large

This may sound like an edge case, but geospatial jp2s are often going to be too large for 32-bit python to handle.

Of course there is another issue in that this program reads the entire file into memory, which is going to be problematic for really large files, but this is a start.

I also changed open to with open to ensure the file is closed on failure.

bitsgalore commented 3 years ago

Hi Tim,

Thanks for the PR, I just merged this into the jpylyzer 2.1 dev branch:

https://github.com/openpreserve/jpylyzer/tree/v2-1-dev

I also applied some minor changes, mostly to avoid problems in case jpylyzer is run for multiple files (the code in the PR would effectively terminate jpylyzer after the first error occurrence, and we don't want that). While at it I also created a separate handler for the case of empty files, and added a warning message.

Could you check if this works, as I don't currently have 32-bit Python installed here.

VirtualTim commented 3 years ago

I don't currently have 32-bit python either, but the code looks good to me.