have tests for image data bigger than 4G ... that use libvips.

[ndushay]

There are specs for this already that are skipped by default; see in image_spec.rb L56, L76. It uses ImageMagick for creating the images, we probably want to switch that to libvips.

Work with Tony C and ... ?? as to how to do this.

In image.rb, image_data_size method https://github.com/sul-dlss/assembly-image/blob/main/lib/assembly-image/image.rb#L143-L146 needs to be exercised with large image.

The test generates 4GB images (and its 4GB for the source and 4GB for the temp tiff), so that storage needs to be available - (on local VMs, on CI ..)

[ndushay]

Tony says:

There were tests in the rspec for it that are skipped by default, but I don't know if they'll still work as intended. They're dynamically generated, so it is likely just a matter of making sure the spec_helper can still generate the images. (If I remember right, it was just using ImageMagick to generate them, so if those binaries are available it should still work.)

ImageMagick is on our VMs, as it is a dependency of libvips.

[ndushay]

ACTION for this ticket:

• [ ] work with Tony on getting large image tested "manually." Maybe that's about uncommenting the current large image spec and updating it, and having Tony run it "locally" for him.
  • [ ] if we're keeping the spec code, make sure it works for libvips, etc.
  • [ ] if the spec code isn't worth keeping, let's get rid of it
• [ ] document in README how to test large images

[jcoyne]

@ndushay can you explain why 4G is the threshold you picked?

[ndushay]

@calavano I'm going to assign this ticket to you;

@jcoyne ... the code we have now has this limit - it may have to do with filetype size limits.

[jcoyne]

Does anyone know what is limiting this to 4G?

[andrewjbtw]

I believe tiff has an upper bound at 4 GB. Larger images (which we do get sometimes) use the bigtiff file format.

[jcoyne]

Would this just be testing that kakadu can handle a Bigtiff image? Presumably there are tests in kakadu for this. I'm wondering if there's anything within the software we are writing that would require such a test.

[ndushay]

Would this just be testing that kakadu can handle a Bigtiff image? Presumably there are tests in kakadu for this. I'm wondering if there's anything within the software we are writing that would require such a test.

https://github.com/sul-dlss/assembly-image/blob/main/lib/assembly-image/jp2_creator.rb#L103-L106

Tony C. brought this up, hence the ticket. This ticket exists for a real reason. There is a real need for us to know OUR code handles images bigger than 4g. Your questiions are welcome, but you see above I said "The code we have now has this limit" ... I would have expected that to be enough for even @jcoyne to have faith in there being a real need for this ticket.

[calavano]

@jcoyne The test-case would be for the temporary TIFF generation. "Normal" TIFF files have a limit of 4GB from the format's use of 32-bit offsets, BigTIFF uses 64-bit offsets to get around this. We need to ensure that when there is >4GB of image data the temporary TIFF is saved using 64-bit offsets. We currently calculate this by checking to see if height * width * channels is >4GB and enabling it when true.

[jcoyne]

Thank you @calavano that's helpful.

[ndushay]

Since we are now using bigtiff for all temporary tiffs, there is no longer a separate code path for large image data and this ticket is moot.