Improve performance for Fill Resourcespace command

[Hobbesball]

Detailed description of the issue. Uploading images to resourcespace using the app:fill-resourcespace command is now rather slow, and won't scale for thousands of images.

Additional context If we want this ETL to run periodically, it's important that it is optimised as much as possible. The biggest bottleneck will always be the speed at which the Resourcespace API handles requests and processes images, so there is a hard limit to how optimised this can be without changing the resourcespace code itself.

Possible implementation

• refactor the code to have less nested loops to cut down on processing time

Current environment This issue was encountered when testing the app:fill-resourcespace command to fill Resourcespace on the VPS with the 185 images used in the test setup.

• Installed setup using the imagehub-box viaa ansible playbook
• Ubuntu 18.04.2
• Symfony 3.4
• PHP 7

[Kitania]

Fixed in 3498f51 - sort of, anyway

NOTE: make sure the following line of code is present in includes/config.php in the ResourceSpace installation the next time you run the command, otherwise it will screw up the filenames: $filename_field = NULL;

The next time you run this command, it will still be really slow as it needs to re-initialize all data. After that, subsequent runs should be really fast.

The bottleneck of the command was (and still is) the following line of code: new Imagick($fullImagePath)

Whenever this line of code is called, PHP reads the entire image into memory, even if we don't actually do anything with it. That being said, we can't expect a process to read and process 17GB of data in just a matter of seconds. One way or another, this bottleneck will always be there the first time the command is run. I don't think there is any way around that.

When removing all references to Imagick, the command is actually really fast and consumes a negligible amount of CPU and RAM. Trying to optimize the rest of the code won't make any kind of noticeable impact.

However, I did manage to pretty much completely remove the bottleneck in subsequent runs by only creating Imagick objects whenever needed. That is, whenever we notice the content of an image file has changed. This presented us with another problem: we can only know an image file has changed by looking at the hash of said image file.

ResourceSpace automatically generates an MD5 hash during upload, but we only upload a scaled down JPEG image as we don't want to upload the high-resolution image. To my knowledge, there is no way to set the 'file_checksum' field inside ResourceSpace through the API. There is also no metadata field in the current model suitable to contain this data. Therefore, we were forced to always create a scaled down JPEG file and take the MD5 hash of that JPEG file before we could know if an image had changed or not.

There are several possible solutions to this problem:

• Upload the high-resolution image after all, far from ideal as we want to avoid using twice the amount of hard disk space as needed
• Assume a file (with the same filename) will never change and drop the whole hashing and comparing, seemed like bad practice
• Perform a direct SQL call to the ResourceSpace MySQL database to manually insert the MD5 hash into the 'resource' table after uploading a resource, also seemed like something we want to avoid
• Create a new metadata field in ResourceSpace containing the file hash, but then either KMSKA would have to modify their resourcespace metadata model or our metadata model wouldn't be consistent with theirs anymore, didn't seem like a good idea either
• Create our own mongodb table containing the filename and MD5 hash of each image to compare against

I chose the last option; although it doesn't seem like a perfect solution either, it's better than any of the other approaches. If you guys have alternative suggestions for this issue, feel free to let me know.

[Hobbesball]

thanks for the detailed writeup!

The solution we're going with to this is yet another option than the once you already listed: Making sure the files in the drop folder are already scaled down with JPEG compression, and uploading those files directly to resourcespace, which should result in the images in RS having the same hash as the ones in the drop folder. The implications of this are that:

• images in RS will always have the same highest resolution as the same images in cantaloupe
• the scaling down to JPEG of images that is now done in this command should be removed
• testing to see if the hash stays the same and isn't changed after all through some resourcespace process is needed
• this should reduce the Imagick bottleneck since the images are already a whole lot smaller than before, which would result in it taking a lot less time for Imagick to read the whole image into memory.

[Kitania]

TL;DR version: the suggested approach does not seem to be viable - even with the scaled down files, uploading 1.7GB of images to ResourceSpace through the API takes half an hour, don't ask me why. Unless there are ways to heavily optimize the ResourceSpace API, I don't think we have any choice but to still scale down the images to small (< 1MB) JPEGs and upload these to ResourceSpace.

To verify the bottleneck in this case really is the ResourceSpace API, I performed a test run where I upload all 182 images directly to ResourceSpace, without using exiftool, Imagemagick or any kind of processing, so the only thing that happens is the actual upload (and subsequent image processing by the ResourceSpace API itself), it took half an hour to complete.

Secondly, I also encountered a bug in ResourceSpace: it can't handle file uploads through the API if the files don't have a filename extension.

ResourceSpace needs to know the extension of a resource and the code has built-in functionality for detecting extensions. It tries to do so through the filename; if no extension is found in the filename then it will use exiftool to auto-detect it. However, there is a bug in their code (line 162 in includes/image_processing.php) that throws an exception, resulting in the file upload failing:

$cmd=$exiftool_fullpath." -filetype -s -s -s ".escapeshellarg($processfile['tmp_name']);

For some reason, $processfile is not set, throwing an exception as a result. I have not yet dug deeper in their code to figure out why $processfile is not being initialized in the first place or if it's actually $processfile that needs to be used in the method call, rather than another variable.

Two possible workarounds (other than the ResourceSpace developers fixing the bug):

• Rename the file to have an extension right before upload, then rename it again right after the upload
• Copy the file to have an extension right before upload, then delete the copied file right after the upload

In my test setup, I have chosen the first approach because it is the least resource intensive. I have made and pushed these changes to a new branch, https://github.com/VlaamseKunstcollectie/Imagehub/tree/uncompressed_upload, where the suggested approach is implemented.

However, considering the command still takes half an hour to complete, the entire implementation seems rather useless to me. Also, further scaling down the already scaled down images also makes them very blurry in ResourceSpace, so that approach doesn't seem very practical either.

One more interesting thing I found is that processsing 1.7GB of data takes about 4 minutes to complete (on my local setup), while processing 17GB of data takes almost 5 minutes (278 seconds). These 278 seconds are spent on:

• ImageMagick, image processing, creating scaled down JPEGs: 120s
• Datahub calls: 25s
• Uploading images to ResourceSpace: 130s

Subsequent runs take 30 seconds to complete, 25 seconds of which are spent retrieving data from the Datahub.

Therefore, I would suggest to stick with the uncompressed TIFF files in the drop folder, have the command store hashes of the uncompressed TIFFs in a local MongoDB database and scale down the images to JPEG before uploading them to ResourceSpace.

[Hobbesball]

To solve this issue we need:

• compressed JPEG files to upload to resourcespace to fix the bottleneck
• compressed JPEGs to give to cantaloupe to fix performance issues there
• a fill-resourcespace command that doesn't read compressed JPEGs into memory so we don't have to allocate more memory than needed.

After extensive discussion this is how we would like to solve this issue:

possible implementation

• uncompressed TIFF files are put in a drop folder: iiif-data/original on the VPS
• a separate script runs a mogrify command to compress the TIFFs into JPEG into a folder: iiif-data/compressed.

  cd iiif-data/original
  mogrify -path ../compressed -define tiff:tile-geometry=256x256 -compress jpeg ptif:* -verbose

• the fill-resourcespace command takes the images from iiif-data/compressed, doesn't read them in to memory but simply copies the image file path, and gives that to the resourcespace API
• the fill-resourcespace command doesn't make any changes to the image, which it can't anyway because it doesn't read it into memory.
• the MD5 hashes that are autogenerated by resourcespace are the ones that are used to see if there are updates to the images in the compressed folder
• cantaloupe server the images in the compressed folder to the IIIF image API

[Hobbesball]

additional comments:

• md5 hashes of the first 50 kb are created by resourcespace, this needs to be configured in ResourceSpace itself
• the fill-resourcespace command creates an md5 hash of the first 50 kb in the same way the ResourceSpace does so they can be compared
• exif data is read in through the PHP exif command without the need to read in the full image

[Hobbesball]

when testing fill-resourcespace with JPG compressed TIFFs, we ran into issues #23 and #24, which have since been fixed. Closing this issue for now.