Unable to save images when certain combinations of them are resize()'d

[TWCCarlson]

Hello,

I am encountering a weird but reproducible (on my machine) bug with writing images to file.

I've written a script to automate resizing four base images (which I call planes) to a number of zoom levels, down/upscaling similar to the way a slippy map is structured. My script iterates through the zoom levels, then the four base images, and then writes the resized results to disk.

Here comes the weirdness:

• A few days ago I ran this script several times and had no issues.
• Resizing planes 0, 1, 2, and 3 consecutively results in the following error:

Traceback (most recent call last):
  File "x:\GitHub\osrs-wiki-maps\scripts\zoomMapImages.py", line 56, in <module>
  File "x:\GitHub\osrs-wiki-maps\scripts\zoomMapImages.py", line 27, in buildZoomLevels
    buildZoomLevel(zoomLevel)
  File "x:\GitHub\osrs-wiki-maps\scripts\zoomMapImages.py", line 47, in buildZoomLevel
    writeFile(zoomedImage, zoomLevel, plane)
  File "x:\GitHub\osrs-wiki-maps\scripts\zoomMapImages.py", line 53, in writeFile
    image.write_to_file(os.path.join(outputDir, f"plane_{plane}.png"))
  File "X:\GitHub\osrs-wiki-maps\.venv\Lib\site-packages\pyvips\vimage.py", line 811, in write_to_file
    return pyvips.Operation.call(name, self, filename,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "X:\GitHub\osrs-wiki-maps\.venv\Lib\site-packages\pyvips\voperation.py", line 305, in call
    raise Error('unable to call {0}'.format(operation_name))
pyvips.error.Error: unable to call VipsForeignSaveSpngFile
  wbuffer_write: write failed
windows error: The printer is out of paper.

• Resizing any plane 4 times in a row does not trigger a crash
• Resizing any 3 planes does not trigger a crash
• Resizing more than 4 images but not using all 4 planes (e.g. 3, 2, 2, 1, 3) does not trigger a crash
• Resizing 4 images but not using all 4 planes (e.g. 0, 1, 2, 2 or 3, 2, 1, 1) does not trigger a crash
• Resizing planes in the order 3, 2, 1, 0 triggers the same crash
• Switching the order of my loops and using all 4 planes does not trigger a crash

If there is no crash, the output is exactly what I expect for any particular image.

It seems to me that for some reason (even though all these operations are done sequentially, not parallelized) that any run of the script which uses all four images causes a problem in the buffer. I can't really imagine why...

Technically my problem is solved by switching my order of iteration, but I am very intrigued as to why the other way would be a problem in the first place. If this isn't interesting please feel free to close as solved.

Here is a link to the images (one is too big to upload here): https://drive.google.com/drive/folders/11N3relUy4yhual9H3h5WUv7sbUsXV0Qq?usp=sharing

[jcupitt]

Hi @TWCCarlson, that does sound bizarre. I'll see if I can reproduce this.

[kleisauke]

The odd printer is out of paper-error (error code 28) might indicate that the disk is full, see: https://github.com/conan-io/conan/issues/15578#issuecomment-1952647188

[jcupitt]

My guess is that your temporary files area is filling up. When you do:

        inputImage = pv.Image.new_from_file(imagePath)

With one of your large PNG images, libvips will unpack this image to a large temporary file (1.6gb for one image from the sample you linked) and add this temp file to the libvips operation cache. When you next load that same source image, it'll reuse the right temporary file and save the unpack operation.

If you repeat this for all your planes, you'll need almost 7gb of temporary storage. Perhaps this is exhausted? It sounds possible.

[TWCCarlson]

That does sound like it—I have just over 6GB free on the disk holding my temp files right now.

Does this mean the temp files persist until the script is completely done? Or is vips aware of whether an unpacked file won't be referenced again in the script's lifetime?

[jcupitt]

I would use access="sequential" for the new_from_file --- you'll avoid the temporary file use.

You could also do all the plane_0 resizing, then all plane_1, etc. This would drop your temporary file use by x4.

And if possible I'd consider switching to something like deepZoom or zoomify. Tiled images are much simpler to work with! You can use pyvips dzsave to make them efficiently. I see:

$ time vips dzsave plane_0.png x --suffix .png

real    0m12.122s
user    0m54.884s
sys 0m18.259s

So it makes a complete deepzoom pyramid for one of your planes in 12s on this PC.

[jcupitt]

Does this mean the temp files persist until the script is completely done?

You can control the cache policy with:

https://libvips.github.io/pyvips/voperation.html

[TWCCarlson]

You could also do all the plane_0 resizing, then all plane_1, etc. This would drop your temporary file use by x4.

I think this is what I was (accidentally) achieving by switching the order of my for loops. Makes sense!

And if possible I'd consider switching to something like deepZoom or zoomify. Tiled images are much simpler to work with!

This was my initial approach, but unfortunately the wiki for whom I am writing these scripts would like me to paste relevant icons with a consistent size at each zoom level. This means that I can't create the tiles in each layer of the pyramid until I've placed those icons in the correct position :(

I could write logic to paste them after slicing the pyramid, but there are some other issues at play (icons spilling out of images which are then translated)... I may try to do this in the future.

Thanks for the swift response!