Can libvips' stack size be modified using pyvips?

[TWCCarlson]

Hello (again, sorry if this is trending toward spam),

I'm writing a fairly long and involved processing pipeline and I'm wondering if I've hit a wall related to the stack size. When I run the process normally it simply terminates with no message. Setting logging to level DEBUG didn't seem to help give any extra information. If I run it using memory_profile.memory_usage() I can get the following:

Traceback (most recent call last):
  File "C:\Users\aepoc\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "X:\GitHub\osrs-wiki-maps\.venv\Lib\site-packages\memory_profiler.py", line 265, in run
    self.pipe.send(self.mem_usage)
  File "C:\Users\aepoc\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\aepoc\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 289, in _send_bytes
    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [WinError 232] The pipe is being closed

Essentially I am trying to composite several thousand smaller images overtop a sizable base image, repeated a few times on different base images. Somewhere between 200 and 300 compositing operations on the first of many images, the program will trigger that error. I get slightly better results using insert on an empty image which I intend to overlay later (up to ~600 insertions on the first image3).

I read in the libvips documentation that it's possible to set an environment variable to configure this but it doesn't seem to solve my issue:

https://gist.github.com/TWCCarlson/aa0e34b2c2dc01479ef8949c7ca4856b

The worst case base image is <pyvips.Image 26112x91136 uchar, 4 bands, srgb>, and the sprites being overlayed are only 15x15 RGBA images.

From what I've understood so far this is simply not a good way to process the image—especially as I will be slicing it into tiles later. What I am trying to accomplish is:

• Load some baseline images
• Do some basic image processing techniques; blur, contrast enhance, grayscale, etc.
• Rescale by factors of two in preparation for slicing each layer into slippy map tiles
• Insert icons which need to be of constant size at each level of zoom (hence scaling before insertion)
• Use dzsave to generate 256x256 tiles from each level of zoom I created

The problem I am trying to avoid is that when I insert the icons they may overflow the tile bounds in any direction. If I were to try and insert the icons after slicing I would need to develop some process which checks if an icon overlaps any given slippy tile (or vice versa). While possible this seems like it would involve a lot of searching and generally be quite slow for the number of tiles and zoom levels I need to handle. Further, I've observed that splitting the process up across many images can result in the same failure, so this might not be working even for more but smaller images...

My hope is that this simpler approach can be made to work, but if this is simply too much then I think I can manage by switching back to pillow for this final operation, which just works albeit more slowly and with massively more memory usage. Any hints about what isn't working or what might work better would be greatly appreciated.

[jcupitt]

Hi again @TWCCarlson,

Yes, you're doing something that pyvips is very bad at, unfortunately. libvips pipelines are trees and on windows you can't have a maximum depth of more than about 1,000 nodes. You could try in WSL, you should be able to get up to a few thousand there, but of course it still might not be enough.

I think the best solution would be to draw the icons as an overlay in the slippy map image viewer, rather than burning them into the file. leaflet, openseadragon, etc. support drawing overlays, it ought to be easy. If you do it that way, you can use libvips dzsave, which will shrink and cut into tiles very quickly.

If that's really not possible, then you need to render the image to memory with copy_memory() and use the draw operations to paint the icons into the memory area:

https://www.libvips.org/API/current/libvips-draw.html

Unfortunately, pyvips does not have good support for this -- it will copy the memory area each time. I wrote a post about this a while ago:

https://www.libvips.org/2021/03/08/ruby-vips-mutate.html

So ruby-vips now has this nice thing:

image = Vips::Image.new_from_file ARGV[0]

image = image.mutate do |mutable_image|
  1000.times do 
    mutable_image.draw_circle! Array.new(3) {rand(255)},
      rand(image.width), rand(image.height), rand(100), fill: true
  end
end

But pyvips still doesn't. If you feel very brave you could try adding something like a mutate class to pyvips (I could help), or you could switch to ruby (it's a nice language), or you could use pillow.

[TWCCarlson]

libvips pipelines are trees and on windows you can't have a maximum depth of more than about 1,000 nodes.

What about on Linux proper? The eventual goal for this pipeline is to run it as part of a weekly GitHub Action on a Linux runner. I'll give it a run in a Docker container to see about WSL.

Also, what constitutes a node in the tree here? One .insert operation? Am I right to guess that if I write and reload the file repeatedly I could get around this (with losses in write time of course)?

I think the best solution would be to draw the icons as an overlay in the slippy map image viewer, rather than burning them into the file. leaflet, openseadragon, etc. support drawing overlays, it ought to be easy. If you do it that way, you can use libvips dzsave, which will shrink and cut into tiles very quickly.

This is what I suggested to the maintainers, but they shot it down due to aesthetic preferences with icons flickering during resizing :( I may take a stronger look to see if anything can be done about that if I can get access to the Leaflet source. I know that Leaflet allows custom marker designs which do not flicker as they are not a "layer" in the same way.

It sounds like I should be sticking with pillow for this.

If you feel very brave you could try adding something like a mutate class to pyvips (I could help)

Brave as I am and interesting as this sounds I think I may not understand enough about vips yet to make this work in any reasonable amount of time :/

[jcupitt]

What about on Linux proper?

A few thousand, though it varies with the exact operations and of course the stack size. But it's not a great solution.

Yes, one node is about one operation, though it depends on the operation. Some are built out of several smaller operations and will add many nodes. Resize (for example) is about 8 smaller operations internally.

You can often trade pipeline depth for width. For example, you could cut your large image into huge chunks, sized so that each would contain under perhaps 800 icons. You'd generate each chunk, then join them together with arrayjoin to make the final image. Maximum depth would then stay well under 1,000.

[TWCCarlson]

A few thousand, though it varies with the exact operations and of course the stack size. But it's not a great solution.

Makes sense, was just wondering.

For example, you could cut your large image into huge chunks, sized so that each would contain under perhaps 800 icons. You'd generate each chunk, then join them together with arrayjoin to make the final image. Maximum depth would then stay well under 1,000.

I tried in the past to supply negative coordinates to insert in the way that is possible with pillow's paste to handle (literal) edge cases where only part of an icon should be rendered but couldn't get it to work.

I suppose I would need to first develop a good way to sort my list of icons and positions against the coordinates contained in each chunk. From there I should calculate the crop using the chunk boundaries and execute it on the icon, then insert it in the appropriate spot. This is basically what I had in mind for drawing the icons onto the dzsave results but using arrayjoin would save a lot of read and write operations.

[jcupitt]

insert should work with -ve coordinates. If you set the expand flag it'll add more background pixels and output the bounding box of the two images, otherwise it clips.

>>> base = pyvips.Image.new_from_file("../IMG_0073.JPG")
>>> overlay = pyvips.Image.new_from_file("../nina.jpg")
>>> base.insert(overlay, -2000, -2000).write_to_file("x.jpg")
>>> 

To make:

I suppose I would need to first develop a good way to sort my list of icons and positions against the coordinates contained in each chunk. From there I should calculate the crop using the chunk boundaries and execute it on the icon, then insert it in the appropriate spot.

Assuming your chunks are on a regular grid, you can just loop over your icons and use icon.x // chunk_width etc. to put them into buckets. Edge icons will be in several buckets, of course.

Yes, arrayjoin is useful, and only makes a single graph node.