Converting GM filters (eg, Catrom, Blackman) for use in ruby-vips

[bnferguson]

General Image processing question here. I'm fairly new to lower level image manipulation but have vips doing my basic things likes resizes and crops.

However, now I'm trying to get my images after resizing up to comprable quality of that in GraphicsMagick. Over there I use a Catrom filter on resize. I've looked up the code that does most of this (and in cases of Hamming and others fits the formulas I've seen) but have no idea how to apply that knowledge to images in vips. I've seen mapping linear equations to a lookup table but spaces with if statements like Catrom, and spaces with parameters I have not.

How might I implement things like these in vips? I feel like it may be something in the FFT space but I'm still so ignorant on all of this that I haven't the foggiest idea and have spent a day beating my head against a wall.

Thanks so much again!

[jcupitt]

You need to downsize in a few stages, perhaps you're doing this already:

https://github.com/jcupitt/ruby-vips/blob/master/examples/thumbnail.rb

It's block shrink to chop it down to about the correct size, then affine to get to the exact dimensions you need, then a finishing sharpen pass if you want to make it "pop" a bit.

That example is using a simple bicubic for the second stage of the resize. You'll probably see better results with something like :nohalo. Type this at the command-line to see the interpolators your vips has:

 $ vips list classes | grep -i interpol

[jcupitt]

... and ruby-vips should include a nice all-in-one image resize, it's annoying to have to cook one up from the low-level components each time.

How are you handling things like CMYK and images with embedded ICC Profiles? Something else to think about.

[bnferguson]

Very cool. I am actually doing the downsize in a few stages as it is but really hadn't messed with the interpolator on the downsize. Wasn't until this morning with a clearer head that I began to consider that what I wanted to do and those may be the same (and that my options may be limited at a C/C++ level).

I'll experiment with these to see if they get me where I need to go. Thanks a ton!

[jcupitt]

Good stuff. You'll need to do the shrink-on-load trick for JPEGs as well or performance will be awful.

[bnferguson]

An all-in-one resize would be beautiful. Has definitely been an education working through the low level components each time but I worry for anyone who inherits this code. I may look at writing a pull request if I get the chance. Another fellow had written a fairly clean wrapper around libvips for resizes, which I ended up basing mine off of. May be a decent starting point?

As for profiles, good question. Currently the answer is "I'm not". Which probably isn't a good answer. ICC profiles I'll probably end up stripping out (I think?) since this is for web usage and in the past I had stripped all meta data. CMYK I hadn't considered - suppose I just hope that our writers upload all RGB space images but then there's press sourced images, probably not a thing to rely on. Eck.

Finally, currently doing the shrink-on-load trick (that's the shrink_factor in the JPEG.new right?) - was doing something similar in GM so not a huge jump fortunately!

[jcupitt]

On ICC profiles, I would convert the final image to sRGB and then strip the profile. You should get the same appearance in all browsers then.

Use icc_import_embedded() to import to CIELAB with the attached profile, then save as a jpeg. You'll need a try/catch around the import in case the image does not have a profile. This will let you handle CMYK gracefully as well.

[bnferguson]

That is beautiful. You just saved me so much time around tracking down edge cases and fixing them. Will try that approach post interpolator experiment (quality is a huge factor here - feels like I'm asking for the impossible sometimes, small, high quality, and fast. haha).

[bnferguson]

So far so good, getting things much, much closer. Now it's a battle of too blurry vs too jagged, but feels like it always is. Seems that vsqbs gets me really close (and with an unsharp mask at smaller sizes) to resizing with the catrom filter on GM (seems artifacts are much fewer too). One really strange issue is a weird anomolies I'm seeing in a resized and cropped image.

Here's the image from the current GM based production machine:

and here's the same thing from the dev machine (I think this one was vsqbs with a mask. But the problem shows up with all interpolators):

If you put them each in a tab and toggle between the two you see strange seams as the image jostles. Any idea what could cause this? Could be really creepy on a face. ;)

[jcupitt]

Oh dear, you're right, it looks like a problem in the vips image.

That's a detail from the vips image just above the knife, you can see there's a double vertical line. It could be caused by giving .shrink() non-integer shrink factors, could you check?

It could also be a horrible vips bug, but I hope not, it's not on a tile boundary, I think. Could you send some sample code that does this bad thing?

[bnferguson]

Interesting! Here's the code that's doing it - only thing is that the file may have already been opened by the generic new before this (sometimes need to just read widths and such).

    # Shamelessly stolen from easy_image for now
    def resize(width, height, min_or_max=:min)
      width ||= self.width
      height ||= self.height

      mask = [
        [-1, -1,  -1],
        [-1,  32, -1],
        [-1, -1,  -1]
      ]
      m = ::VIPS::Mask.new mask, 24, 0

      ratio = get_ratio width, height, min_or_max
      return @image if ratio == 1

      if format == 'jpeg' # find the shrink ratio for loading
        shrink_factor = [8, 4, 2, 1].find {|sf| 1.0 / ratio >= sf }
        shrink_factor = 1 if shrink_factor == nil
        @image = ::VIPS::Image.jpeg @path, shrink_factor: shrink_factor, sequential: true
        ratio = get_ratio width, height, min_or_max
      elsif format == 'png'
        @image = ::VIPS::Image.png @path, :sequential => true
      end

      if ratio > 1
        @image = @image.affinei_resize :nearest, ratio
      else
        if ratio <= 0.5 && ratio != 0
          factor = (1.0 / ratio).floor
          @image = @image.shrink(factor)
          @image = @image.tile_cache(@image.x_size, 1, 30)
          ratio = get_ratio width, height, min_or_max
        end

        @image = @image.affinei_resize :vsqbs, ratio
        @image = @image.conv(m)
      end

    end

    def get_ratio(width, height, min_or_max=:min)
      width_ratio = width.to_f / @image.x_size
      height_ratio = height.to_f / @image.y_size
      [width_ratio, height_ratio].send(min_or_max)
    end

That should exhibit the problem. If it doesn't let me know and I can pick apart any other code that may be messing with the image.

[jcupitt]

Could you check the before and after sizes for that image too? How big was it before it was shrunk?

[bnferguson]

Yup! 540x397.

[bnferguson]

Ah, just double checked something. As you've probably noticed there's a center crop going on there. Tried the same image without the crop, while the image dances a little compared to GM it doesn't warp around like the other. The centercrop code looks like this:

    def centercrop(width, height, min_or_max = :max)
      @image = resize(width, height, :max)

      # Offsets for region
      nx = @image.x_size/2 - width/2;
      ny = @image.y_size/2 - height/2;

      @image = @image.extract_area(nx, ny, width, height)
    end

[jcupitt]

I think I've found it: you are enlarging the image. The before image is only 397 pixels high, but after processing it's 400x400.

This line in your code:

      if ratio > 1
        @image = @image.affinei_resize :nearest, ratio

Is using nearest-neighbour interpolation for enlargements. Try swapping that :nearest for :nohalo or similar.

Background: normally you'd avoid upsampling, images generally look fuzzy and awful. If you HAVE to upsample, use nearest, since at least it'll look sharp. PDF/Postscript does this, for example.

[bnferguson]

Oh, aha! Totally missed that. Enlargements are a special case (and somewhat of a bug for us) so that shouldn't be a problem. Oddly enough seems that the 400px was a bit of a magic number too. Upping it to say, 1000x1000 doesn't exhibit the same deformations. Strange, but we should do just fine. Now it seems if I drop the unsharp mask with anything below ~150x150 or 200x200 I get results very close to the catrom filter in GM. Beautiful!

Thanks so much for your help in all of this, I'd be spinning my wheels still without it. :)

[bnferguson]

Running into a small problem with very small images (150x150 or less). It's very slight but it has come up with some of the co-workers.

Here's our current system:

and here's the new vips one:

The areas of concern have been around the bowl of nuts and the top of the jar. Seem a little more artifacty? I think some of this is that our current system errors on the side of soft. I spent some time trying to apply very, very slight blurs and then blending them (couldn't find a good way to do image blending) to soften up edges. Even tried doing it using and edge detection mask, bluring, and applying it (that got weird, I think I did it wrong).

Starting to wonder if I'm just running into the limits of the interpolators. Would my best approach here be looking into submitting a patch in libvips itself for the interpolators we use now? Or is there some approaches that I'm missing?

Finally - is there a good introductory text for this sort of stuff? I find it wildly interesting but I'm constantly bumping into the extents of my knowledge and it seems hard to find helpful info (other than constantly submitting GitHub issues :P ).

Thanks once again, and hope you have a great weekend!

[jcupitt]

I opened a new issue for this, since it seems like a separate question:

https://github.com/jcupitt/ruby-vips/issues/39