carrierwave-daltonize

[gingerlime]

Hey Guys,

I just created a small gem that relies on carrierwave-vips and implements the daltonize algorithm for processing images for colour-blind people.

The actual algorithm was implemented by @jcupitt since I have extremely limited (read, nonexistent) experience with image processing.

However, I need help with a couple of small-ish items that might be easier for anybody with a bit more experience with ruby-vips. Namely:

1. I couldn't work out how to run the algorithm on a png image. It complains with VIPS error: im_disp2XYZ: input not 3-band uncoded char. I hope it's a trivial thing to convert / make this work.
2. Whilst the conversion for deuteranope, protanope and tritanope all work, I'm getting slightly different results than, say, on http://daltonize.appspot.com/ - looking at both the python and javascript imlementations, the last step includes some error correction, which is missing from @jcupitt's algorithm. This might be the reason for the difference?? any help would be much appreciated.

As I said, I have very limited image processing experience, but would really like to get this working, so images can be processed on any app using carrierwave to support colour-blind people. Thanks in advance!

update: I managed to apply some error correction, and it looks closer, but it's still not 100% the same as the results I'm getting using the javascript chrome-plugin or bookmarklet...

[eltiare]

You know, I have no experience with this type of image correction - though it's something I would consider putting in the gem itself. Seems as though something that should be a little more universal.

[eltiare]

As far as the PNG goes, I found this article:

http://stackoverflow.com/questions/2290336/converting-png-into-jpeg

It looks like you can't use 4-band (Alpha, Red, Green, Blue) in VIPS for this. See if there's some way to do this. Perhaps @jcupitt could help you out. I think VIPS is amazing, but the documentation sure needs some help. :)

[gingerlime]

Thanks @eltiare . I'm hoping we won't need to convert to jpeg and keep all channels, but I really haven't got a clue how to deal with this, vips or otherwise. Yeah, hopefully one of the vipsperts can jump in and do their magic. @stanislaw maybe?

As for embedding this inside carrierwave-vips, In my opinion it's best to leave daltonize outside as a separate gem, so only if someone needs this functionality they can use it. This won't bloat the carrierwave-vips gem with unnecessary functionality. First priority is getting it to work fully though...

[stanislaw]

Sorry guys, I am really far from this issue. I sent John a link to this issue - hopefully he will look at it soon.

+1 for separate gem, but carrierwave-vips README should have a link to it.

[gingerlime]

Thanks for your help guys. Just a quick update:

I haven't heard anything back from John or anybody else who can help with this, and got stuck figuring out how to support other image formats or doing the error correction fully on the final step.

Not wanting to give up so easily, I ended up creating a branch of carrierwave-daltonize that wraps the python code and calls it to perform the conversion. It seems slower than vips, and might take more memory, but it gives a more complete implementation it seems.

[jcupitt]

Sorry, I've been on holiday, I'm still catching up. I'll try to look at this tomorrow.

[gingerlime]

thanks @jcupitt - I didn't mean to sound ungrateful, just that I didn't hear back and wasn't sure what to do. I think vips is probably the better approach (and certainly runs faster), but only if we can make this work. Unfortunately I just couldn't figure this out on my own :-/

If you can take a closer look some time that would be great!

[eltiare]

Since when do you get vacations, @jcupitt? Is that in the budget?

[jcupitt]

Hi again, here's a better Daltonize program:

#!/usr/bin/ruby

# daltonize an image with ruby-vips
# based on
# http://scien.stanford.edu/pages/labsite/2005/psych221/projects/05/ofidaner/colorblindness_project.htm

require 'rubygems'
require 'vips'

im = VIPS::Image.new(ARGV[0])

# remove any alpha channel before processing
alpha = nil
if im.bands == 4
    alpha = im.extract_band(3)
    im = im.extract_band(0, 3)
end

begin
    # import to CIELAB with lcms
    # if there's no profile there, we'll fall back to the thing below
    lab = im.icc_import_embedded(:relative)
    xyz = lab.lab_to_xyz()
rescue VIPS::Error
    # nope .. use the built-in converter instead
    xyz = im.srgb_to_xyz()
end

# and now to bradford cone space (a variant of LMS)
brad = xyz.recomb([[0.8951,  0.2664, -0.1614],
                   [-0.7502,  1.7135,  0.0367],
                   [0.0389, -0.0685,  1.0296]])

# through the Deuteranope matrix
# we need rows to sum to 1 in Bradford space --- the matrix in the original
# Python code sums to 1.742
deut = brad.recomb([[1, 0, 0],
                    [0.7, 0, 0.3],
                    [0, 0, 1]])

# back to xyz (this is the inverse of the brad matrix above)
xyz = deut.recomb([[0.987, -0.147, 0.16],
                   [0.432, 0.5184, 0.0493],
                   [-0.0085, 0.04, 0.968]])

# .. and back to sRGB 
rgb = xyz.xyz_to_srgb()

# so this is the colour error 
err = im - rgb

# add the error back to other channels to make a compensated image
im = im + err.recomb([[0, 0, 0],
                      [0.7, 1, 0],
                      [0.7, 0, 1]])

# reattach any alpha we saved above
if alpha
    im = im.bandjoin(alpha)
end

im.write(ARGV[1])

It strips off any alpha at the start and reattaches at the end, which should fix your point 1. above, @gingerlime.

On 2., I'd misunderstood the point of the program, I thought it was just simulating the appearance of colour-blindness, not trying to correct for it. Reading too fast :-( Anyway, there's a chunk at the end now to add back a modified error.

You won't get the same results as the daltonize.py program you linked, but I think the results are equally valid, and probably better. The original program is not taking account of varying CRT phosphors, nor of varying image gamma. They are also adding the error term back in RGB space, not in LMS space, which has to be wrong. I've kept this wrongness above so as not to give results that are too different. They are also using the same error compensation matrix for all types of colour blindness, which again seems wrong. Won't adding the error to blue-yellow be wrong for a protanope? Some of the follow-up papers seem to try to address this.

The Python version takes about 40s to process a 1000x2000 pixel image, the ruby-vips one 0.3s. I think the Python one could be made much faster if they used the numpy vector operations, they seem to be looping over every pixel multiple times.

I'll update the version on stackexchange too.

[gingerlime]

Thanks again @jcupitt ! that looks pretty awesome. I'd love to get this into the carrierwave-daltonize gem. However, I ran a quick test and got this error with a png image

daltonize.rb:59:in `bandjoin': VIPS error: im_gbandjoin: input images differ in format (VIPS::Error)
        from daltonize.rb:59:in `<main>'

Seems fine with a jpeg however.

[jcupitt]

You have an older version of the vips library. 7.30 and later will fix the formats for you.

The problem is that the alpha channel is 8-bit, and the colour image is float, since it's gone through all those matrices. bandjoin is complaining that you can't join a float to an 8-bit image.

Change:

im = im.bandjoin(alpha)

to be

im = im.bandjoin(alpha.clip2fmt(im.format))

ie. cast the alpha up to im's format before joining. The jpeg save will cast back down to 8-bit for you.

[gingerlime]

Thanks again... I'm using libvips-dev from debian squeeze, so it's very likely that it's not the very latest version (I just checked and it looks like the debian package provides version 7.20). I can probably upgrade/install manually a later version, but I believe that compatibility is important/desirable to get a large user-base to use it. The prerequisite to install libvips is already something that might keep people from using it...

I tried it again with your modification and now get this error:

daltonize.rb:59:in `<main>': private method `format' called for #<VIPS::Image:0x000000017f6e70> (NoMethodError)

[jcupitt]

I agree, compatibility is good. I didn't realise the debian one was so old.

Try .band_fmt instead (the very old name for this field).

[gingerlime]

super-duper. This seems to work :)

Is this .band_fmt supported in future versions? or do we have to try whichever version works with some begin/rescue clause until the right version is found?

I'll get started integrating the algorithm into the carrierwave-daltonize gem, so it's easier to play with different versions. Thanks again John.

As a side-note regarding python performance and the whole daltonize attempt: my level of understanding of image processing applies across the board, python, ruby or any other language... it's basically non-existent in all of those languages. I just pulled whatever was there with some sticky-tape and good intentions. It's obviously you John that's the real brain behind this. Unfortunately I don't know anybody else with this knowledge so am very glad you stepped-in. I think it would be nice if you feel like contributing a faster python version, but it's entirely up to you of course.

[jcupitt]

Oh no problem, it's interesting to fiddle with this stuff. Yes, .band_fmt is in all versions.

There's a Python binding for vips so it'd be easy to move the ruby version over. I'd need to learn more numpy to be able to fix the linked version, that'd be better left to a numpy expert.

In my opinion there are a number of serious problems with the daltonize algorithm which ought to be fixed as well. That's more than I have time to work on right now though, unfortunately.

[eltiare]

I have updated the README to mention this gem. Let me know when I can close this thread.

[gingerlime]

@jcupitt - one more question. I noticed that you changed the deuteranope matrix in your code

        # through the Deuteranope matrix
        # we need rows to sum to 1 in Bradford space --- the matrix in the original
        # Python code sums to 1.742
        deut = brad.recomb([[1, 0, 0],
                            [0.7, 0, 0.3],
                            [0, 0, 1]])

How did you reach these new values then? and how can I do the same for the other matrices for tritanope and protanope?

Currently my matrices in the carrierwave-daltonize gem look like this:

    def deuteranope
      daltonize([[1, 0, 0],
                 [0.494207, 0, 1.24827],
                 [0, 0, 1]])
    end

    def protanope
      daltonize([[0, 2.02344, -2.52581],
                 [0, 1, 0],
                 [0, 0, 1]])
    end

    def tritanope
      daltonize([[1, 0, 0],
                 [0, 1, 0],
                 [-0.395913, 0.801109, 0]])
    end

How do I adapt them to the bradford space?

(and thanks @eltiare for updating the README, we'll hopefully have a working version pretty soon)

[jcupitt]

Rows need to sum to 1, so I just normalised them (ie. divide each by the sum).

I found that the deut. matrix they used, though it worked, gave a very pink look, so I swapped the first and last columns. It gives it more of the khaki that the original program used for indistinguishable red/green.

[gingerlime]

thanks John. I'll give it a try. On quick inspection I was getting somewhat different results than here, but maybe I used the wrong values... a bit caught with other stuff but will try to get back to this soon.

[jcupitt]

It won't give the same results, but they should be as good as or better in practice.

[gingerlime]

ok, I've updated the code based on your instructions, but still not so sure the results are correct.

code

require 'rubygems'
require 'vips'

def deuteranope
  # we need rows to sum to 1 in Bradford space, so the matrix was normalized
  # (each value divided by the sum of the values in a row)
  [[1, 0, 0],
   [0.2836, 0.0, 0.7164],
   [0, 0, 1]]
end

def protanope
  [[0, 4.0277, -5.0277],
   [0, 1, 0],
   [0, 0, 1]]
end

def tritanope
  [[1, 0, 0],
   [0, 1, 0],
   [-0.977, 1.977, 0.0]]
end

def get_matrix (prefix)
  matrix = case prefix
    when "d" then deuteranope
    when "p" then protanope
    when "t" then tritanope
    else nil
  end
end

def daltonize (image, matrix)

  # remove any alpha channel before processing
  alpha = nil
  if image.bands == 4
      alpha = image.extract_band(3)
      image = image.extract_band(0, 3)
  end

  begin
      # import to CIELAB with lcms
      # if there's no profile there, we'll fall back to the thing below
      lab = image.icc_import_embedded(:relative)
      xyz = lab.lab_to_xyz()
  rescue VIPS::Error
      # nope .. use the built-in converter instead
      xyz = image.srgb_to_xyz()
  end

  # and now to bradford cone space (a variant of LMS)
  brad = xyz.recomb([[0.8951,  0.2664, -0.1614],
                     [-0.7502,  1.7135,  0.0367],
                     [0.0389, -0.0685,  1.0296]])

  # through the provided daltonize matrix
  mat = brad.recomb(matrix)

  # back to xyz (this is the inverse of the brad matrix above)
  xyz = mat.recomb([[0.987, -0.147, 0.16],
                     [0.432, 0.5184, 0.0493],
                     [-0.0085, 0.04, 0.968]])
  # and now to bradford cone space (a variant of LMS)
  brad = xyz.recomb([[0.8951,  0.2664, -0.1614],
                     [-0.7502,  1.7135,  0.0367],
                     [0.0389, -0.0685,  1.0296]])

  # through the provided daltonize matrix
  mat = brad.recomb(matrix)

  # back to xyz (this is the inverse of the brad matrix above)
  xyz = mat.recomb([[0.987, -0.147, 0.16],
                     [0.432, 0.5184, 0.0493],
                     [-0.0085, 0.04, 0.968]])

  # .. and back to sRGB
  rgb = xyz.xyz_to_srgb()

  # so this is the colour error
  err = image - rgb

  # add the error back to other channels to make a compensated image
  image = image + err.recomb([[0, 0, 0],
                              [0.7, 1, 0],
                              [0.7, 0, 1]])

  # reattach any alpha we saved above
  if alpha
    # image = image.bandjoin(alpha)
    image = image.bandjoin(alpha.clip2fmt(image.band_fmt))
  end
  image
end

if __FILE__ == $0
  if ARGV.length != 3
    puts "Calling syntax: daltonize.rb [fullpath to image file] [target image full path] [deficit (d=deuteranop, p=protanope, t=tritanope)]"
    puts "Example: daltonize.rb /path/to/pic.png /path/to/deuteranope.png d"
    exit 1
  end

  matrix = get_matrix(ARGV[2])
  if matrix.nil?
    puts "color-deficiency must be either d,t or p"
    exit 1
  end

  image = VIPS::Image.new(ARGV[0])
  image = daltonize(image, matrix)
  image.write(ARGV[1])
end

Here's the original image, and the results for deuteranope, protanope and tritanope for both the ruby and python versions:

original

python deuteranope

ruby deuteranope

python protanope

ruby protanope

python tritanope

ruby tritanope

Did I make a mistake with the normalization process or something? It "feels" like something isn't right here, but I'm not entirely sure. Unfortunately I don't even know any colour-blind people to test this with...

[eltiare]

Well, if you don't have anyone to test it then you're just shooting in the dark. :) I'll see if I can dig up anyone that has color-blindness to test this. You're looking for the "right" way to do this, but instead of testing it out you're just trying to mimic what everyone else is doing. What everyone else is doing may not be a very good way to do it.

[gingerlime]

Thanks @eltiare . I've also asked on HN so hope some volunteers can step in and take a closer look.

[gingerlime]

got one response on HN so far:

I believe I'm a Strong Deutan. I'm reviewing your deuteranope set at that link you provided. The Python colors look a lot different than the original but I see the numbers the same in those two, with the exception of the middle right image, the "5" and the middle bottom, the "15". Each of those in the original is less defined and the 5's look closer to 8's for me (although I can still tell it's a 5). With the Ruby, every one of your numbers is hazier and harder to read with the exception of the top-left, the "12". However, I believe if given this test, my answers on the Ruby version would only deviate from the original for the middle-right (5), bottom-left (3), and middle-bottom (15). Also it seems the tan and yellow hues are more accentuated in the Ruby version than the original, but the colors are a lot closer than the Python version. I can give my review of the other tests if you want but with my color-blindness, I don't believe my opinion is relevant for those.

[gingerlime]

Following further discussion on HN, here's another trial with a different run of python vs ruby (thanks again to logn on HN who pointed out http://www.toledo-bend.com/colorblind/Ishihara.asp )

http://imgur.com/a/nCzap

[jcupitt]

Thank you for this @gingerlime, it's interesting to see these responses. As usual I think I was probably much too ambitious and overconfident, argh. I'm sorry I made all this extra work for you.

How about the following approach: first, change the ruby-vips version to use exactly the algorithm in the Python version. I don't like it and I think it's broken, but it is tested and known to work. Plus a ruby-vips version should be quick, which would be useful.

Second, make a new, experimental daltonize using the colour tools in ruby-vips. It should be possible to make something quite a bit better, I think.

I'll do 1. for you later today and put 2. on my to-do list.

[jcupitt]

Actually, maybe I jumped the gun and the new one's not taht bad. I had an idea for improving the way the error is added back. Try this version:

#!/usr/bin/ruby

# daltonize an image with ruby-vips
# based on
# http://scien.stanford.edu/pages/labsite/2005/psych221/projects/05/ofidaner/colorblindness_project.htm

require 'rubygems'
require 'vips'

im = VIPS::Image.new(ARGV[0])

# remove any alpha channel before processing
alpha = nil
if im.bands == 4
    alpha = im.extract_band(3)
    im = im.extract_band(0, 3)
end

begin
    # import to CIELAB with lcms
    # if there's no profile there, we'll fall back to the thing below
    lab = im.icc_import_embedded(:relative)
    xyz = lab.lab_to_xyz()
rescue VIPS::Error
    # nope .. use the built-in converter instead
    xyz = im.srgb_to_xyz()
    lab = xyz.xyz_to_lab()
end

# and now to bradford cone space (a variant of LMS)
brad = xyz.recomb([[0.8951,  0.2664, -0.1614],
                   [-0.7502,  1.7135,  0.0367],
                   [0.0389, -0.0685,  1.0296]])

# through the Deuteranope matrix
# they have no green receptors, so we divide the green signal between red and
# blue, 70/30
deut = brad.recomb([[1, 0, 0],
                    [0.7, 0, 0.3],
                    [0, 0, 1]])

# back to xyz (this is the inverse of the brad matrix above)
xyz2 = deut.recomb([[0.987, -0.147, 0.16],
                   [0.432, 0.5184, 0.0493],
                   [-0.0085, 0.04, 0.968]])

# now find the error in CIELAB
lab2 = xyz2.xyz_to_lab()
err = lab - lab2

# deuts are insensitive to red/green, so add the red/green channel error 
# to the other two
lab = lab + err.recomb([[1, 0.5, 0],
                        [0,   0, 0],
                        [0,   1, 1]])

# .. and back to sRGB 
im = lab.lab_to_xyz().xyz_to_srgb()

# reattach any alpha we saved above
if alpha
    im = im.bandjoin(alpha.clip2fmt(im.band_fmt))
end

im.write(ARGV[1])

This find the error in CIELAB, then adds 50% of the red/green error to lightness and 100% to yellow/blue.

I get this result with the ishihara images:

http://www.rollthepotato.net/~john/demo.png

That's plain ishihara on the left, the program above in the centre, and the Python version on the right. Colourblind friends tell me the middle one (the new one) is clearer.

I tried on a few photos and it doesn't seem to distort pictures too annoyingly for non-colourblind users. No worse than the Python one anyway.

To make the above thing work for other forms of colourblindness, you need to modify the modelling matrix (the one that predicts the appearance of the image to the colourblind person) as before, and also modify the error-adding matrix.

[gingerlime]

Thanks for not giving up on this @jcupitt

I managed to "recruit" 2 colour-blind people so far, probably not enough to test this thoroughly, not to mention with different types of deficiencies, but it's a start :)

For these two people, on the previous version I think it looked like the python code was doing better for them. Again, this is not conclusive, but kinda of the impression I was getting. I sent your demo image to them and hope to get more feedback, meanwhile I'm trying to find more colour-blind volunteers.

As for the approach, I'd be happy to have a "stable" version that mimics the javascript/python algorithm as you suggested, if that's not too much work. Then we can try to improve on it. I personally think that's the safest approach, at least for now. However, given that you're the real brain behind this anyway, I'd go for whatever you feel is best.

Assuming the latest version works better, I'd love to use it of course. However, I am still a bit lost on how to adapt the matrices / error correction matrices for protanope and tritanope. I normalized the matrices for the bradford space based on your suggestion, but as far as swapping colours or using different error correction - I have no idea how to do this. Do you have the time / energy to modify the other two transformations? (or just give me the matrices to use in each, and I'll stick it together)? Would you be able to build a 100% python-compatible version first as a starting point? Should we wait until we get some more feedback??

Thanks again for stepping in and helping with this John!

[jcupitt]

I forked your repro and did a pull request:

https://github.com/gingerlime/carrierwave-daltonize/pull/1

I've not tested it though, I don't have a harness for carrierwave handy. You'll need to verify that you get the same result for the Ishihara as above.

I made up the matrices for prot and trit, but they ought to work. We'd need some testing.

I think this approach is so much better than the original Python one that it's not worth implementing that as well.

TODO:

1. I don't think it'll handle alpha images, this might need adding
2. we can fold the recombs together. If you have a.recomb(M1).recomb(M2).recomb(M3), you can rewrite that as a.recomb(M1 * M2 * M3), for an obvious speedup
3. we should find a trit and a prot to confirm that it's working OK -- we'd need different Ishihara images for them, of course

[jcupitt]

I posted that link on shacknews

http://www.shacknews.com/chatty?id=29708262#item_29708262

They voted 4-0 in favour of the ruby-vips version.

[gingerlime]

That's cool. I'm trying to test it with carrierwave and will post the results when I get them. I actually also want to create a standalone ruby version as well, so I might do both so it's even easier to test without carrierwave.

I appreciate your approach John, and happy to try to build something that's actually better.

As for the todo list, Alpha meaning png images with transparency? I thought this was already solved, or is the new code not compatible with the old method? This is important once we verify that the algorithm is working.

Performance optimization would be really nice, but it's already an order of magnitude faster than the python version (maybe a few orders of magnitude), so I wouldn't worry that much about it. Code clarity in those situations might be better than performance.

I'll carry on trying to recruit colour-blind people. I'm hoping to get more volunteers so hopefully we can validate it better. I'll keep everyone posted on this. (just saw the update about shacknews - that looks aweseome!)

[gingerlime]

Is it ok if we move this discussion over to https://github.com/gingerlime/carrierwave-daltonize/pull/1 ?

[eltiare]

I'll go ahead and close this thread now.