search

danforthcenter / plantcv

Plant phenotyping with image analysis
Mozilla Public License 2.0
646 stars 263 forks source link

readbayer color channels are out of order #642

Open dschneiderch opened 3 years ago

dschneiderch commented 3 years ago

Describe the bug readbayer() is producing color images where the bands are out of order and image is rotated. I am actually getting a RGB image back (with bayertype 'BG') which means all the pcv functions are converting it to BGR when displaying, for example.

I tracked down cvtColor bug that seems to exist on and off in opencv from 2.4 -> 4.1 https://github.com/opencv/opencv/issues/4857 The last version on conda forge with the correct behavior is 3.4.2. (did not try 4.x series)

To Reproduce download https://github.com/danforthcenter/plantcv/files/4314758/Pats.Wheat.Root.Growth_002-0001_A1_2019-06-11_07-19-08.461_0.zip

import cv2
from plantcv import plantcv as pcv
filename='data/images/Pats Wheat Root Growth_002-0001_A1_2019-06-11_07-19-08.461_0.tif'
image_raw=cv2.imread(filename, -1)
img = cv2.cvtColor(image_raw, cv2.COLOR_BayerBG2BGR)
pcv.plot_image(img)

image

Expected behavior

image or even better rotated too which happens with opencv 3.4.2 image

Local environment (please complete the following information): windows 10, conda, pcv 3.10.1 and latest master. seems dependent on opencv >3.4.2

Additional context 2 possible solutions until upstream gets fixed:

  1. limit opencv to 3.4.2

  2. switch the color rendering to color demosaicing package

dschneiderch commented 3 years ago

@HaleySchuhl @nfahlgren any chance you could reproduce this?

nfahlgren commented 3 years ago

I haven't looked at this from every angle yet but here's what I have so far.

Using OpenCV 3.4.10 I reproduce the incorrect first image above. In that version cv2.COLOR_BayerBG2BGR encodes code 46.

If I use the same Bayer filter we use on our LemnaTec camera (cv2.COLOR_BayerRG2BGR, which is code 48), I get the correct second image above (not rotated).

nfahlgren commented 3 years ago

I ran through 3.4.2, 3.4.4, 3.4.7, 3.4.8, and 3.4.9 from conda-forge and 3.4.2.17, 3.4.4.19, 3.4.7.28, 3.4.8.29, 3.4.9.33, 3.4.10.37, 3.4.11.45 from PyPI and can confirm that with BG2BGR for 3.4.2 and 3.4.4 the color is correct but the images are rotated. From 3.4.7 onwards the images are the correct orientation but the color is incorrect.

I checked the value of each of the cv2.COLOR_Bayer* modes and they match across all versions and sources. Based on the thread I tested the idea that the the BG2BGR and BG2RGB were identical because BGR pointed to the wrong method but the images are different across all versions.

I'm wondering if the issue was in 3.4.2 and 3.4.4 and that the RG method is correct but looked wrong previously because of the potential bug?

dschneiderch commented 3 years ago

oh! that's interesting. I potentially developed readbayer() with a buggy cv2. I don't know what version of opencv I was using then. did you go backwards from 3.4.2 ?

nfahlgren commented 3 years ago

I was hoping to look back at the build logs to see exactly what version tests were running on but because we switched platforms I don't seem to have access to the old logs. But readbayer was added to release v3.0, which was about 22 months ago. On conda-forge, the opencv package was at 3.4.4 around that time. The PyPI version was in the 3.4.4 to 3.4.5 range.

dschneiderch commented 3 years ago

oh good sleuthing! sounds like we should fix as if >=3.4.7 is correct.

I'm looking at the camera tech docs for my rgb camera and it is list as BayerGR8 as the format even though I was using BG. I don't remember if I looked this up and didn't understand why it didn't work (or just had no clue more generally) but it seems we should change the default in readbayer() to 'GR' to maintain the same visual behavior. I can test it this afternoon.

thanks for your help!

dschneiderch commented 3 years ago

Ok, this is making sense. If I convert the color in 3.4.7 with COLOR_BayerRG2BGR (instead of BG2BGR) then I get the correct band order. that is, pcv.plot_image() shows true RGB. The difference is the rotation is off 90. In 3.4.7 the image is 90 ccw from what I was getting with 3.4.2. I can fix my roi locations if we think this is resolved. i'd suggest changing the default readbayer() to 'RG' so it is as we meant and setting the min opencv to 3.4.7.

nfahlgren commented 3 years ago

Awesome! Changing the default and adding the version restriction to our requirements makes sense to me too.