Error while running landmark registration using RGB threshold method

[bansal-udita]

Hi Steven,

I have been trying different methods in patternize to characterise various colour morphs of a lizard species based on the throat colours. Some of the individuals have only a single colour along with brown, while others have two different colours. I basically want to compare these individuals to see if they form discrete groups. I also want to get the area of each colour in each individual's throat. I have been running into a few issues.

1. I am using the RGB threshold method in patternize as a first step towards trying to get the area of each colour. It runs smoothly for two colours but keeps giving me an error after few images for yellow. I used different RGB values for defining this yellow but it always stops in between with the following error

Error in if (all(map == FALSE)) { : missing value where TRUE/FALSE needed

Any idea why this might be happening?

2. I also wanted to clarify one concept. While using the RGB threshold method, does the colOffset option define the boundaries around the particular RGB values which should be extracted? What does a colOffset value of 0.10 indicate?

3. Is there a chance that when I run the process for two different RGB values, they could extract overlapping regions from the image? Does this depend on colOffset and is there a way to ensure that there is no overlap?

4. Right now I sample the RGB values in ImageJ and then use those as thresholds here. However, I wanted to try the sampleRGB function but I am unable to import a single image as a raster into R.

5. I tried k- means clustering and when I plot the heatmaps, it seems to have assigned a cluster or two to the background as well even if crop = TRUE. This is what the code looks like

rasterList_lanK_SCmale_6 <- patLanK(imageListSC_male, landmarkListSC_male, k = 6, crop = TRUE, res = 300, transformRef = "meanshape", transformType = "tps", adjustCoords = TRUE, plot = TRUE)

Any insights will be extremely helpful.

Many thanks Udita

[GinBay-Min]

Hi Steven and Udita,

I have similar issue about the RGB extraction by patlanRGB. I want to compare color pattern of many bracts of Bougainvillea althought lots of are single color and some are variegated. Anyway I landmarked a circle on the bract for them and tried to extract the RGB by patlanRGB and showed up error like:

IDlistb<-c('A15',

• 'B05',
• 'C11',
• 'C14') prepath<-'images' extension<-'.jpg' imageListb<-makeList(IDlistb,'image',prepath,extension) "sample 1 A15 added to list" "sample 2 B05 added to list" "sample 3 C11 added to list" "sample 4 C14 added to list"

prepath<-'landmarks' extension<-'_landmark.txt' landmarkListb<-makeList(IDlistb,'landmark',prepath,extension) "sample 1 A15 added to list" "sample 2 B05 added to list" "sample 3 C11 added to list" "sample 4 C14 added to list" RGB<-c(114,17,0) rasterList_lanRGBb <- patLanRGB(imageListb, landmarkListb, RGB,resampleFactor=3,transformRef="A15",

• colOffset = 0.1, crop = TRUE, res = 300, adjustCoords = TRUE, plot = 'stack') "sample 1 A15 added to array" "sample 2 B05 added to array" "sample 3 C11 added to array" "sample 4 C14 added to array" Error in if (all(map == FALSE)) { : missing value where TRUE/FALSE needed

  for the beginning, some warnings comed with the error :

  x-RGB: longer object length is not a multiple of shorter object length

  I thought it might caused by my low pixel picture ,so I change the pixel higher and the warnings didn't show up but error still.

By the way, I am a postgraduate student from China, I don't know whether my English is ok for you. I do appreciate for your help and the Patternize package, I want to use it in plants, it's really cool! Thanks!

GinBay

[StevenVB12]

Hi Udita,

Sorry you’re experiencing some difficulties.

1. Did you install patternize through GitHub? That might solve the issue. In case a color is not present, you should be getting a warning, not an error. That part might not be updated in CRAN yet.

install.packages("devtools")

library(devtools)

install_github("StevenVB12/patternize")

1. colOffset is a proportion of the colorspace (which goes up to 255), so 0.10 * 255 for R, G and B.

2. Yes, you might get overlap. You can calculate that with the colOffset. Alternatively, the Kmeans could help.

3. This should do it: sampleRGB(imageList[[1]]) , but could be a version thing again, which you can solve by installing through GitHub.

4. The GitHub version has improvements for the kmeans analysis.

first run image alignment only (with this function you can add a mask polygon ~outline. Everything outside of it will be NA. You can create a mask with setMask() and choose to have everything inside or outside of it removed (see, ‘inverse’ in alignLan()):

alignLan()

next run kmeans. You can choose to run this over all images combined. That makes sure that a color can be missing in one of the images (otherwise it will for example always force to find K clusters in an image).

patK()

Example:

imageList_aligned <- alignLan(imageList, landmarkList, transformRef = target, adjustCoords = TRUE,

                          plotTransformed = FALSE, resampleFactor = 5,  cartoonID = 'XXX,

                          maskOutline = list(mask1,mask2,mask3), inverse = list(FALSE,TRUE,TRUE))

out <- patK(imageList_aligned, kmeansOnAll = TRUE, maskToNA = 0)

I hope this helps. But please don’t hesitate to ask if you get stuck.

Best,

Steven

From: @.> Sent: Friday, May 7, 2021 12:24 AM To: @.> Cc: @.***> Subject: [StevenVB12/patternize] Error while running landmark registration using RGB threshold method (#21)

Hi Steven,

I have been trying different methods in patternize to characterise various colour morphs of a lizard species based on the throat colours. Some of the individuals have only a single colour along with brown, while others have two different colours. I basically want to compare these individuals to see if they form discrete groups. I also want to get the area of each colour in each individual's throat. I have been running into a few issues.

1. I am using the RGB threshold method in patternize as a first step towards trying to get the area of each colour. It runs smoothly for two colours but keeps giving me an error after few images for yellow. I used different RGB values for defining this yellow but it always stops in between with the following error

Error in if (all(map == FALSE)) { : missing value where TRUE/FALSE needed

Any idea why this might be happening?

1. I also wanted to clarify one concept. While using the RGB threshold method, does the colOffset option define the boundaries around the particular RGB values which should be extracted? What does a colOffset value of 0.10 indicate?
2. Is there a chance that when I run the process for two different RGB values, they could extract overlapping regions from the image? Does this depend on colOffset and is there a way to ensure that there is no overlap?
3. Right now I sample the RGB values in ImageJ and then use those as thresholds here. However, I wanted to try the sampleRGB function but I am unable to import a single image as a raster into R.
4. I tried k- means clustering and when I plot the heatmaps, it seems to have assigned a cluster or two to the background as well even if crop = TRUE. This is what the code looks like

rasterList_lanK_SCmale_6 <- patLanK(imageListSC_male, landmarkListSC_male, k = 6, crop = TRUE, res = 300, transformRef = "meanshape", transformType = "tps", adjustCoords = TRUE, plot = TRUE)

Any insights will be extremely helpful.

Many thanks Udita

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/StevenVB12/patternize/issues/21, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQOC4YFR3YDKTBI5HASH7DTMNTONANCNFSM44I3VY5Q.

[StevenVB12]

Hi Ginbay,

Nice meeting you!

Did you install patternize through GitHub? That may resolve the first issue. Also, are your images in .jpeg?

install.packages("devtools")

library(devtools)

install_github("StevenVB12/patternize")

Just as a tip. You might want to use a single target and use that single target for all your analysis. In that way every analysis will be transformed to the same reference and comparable (for subtracting and PCA).

For example:

target <- landmarkListb[[1]]

rasterList_lanRGBb <- patLanRGB(imageListb, landmarkListb, RGB,resampleFactor=3,transformRef=target, colOffset = 0.1, crop = TRUE, res = 300, adjustCoords = TRUE, plot = 'stack') Hope this helps. Let me know!

Best,

Steven

From: @.> Sent: Friday, May 7, 2021 5:27 AM To: @.> Cc: @.***> Subject: Re: [StevenVB12/patternize] Error while running landmark registration using RGB threshold method (#21)

Hi Steven and Udita,

I have similar issue about the RGB extraction by patlanRGB. I want to compare color pattern of many bracts of Bougainvillea althought lots of are single color and some are variegated. Anyway I landmarked a circle on the bract for them and tried to extract the RGB by patlanRGB and showed up error like:

IDlistb<-c('A15',

· 'B05',

*

· 'C11',

*

· 'C14')

*

prepath<-'images' extension<-'.jpg' imageListb<-makeList(IDlistb,'image',prepath,extension) "sample 1 A15 added to list" "sample 2 B05 added to list" "sample 3 C11 added to list" "sample 4 C14 added to list"

prepath<-'landmarks' extension<-'_landmark.txt' landmarkListb<-makeList(IDlistb,'landmark',prepath,extension) "sample 1 A15 added to list" "sample 2 B05 added to list" "sample 3 C11 added to list" "sample 4 C14 added to list" RGB<-c(114,17,0) rasterList_lanRGBb <- patLanRGB(imageListb, landmarkListb, RGB,resampleFactor=3,transformRef="A15",

· colOffset = 0.1, crop = TRUE, res = 300, adjustCoords = TRUE, plot = 'stack')

• "sample 1 A15 added to array" "sample 2 B05 added to array" "sample 3 C11 added to array" "sample 4 C14 added to array" Error in if (all(map == FALSE)) { : missing value where TRUE/FALSE needed

for the beginning, some warnings comed with the error : x-RGB: longer object length is not a multiple of shorter object length

I thought it might caused by my low pixel picture ,so I change the pixel higher and the warnings didn't show up but error still.

By the way, I am a postgraduate student from China, I don't know whether my English is ok for you. I do appreciate for your help and the Patternize package, I want to use it in plants, it's really cool! Thanks!

GinBay

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/StevenVB12/patternize/issues/21#issuecomment-834207920, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQOC44VQZKO22UQKQ2Y3ALTMOW77ANCNFSM44I3VY5Q.

[GinBay-Min]

Hi Steven, I am sorry I can't solve the issue even after changing the package through your github version. The error is the same : Error in if (all(map == FALSE)) { : missing value where TRUE/FALSE needed. It occurs to me maybe the reasons is the landmark I made which is a 15-points circle. Because the bract's color is single, the package could not identify the landmark boundary. So the error showed up? Here is the photos, Also I tried the single target and it didn't work. Any other insights will be extremely helpful ! Thanks!

GinBay

[bansal-udita]

Hi Steven,

Thank you so much for your quick and detailed response, that is extremely helpful!

1. I am using patternize from github, but just to make sure, I re-installed it and re-ran my code. The same error occurred.

2., 3., 4. Thank you so much, this clarifies a lot.

5. Ah, I did not know of these functions since I was referring to the CRAN pdf for the package usage. I will try this out, thank you so much. edit: I tried it out and still get one empty cluster (see attached image). Is it because of some mistake I may be making while masking (like one outline instead of multiple)? Is there a way to check the alignment of images?

my code: target <- landmarkList[[1]]

outline <- read.table("../Data/outline.txt", h = F) imageList_aligned <- alignLan(imageList, landmarkList, transformRef = target, adjustCoords = TRUE, plotTransformed = FALSE, resampleFactor = 5, cartoonID = 'SC-VM1', maskOutline = outline, refImage = raster::stack('SC-VM1'))

out <- patK(imageList_aligned, kmeansOnAll = TRUE, maskToNA = 0, maskOutline = outline)

summedRaster <- sumRaster(out, IDList, type = 'k')

plotHeat(summedRaster, IDList)

6. Should I be giving a list of outlines for masking or one outline works for all even if the images have lizards in different positions?

7. maskoutline command can be included at alignment step and patK. Which step is it preferable to include at?

8. Is it possible to define the cluster centers such that they don't change? Right now when I define it (using fixedCenters), they eventually change and become something else.

9. If kmeansOnAll = FALSE, then different cluster centers are chosen for each image or is just the first image used as the reference?

10. In case I increase k, I also get a warning as follows: Warning messages: Quick-TRANSfer stage steps exceeded maximum (= 170357900)

What does this indicate, anything to worry about?

11. You mentioned in one of your responses here that one can also use transformRef = target while doing landmark registration to make things comparable. Will that not be the case if we use "meanshape"?

12. I tried PCA for males and females of a population of lizards, but had run the k-means clustering separate for males and females like in the spider example where different populations were being compared. Does that not lead to separate cluster centers for the different groups being compared? How does the PCA work in such cases?

13. Is it possible to encode different colours for each individual data point in the PCA instead of attributing the same colour to a whole group for easy inter-individual comparison?

Apologies for the extensive questions and thank you so much for taking the time out to respond. Really appreciate the package and your help. Thank you!

Kind regards Udita

[StevenVB12]

Hi Ginbay,

I was able to reproduce the error and I think I fixed (the all function crashed when there were only NA or FALSE results in the map object, but worked when there were TRUE results. Not sure why that happens, but I fixed it). Can you reinstall the package through Github and give it a try again?

Steven

From: @.> Sent: Saturday, May 8, 2021 1:22 AM To: @.> Cc: Steven M. Van @.>; @.> Subject: Re: [StevenVB12/patternize] Error while running landmark registration using RGB threshold method (#21)

Hi Steven, I am sorry I can't solve the issue even after changing the package through your github version. The error is the same : Error in if (all(map == FALSE)) { : missing value where TRUE/FALSE needed. It occurs to me maybe the reasons is the landmark I made which is a 15-points circle. Because the bract's color is single, the package could not identify the landmark boundary. So the error showed up? Here is the photos,

Also I tried the single target and it didn't work. Any other insights will be extremely helpful ! Thanks!

GinBay

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/StevenVB12/patternize/issues/21#issuecomment-835108816, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQOC45KNNDGX2NIYS3WQUTTMTDBHANCNFSM44I3VY5Q.

[StevenVB12]

Hi Udita,

1. I was able to reproduce the error and I think I fixed (the all function crashed when there were only NA or FALSE results in the map object, but worked when there were TRUE results. Not sure why that happens, but I fixed it). Can you reinstall the package through Github and give it a try again?

5. What I usually do is run the function ones without storing the cluster centers and then storing the centers from that test run and keep using them. That way, they will always be the same order.

6. With meanshape, an average shape will be calculate from the samples in your list. From experience, I think that gets you in trouble when you want to compare different sets of samples (which will have a different meanshape). Therefore, if you use one fixed target, everything will be comparablecompatible at the end to run PCA and subtract rasters. The target should just be the landmarks of one of your samples.

7. Yes, you’ll have to know the order of the clusters and select the correct ones. Using the fixedCenters should help with that.

8. Try plotType = ‘labels’ I hope that works for you.

Steven

From: @.> Sent: Saturday, May 8, 2021 4:07 AM To: @.> Cc: Steven M. Van @.>; @.> Subject: Re: [StevenVB12/patternize] Error while running landmark registration using RGB threshold method (#21)

Hi Steven,

Thank you so much for your quick and detailed response, that is extremely helpful!

1. I am using patternize from github, but just to make sure, I re-installed it and re-ran my code. The same error occurred.

2., 3., 4. Thank you so much, this clarifies a lot.

1. Ah, I did not know of these functions since I was referring to the CRAN pdf for the package usage. I will try this out, thank you so much. Just one quick question, is there a way to know the RGB values for the cluster centers or define them by ourselves instead of letting the computer decide?
2. You mentioned in one of your responses here that one can also use transformRef = target while doing landmark registration to make things comparable. Will that not be the case if we use "meanshape"?
3. I tried PCA for males and females of a population of lizards, but had run the k-means clustering separate for males and females like in the spider example where different populations were being compared. Does that not lead to separate cluster centers for the different groups being compared? How does the PCA work in such cases?
4. Is it possible to encode different colours for each individual data point in the PCA instead of attributing the same colour to a whole group for easy inter-individual comparison?

Apologies for the extensive questions and thank you so much for taking the time out to respond. Really appreciate the package and your help.

Kind regards Udita

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/StevenVB12/patternize/issues/21#issuecomment-835194859, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQOC46MOTZLRKHTVDTD5Y3TMTWLZANCNFSM44I3VY5Q.

[StevenVB12]

Hi Udita,

kmeansOnAll =TRuE will combine all your image pixels (like creating one giant composite image) before running K-means and then search for k clusters over all your images. After that, it will go through each image separately and assign pixels to one of the k clusters. That ultimately means that if in one of your images a colour is not present, it will only assign k-1 clusters to that image.

KmeansOnAll=FALSE will run the kmeans clustering on one image at a time. That means if a colour is not present in that image, it will still force to assign k clusters. That could result in very different result for different sets of images.

Does that make sense? I would thus suggest to use kmeansOnAll in most circumstances.

There’s also patGMM. It works similarly as kmeans, to assign pixels to clusters, but it differs that kmeans will try to find groups of pixels with equal frequencies, while GMM should identify better clusters that are markedly different, but less frequent (like a small spot of black in your images).

Hope this clarifies it.

Best,

Steven

From: @.> Sent: Saturday, May 8, 2021 4:15 AM To: @.> Cc: Steven M. Van @.>; @.> Subject: Re: [StevenVB12/patternize] Error while running landmark registration using RGB threshold method (#21)

I just found that the new patK function has an option to specify cluster centers, that is extremely helpful and kind of answers point 5 and 7. However, it will still be good to know what happens when we don't specify manually.

Also, patK has one option: kmeansOnAll | Whether to perform the kmeans clusters on the combined set of pixels of all images first (default = FALSE).

What does this exactly do, sorry I was unable to understand.

Thank you again, Udita

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/StevenVB12/patternize/issues/21#issuecomment-835199566, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQOC46VCYBOTYYXUEYPQ4LTMTXIHANCNFSM44I3VY5Q.

[bansal-udita]

Hi Steven,

Thanks again for explaining everything in much detail. The error is sorted now, thank you! :) I will try this stuff, patGMM sounds more like what I need.

Sorry for the trouble, but I think you replied to my older versions of the comments. I had edited the one with multiple points. Could you please let me know about points 5-8 and 10 from the most updated version of that comment? Or I can put those here again?

Thank you so much for your time and effort.

Kind regards Udita

[bansal-udita]

Hi Steven,

I tried running patGMM but it doesn't seem to give the RGB values for cluster centers like patK does. Any way to know the cluster centers?

Thanks Udita

[StevenVB12]

Hi Udita,

sorry about not responding to all your questions earlier. I did not get the whole message through email it seems.

5. I think your code is working. The first plot just seems to identify the outer part of the dewlaps as a different color. Maybe try making your outline slightly smaller, so you make sure no background pixels are preserved.

6. One outline should work for all. If you would give multiple outlines (because the mask happens after the alignment), it would need to be for different areas in the image.

7. Preferred at alignment. But both should work.

8. The predefined cluster centers are only a start for the k-means algorithm. The final values will change a bit, but it just makes sure that the order of the color 'types/clusters' is always the same and comparable between separate analyses.

9. If kmeansOnAll = FALSE, the centers given to you will be for the first image only. If If kmeansOnAll = TRUE, the centers will be calculated over all the iages combined.

10. Is this with fixed startCenters? It might be because colors are very similar overall, and the algorithm has difficulty to converge on a solution.

Steven

[bansal-udita]

Hi Steven,

Thank you so much, I understand the process much better now. This has been extremely helpful.

The Warning message [Quick-TRANSfer stage steps exceeded maximum (= 170357900)] was without fixed startCenters, just in the cases where I increase k to 8-10.

Thank you again for your time and patience with all my queries. All the help is extremely appreciated.

Kind regards Udita

[StevenVB12]

Hi Udita,

as the GMM function is now, it does not output the cluster 'centroids' (in this case). I could add that option, but I can't add an option to fix them. There is the potential to add a seed number, so that for a single analysis the result would always bee the same (including order).

K 8-10 seems rather high to keep track of which cluster is which color.

Best,

Steven

[bansal-udita]

Hi Steven,

Ok, thank you so much. I understand now, thanks again for your time and all the wonderful explanations.

Kind regards Udita