Leaf ID number ordering

[Maryan21]

Hello, My name is Maryan; a student at the University of Minnesota. I work in the Cory Hirsch lab and now I am working with the Bellwether phenotyping data, specifically the leaf number and angle part. I am trying to understand the pcv.morphology.segment_id function and the order of its counting. I am trying to figure out a way to keep the leave ordering consistent so that I can track each leave through images that were taken on different dates.

[HaleySchuhl]

Regarding segment_id ordering. This is currently just using the order that OpenCV puts segments into. The way these are ordered are by y-coordinate, but not in the biologically relevant way that a human might order leaves based on development age.

I think we should set up a video call to discuss the proposed methods for resolving this.

[Maryan21]

Hello,

Thank you for your response, and yes I would like to set up a video call to discuss the segment_id ordering. Currently, I am in the middle of my final weeks so I am not available this week; however, I am free next week except Monday. I can meet anytime during the day from 9am-4pm all weekdays. Thank you again for your response; I hope to meet you soon.

Thanks, Maryan Said.

On Tue, Dec 13, 2022 at 2:16 PM Haley Schuhl @.***> wrote:

Regarding segment_id ordering. This is currently just using the order that OpenCV puts segments into. The way these are ordered are by y-coordinate, but not in the biologically relevant way that a human might order leaves based on development age.

I think we should set up a video call to discuss the proposed methods for resolving this.

[image: image] https://user-images.githubusercontent.com/44006936/207434431-8dd36d34-9cc4-4397-80a5-b465655f04f7.png

— Reply to this email directly, view it on GitHub https://github.com/danforthcenter/plantcv/issues/989#issuecomment-1349630474, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVX2K22E4VVWJRU55MIOWFLWNDKQXANCNFSM6AAAAAASZTMLI4 . You are receiving this because you authored the thread.Message ID: @.***>

[HaleySchuhl]

Hi @Maryan21 ,

We are back on track with weekly PlantCV developers meetings. We meet on Thursdays at 10AM Central; would this time work for you in the next few weeks? Otherwise we can find a different time to chat based on your 2023 availability!

[Maryan21]

Hello,

I am currently on a vacation outside of the country. I will be back on track by January the 15th. From then on, I will be available on Thursdays from 11:30 a.m to 2:00 p.m. If the developers' meeting concludes before then, I am available on Monday, Wednesday, and Friday mornings from 9 a.m. to 2 p.m.

Thank you, Maryan Said.

On Thu, Jan 5, 2023 at 10:37 PM Haley Schuhl @.***> wrote:

Hi @Maryan21 https://github.com/Maryan21 ,

We are back on track with weekly PlantCV developers meetings. We meet on Thursdays at 10AM Central; would this time work for you in the next few weeks? Otherwise we can find a different time to chat based on your 2023 availability!

— Reply to this email directly, view it on GitHub https://github.com/danforthcenter/plantcv/issues/989#issuecomment-1372591039, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVX2K2ZTEV3FBUW45T4CE3TWQ4IG5ANCNFSM6AAAAAASZTMLI4 . You are receiving this because you were mentioned.Message ID: @.***>

[HaleySchuhl]

Awesome, how about Wednesday the 18th at 9am-10am?

[Maryan21]

Yes, that works for me.

On Thu, Jan 12, 2023 at 5:56 PM Haley Schuhl @.***> wrote:

Awesome, how about Wednesday the 18th at 9am-10am?

— Reply to this email directly, view it on GitHub https://github.com/danforthcenter/plantcv/issues/989#issuecomment-1380719353, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVX2K257RQILX4F7CNGYAZTWSAZU5ANCNFSM6AAAAAASZTMLI4 . You are receiving this because you were mentioned.Message ID: @.***>

[Maryan21]

Hello,

Would you like me to schedule a Zoom meeting for 9 a.m. tomorrow? If so, do you want me to include anyone else or just us?

Thank you, Maryan Said.

[HaleySchuhl]

@Maryan21 Yes, could you schedule a zoom call for tomorrow? I have already invited Katie Murphy, Jorge, Noah, and Josh. If you email me the invite I can fwd it along :D

[Maryan21]

Ok so, I just scheduled a meeting, I was not sure which email to add as a guest; just wondering if the invitation came through? Also, I made the meeting using my school email; so it is a different email address.

On Tue, Jan 17, 2023 at 12:42 PM Haley Schuhl @.***> wrote:

@Maryan21 https://github.com/Maryan21 Yes, could you schedule a zoom call for tomorrow? I have already invited Katie Murphy, Jorge, Noah, and Josh. If you email me the invite I can fwd it along :D

— Reply to this email directly, view it on GitHub https://github.com/danforthcenter/plantcv/issues/989#issuecomment-1385868389, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVX2K26YQXSQQH57OYR4MQTWS3RYZANCNFSM6AAAAAASZTMLI4 . You are receiving this because you were mentioned.Message ID: @.***>

[HaleySchuhl]

I'm starting to think our proposed changes to enable 1:1 between leaves and branch points would be better done in a new function. I started out testing how things would play out if I updated segment_sort but the method I had in mind would involve deeply nested for loops. It feels safer to introduce more complexity for researchers that will do the downstream work of leaf ID matching.

So far I have just updated segment_sort so the number of "tips" matches the number of leaf segments now.

[HaleySchuhl]

import cv2
import numpy as np
from plantcv.plantcv import dilate
from plantcv.plantcv import params
from plantcv.plantcv import outputs
from plantcv.plantcv import find_objects
from plantcv.plantcv.morphology import find_branch_pts
from plantcv.plantcv import logical_and

params.debug = None
skel_img = skeleton
leaf_objects = secondary_objects
branch_img = find_branch_pts(skel_img)
branch_img = dilate(branch_img, 5, 1) #trial and error showed that ksize needs to be 5 to ID all 6 example leaves) 

branch_objects, _ = find_objects(branch_img, branch_img)
ordered_branch_pts = [] #empty list with ordered branch points to get stored

for i, leaf in enumerate(leaf_objects):
    segment_plot = np.zeros(skel_img.shape[:2], np.uint8)
    cv2.drawContours(segment_plot, leaf_objects, i, 255, 1, lineType=8)

    for j, branch_pt in enumerate(branch_objects):
        branch_pt_plot = np.zeros(skel_img.shape[:2], np.uint8)
        cv2.drawContours(branch_pt_plot, branch_objects, j, 255, 1, lineType=8)
        overlap_img = logical_and(segment_plot, branch_pt_plot)
        if np.sum(overlap_img) > 0:
            print("we have found a match!")
            ordered_branch_pts.append(branch_pt)
branch_labels = list(range(0, len(ordered_branch_pts)-1)
outputs.add_observation(sample="default", variable='ordered_branch_pts',
                            trait='ordered list of branch-point coordinates identified from a skeleton',
                            method='plantcv.plantcv.morphology.sort_branch_pts', scale='pixels', datatype=list,
                            value=ordered_branch_pts, label=branch_labels)

[nfahlgren]

How about this? We don't need to order them (yet), just pair them 1:1 with leaves

import cv2
import numpy as np
from plantcv.plantcv import dilate
from plantcv.plantcv import params
from plantcv.plantcv import outputs
from plantcv.plantcv import find_objects
from plantcv.plantcv.morphology import find_branch_pts
from plantcv.plantcv import logical_and

params.debug = None
skel_img = skeleton
leaf_objects = secondary_objects
branch_img = find_branch_pts(skel_img)
branch_img = dilate(branch_img, 5, 1) #trial and error showed that ksize needs to be 5 to ID all 6 example leaves) 

branch_objects, _ = find_objects(branch_img, branch_img)
ordered_branch_pts = [] #empty list with ordered branch points to get stored

# Make a mask for each leaf segment
leaf_masks = []
for i in range(len(leaf_objects)):
    segment_plot = np.zeros(skel_img.shape[:2], np.uint8)
    cv2.drawContours(segment_plot, leaf_objects, i, 255, 1, lineType=8)
    leaf_masks.append(segment_plot)

# Make a mask for each branch point
branch_masks = []
for i in range(len(branch_objects)):
    segment_plot = np.zeros(skel_img.shape[:2], np.uint8)
    cv2.drawContours(segment_plot, branch_objects, i, 255, 1, lineType=8)
    branch_masks.append(segment_plot)

# Match branch points to leaf segments
for leaf_id, leaf_mask in enumerate(leaf_masks):
    for branch_id, branch_mask in enumerate(branch_masks):
        overlap_img = logical_and(leaf_mask, branch_mask)
        if np.sum(overlap_img) > 0:
            ordered_branch_pts.append(outputs.observations["default"]["branch_pts"]["value"][branch_id])

# Overwrite original branch point values
outputs.observations["default"]["branch_pts"]["value"] = ordered_branch_pts
outputs.observations["default"]["branch_pts"]["label"] = list(range(0, len(ordered_branch_pts))

[HaleySchuhl]

Hmm it makes sense to me but when I try it out on my test image I get this error "TypeError: 'Outputs' object is not subscriptable" .

It's not the first time I've gotten that error but I can't remember how/if I got around it.

[nfahlgren]

It should have been outputs.observations etc. We would also need to deal with different sample names rather than hardcoding "default"

[HaleySchuhl]

Agreed, that's part of my argument for it being it's own function. I'd also need some convincing why we should overwrite the original values rather than storing them into a new variable.

[nfahlgren]

Maybe find_tips and find_branch_pts could become private functions and a label input could be added to segment_sort, where segment_sort would run the other two?

[HaleySchuhl]

Ok! @Maryan21 you should have an email in your university inbox regarding access for Slack! Once you're set up feel free to move the conversation there for ease.

The functionality that Noah has proposed (I've wrapped it up in a function below) ought to pair leaves:branch points exactly 1:1 but we're hoping that you can test it out on a subset of your images to let us know how it goes with your dataset. I would be most interested in the full time series of just a handful of replicates and we can refine the "leaf and branch point matching" function based on situations that might cause our logic to break.

import cv2
import numpy as np
from plantcv.plantcv import dilate
from plantcv.plantcv import params
from plantcv.plantcv import outputs
from plantcv.plantcv import find_objects
from plantcv.plantcv.morphology import find_branch_pts
from plantcv.plantcv import logical_and

def pair_leaf2branch_pts(skel_img, leaf_objects)

    # Store debug
    debug = params.debug
    params.debug = None

    branch_img = find_branch_pts(skel_img)

    # THIS part might need adjustment depending on image resolution?? 
    #trial and error showed that ksize needs to be 5 to ID all 6 example leaves) 
    branch_img = dilate(branch_img, 5, 1) 

    branch_objects, _ = find_objects(branch_img, branch_img)
    ordered_branch_pts = [] #empty list with ordered branch points to get stored

    # Make a mask for each leaf segment
    leaf_masks = []
    for i in range(len(leaf_objects)):
        segment_plot = np.zeros(skel_img.shape[:2], np.uint8)
        cv2.drawContours(segment_plot, leaf_objects, i, 255, 1, lineType=8)
        leaf_masks.append(segment_plot)

    # Make a mask for each branch point
    branch_masks = []
    for i in range(len(branch_objects)):
        segment_plot = np.zeros(skel_img.shape[:2], np.uint8)
        cv2.drawContours(segment_plot, branch_objects, i, 255, 1, lineType=8)
        branch_masks.append(segment_plot)

    # Match branch points to leaf segments
    for leaf_id, leaf_mask in enumerate(leaf_masks):
        for branch_id, branch_mask in enumerate(branch_masks):
            overlap_img = logical_and(leaf_mask, branch_mask)
            if np.sum(overlap_img) > 0:
                ordered_branch_pts.append(outputs.observations["default"]["branch_pts"]["value"][branch_id])

    # Reset debug mode
    params.debug = debug

    # Overwrite original branch point values
    outputs.observations["default"]["branch_pts"]["value"] = ordered_branch_pts
    outputs.observations["default"]["branch_pts"]["label"] = list(range(0, len(ordered_branch_pts))