ADE-split.ipynb question

[JIA-HONG-CHU]

Hello,I want to ask about the logic of cutting data, how a photo puts it on the incremental learning steps. Sorry to bother you.Tks.

# This is the code who actually make the split.
all_added = set()

imgs={}
others = {}
every=set()
added = [0 for i in range(151)]
for i in range(sets+1):
    others[i]=set()
    imgs[i]=set()

# This is the most imporant function. 
# It computes the score of the image to be assinged to a class
def score(ratio,imgs,expected,minval=False,w1=120,w2=1000,w3=1 ):
    return (1.-ratio)*w1+w3*minval

for i in imgs_to_cls.keys(): # loop for every image in the dataset
    # img_counts = num of images per class for the class in the actual image  
    img_counts = [class_members[j] for j in imgs_to_cls[i]] 
    # ratio = images assigned to the class / total number of images for the class  
    ratios = [(added[j]+0.0)/class_members[j] for j in imgs_to_cls[i]]
    added_counts = [added[j] for j in imgs_to_cls[i]]
    # assignments = set of each class in the image
    assignments = [map_class_to_set[j] for j in imgs_to_cls[i]]

    scores = [score(ratios[c],added_counts[c],class_members[j],1./(img_counts[c]/sum(img_counts))) for c,j in enumerate(imgs_to_cls[i])]

    # take the highest scorer class
    cl = scores.index(max(scores))
    # take the set of the higher scorer class
    a=assignments[cl]

    # add the image to the step images
    imgs[a].update([i])
    for j,ac in enumerate(assignments):
        if ac==a:
            # increment the number of images assigned to the classes in the current image contained in the same step.
            added[imgs_to_cls[i][j]] += 1

for i in range(1,len(added)):        
        assignment = map_class_to_set[i]
        ratios = [len(set(idxs[i]).intersection(imgs[j]))/class_members[i] for j in range(0,sets+1)]
        if ratios[assignment]<0.5 or (ratios[assignment]<1. and added[i]<100):
            print(i,ratios[assignment], sum(ratios[1:]), class_members[i], len(set(idxs[i]).intersection(imgs[assignment])))
s=0
for i in range(sets+1):
    print(len(imgs[i]))
    s+=len(imgs[i])

print(s)

[fcdl94]

Hi @JIA-HONG-CHU. Thanks for your interest. The goal is to split the dataset (ADE in our case) into different splits, one for each subset of classes. However, in ADE neither the overlapped or disjoint protocol can be employed due to too many cooccurrences.

In this script, we divided the dataset following this algorithm. We iterate on every image of the dataset and we: 1) we compute how many images are available for a certain class. 2) we compute how many images we already assigned to that class and we compute a ratio added/available. 3) we compute a score for each class which represents the importance of assigning that image to the class. 4) we assign the image to the max scoring class and we update the sets.

Overall, the idea is that we should divide images to maintain enough images for each class. Let make an example to explain this: assume to have a class A in step 1 (let say bird) that is contained in few images (e.g. 100) and another class B in step 0 (person) that is contained in many (e.g. 1000). Assume also that in all images of A there's also B (so, in every image where bird is contained there's also person). If when dividing the dataset we randomly split the images, the resulting split may include all the A class images in the step 0 (since they also contain class B), and none of A in the step 1 (thus, we will have NO annotation for it in the whole training). Differently, our heuristic assigns the images based on the number of images and how many images are left. Doing so, when encountering an image containing both class A and B, we will assign it in step 1, enabling us to learn it properly.

I hope that I gave you the general idea. If you aimed for a different answer, please let me know. Best.

[JIA-HONG-CHU]

Tks for reply. After I read your code, my idea is that the number of newly added category photos of this incremental training step cannot be less than 50% of the total number of photos in that category or less than 100 photos (Except for the fact that there are very few photos in this category) . Is it also right?

[fcdl94]

Well, this is not enforced by the algorithm. The line you see there was just a check that we had enough images in each step... Ideally, you want to assign more than 50% of the images and more than 100 images, but there may be cases where it is not possible. In our case, if I remember correctly (sorry, but already 2 years had passed) we manually discarded such splits.

[JIA-HONG-CHU]

Thank you for your reply. I will close this issue.