How to deal with crowd labels?

[tomjerrygithub]

my dataset has some crowd box which has no single box in it. (can‘t modify these crowd box to add single object box) i want to modify the loss to setting those crowd boxes loss to zero and keep the classify loss。 how to deal with this problem？ thx~

[AlexeyAB]

I do not understand, can you explain in more detail and show examples of what is and what you want?

[tomjerrygithub]

Hi Alexey ： Thank you for your reply！

  Now, i have a dataset which has 365 categories objects  and 630,000 pictures。 Some image has crowd label box，such as ：

     Pic 1： All the people has just one box for crowd detect task。

     Pic 2：Four sneakers just one box。each one sneaker has no box。

 I want to train  a object detect model  that can detect single object（single person，single sneaker） ，i can not  re-annotation these images（huge number images）。
 I've tried some methods：
 a、Discard all crowd box ，that is not a good idea；
 b、discard all images include a crowd box， that is not a good idea too；
 c、Is it possible to retain classified loss without calculating box loss? so，i can  use this dataset to train my model ？ 

 I don't know if it's clear. Thank you very much~

At 2019-05-08 02:54:16, "Alexey" notifications@github.com wrote:

I do not understand, can you explain in more detail and show examples of what is and what you want?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[tomjerrygithub]

I do not understand, can you explain in more detail and show examples of what is and what you want?

[AlexeyAB]

 c、Is it possible to retain classified loss without calculating box loss? so，i can  use this dataset to train my model ？

I don't understand how can it help you?

---

 b、discard all images include a crowd box， that is not a good idea too；

I think it is the best idea in your case.

[tomjerrygithub]

 c、Is it possible to retain classified loss without calculating box loss? so，i can  use this dataset to train my model ？

I don't understand how can it help you?

 b、discard all images include a crowd box， that is not a good idea too；

I think it is the best idea in your case.

thank you , but if i discard all images including crowd box half more images gone... so i wander if add crowd label in the
train.txt as xxxxx0.jpg

xxxxx0.jpg 0 x y w h crowd 1 x y w h
2 x y w h ... if the box is 'crowd' , when calculate the loss only calculate the classified loss but the box loss.

[AlexeyAB]

xxxxx0.jpg 0 x y w h crowd 1 x y w h 2 x y w h ... if the box is 'crowd' , when calculate the loss only calculate the classified loss but the box loss.

classified loss will be calculated for which (final activation) cells 13x13/26x26/52x52 and which of 3 anchors? And for which of 3 [yolo]-layers? Usually it is based on IoU(truth, detected_bbox)

[tomjerrygithub]

xxxxx0.jpg 0 x y w h crowd 1 x y w h 2 x y w h ... if the box is 'crowd' , when calculate the loss only calculate the classified loss but the box loss.

classified loss will be calculated for which (final activation) cells 13x13/26x26/52x52 and which of 3 anchors? And for which of 3 [yolo]-layers? Usually it is based on IoU(truth, detected_bbox)

Thanks!
Alexey , how can i change the way of calculating loss. For example, when calculate the crowd box, ignore it.

[tomjerrygithub]

xxxxx0.jpg 0 x y w h crowd 1 x y w h 2 x y w h ... if the box is 'crowd' , when calculate the loss only calculate the classified loss but the box loss.

classified loss will be calculated for which (final activation) cells 13x13/26x26/52x52 and which of 3 anchors? And for which of 3 [yolo]-layers? Usually it is based on IoU(truth, detected_bbox)

Or How to ignore some specific ground truth (not by size but i labebed it 'crowd')

[AlexeyAB]

@tomjerrygithub Change this line: https://github.com/AlexeyAB/darknet/blob/cce34712f6928495f1fbc5d69332162fc23491b9/src/data.c#L194

to these:

char crowd_buff[100];
while(fscanf(file, "%d %f %f %f %f %s", &id, &x, &y, &w, &h, crowd_buff) >= 5){
    if(strcmp(crowd_buff, "crowd") == 0) continue; 

so if there will be parameters more or less than 5, then this

[tomjerrygithub]

@tomjerrygithub Change this line:

https://github.com/AlexeyAB/darknet/blob/cce34712f6928495f1fbc5d69332162fc23491b9/src/data.c#L194

to these:

char crowd_buff[100];
while(fscanf(file, "%d %f %f %f %f %s", &id, &x, &y, &w, &h, crowd_buff) >= 5){
    if(strcmp(crowd_buff, "crowd") == 0) continue; 

so if there will be parameters more or less than 5, then this

Thanks a lot，i will try.

[tomjerrygithub]

@tomjerrygithub Change this line:

https://github.com/AlexeyAB/darknet/blob/cce34712f6928495f1fbc5d69332162fc23491b9/src/data.c#L194

to these:

char crowd_buff[100];
while(fscanf(file, "%d %f %f %f %f %s", &id, &x, &y, &w, &h, crowd_buff) >= 5){
    if(strcmp(crowd_buff, "crowd") == 0) continue; 

so if there will be parameters more or less than 5, then this

Sorry , if just ignore the crowd box, i can do it when generating the trainning txt file.
i mean when calculate the loss with crowd box, set it zero.

[khasmamad99]

I have a similar issue, I don't know how to deal with it. Were you able to come up with a solution?

I think what @tomjerrygithub means is, if an individual person is detected inside a truth ground which is labeled as crowd, loss function is going to penalize the detection as false positive, since that area belongs to another class (crowd class in this case). Is there a way to avoid that?

To visualize the problem, imagine a person indicated with the purple box is detected.

That will be interpreted as a false positive since that area belongs to another class indicated by the red box.

[hemp110]

@AlexeyAB Actually I added the feature of multiple ignore labels myself and it did improve mAP in my case of person detection. Just several changes, hope you can add this feature to help others.

darknet.h

struct layer{
//...
    int *ignore_label;
    int num_ignore_label;
//...
}

layer.c

void free_layer(layer l)
{
//...
    if (l.ignore_label)       free(l.ignore_label);
//...
}

parser.c

layer parse_yolo(list *options, size_params params)
{
//...
    a = option_find_str(options, "ignore_label", 0);
    if(a) {
        printf("ignore_label=%s\n", a);
        l.ignore_label = parse_yolo_mask(a, &l.num_ignore_label);
        printf("num_ignore_label=%d\n", l.num_ignore_label);
    }
    return l;
}

yolo_layer.c

layer make_yolo_layer(int batch, int w, int h, int n, int total, int *mask, int classes, int max_boxes)
{
//...
    l.ignore_label = 0;
    l.num_ignore_label = 0;
//...
}
void forward_yolo_layer(const layer l, network_state state)
{
//...
           int mask_n = int_index(l.mask, best_n, l.n);
            if(mask_n >= 0){
                int class_id = state.truth[t*(4 + 1) + b*l.truths + 4];
                if (l.map) class_id = l.map[class_id];
                int ll=0, ignore = 0;
                for (ll=0;ll<l.num_ignore_label;ll++)
                    if (class_id == l.ignore_label[ll]) {
                    printf("ignoring image %d bbox %d with label %d\n",b,t,class_id);
                    ignore=1;}
                if (ignore) continue;
//...
}

yolov3.cfg

[yolo]
#...
#support multiple ingore labels
ingore_label=0,1
#...