slows down when there are lots of labels in a video sequence

[jascase901]

When there are a lot of labels in your video sequence, drawing boxes, moving boxes, and playing the video slows down.

every time you do one of these operations eva loops through all our bounding boxes for every annotation in your video sequence, and sets the box to hidden if its outside the frame, then calls set bounds on the rect. The math involved in set bounds can slow down the entire system if there are a lot of rectangles.

see pull request #14 for a potential fix.

[manishs17]

@jascase901 , Thanks for sharing the bug and suggesting the fix. Just to verify

How many images and no of bounding boxes per image did you had in your data set ? also dimension of each image ?

Thanks

[jascase901]

@manishs17 Thanks for working with me.

I spent a little more time digging into the issue. And it turns out reproducing it is a bit more complicated then just adding labels to a video sequence. While adding labels, via the track function does slow it down. The main issue is adding new chains of labels.

Heres what I ended up doing to come to this conclusion.

1. )I modified the code to time how long it takes to draw annotations

   //in static/player.js
    drawOnscreenAnnotations() {
        //hack to stop drawing the annotation when we are with frame player
        if (this.view.video.currentTime-this.offset >= this.view.video.duration) {
            this.view.video.pause();
        }
        var time_a = performance.now()//jh added
         for (let {annotation, rect} of this.annotationRectBindings) {
            this.drawAnnotationOnRect(annotation, rect);
        }
        var time_b = performance.now()//jh added
        console.log(time_b-time_a)//jh added
    }

2.) I applied my merge request. The whole function I modified is here.

//static/player.js
 drawAnnotationOnRect(annotation, rect) {
        if (this.metrics.annotationsStartTime == null) {
            this.metrics.annotationsStartTime = Date.now();
            // force the keyboard shortcuts to work within an iframe
            window.focus();
        }

        var frameNo = this.view.video.currentTime;

        var {bounds, prevIndex, nextIndex, closestIndex, continueInterpolation, state} = annotation.getFrameAtTime(frameNo, this.isImageSequence);
        /* singlekeyframe determines whether we show or hide the object
         we want to hide if:
            - the very first frame object is in the future (nextIndex == 0 && closestIndex is null)
            - we're after the last frame and that last frame was marked as continueInterpolation false */
        var appear_params = {
            real: closestIndex != null,
            selected: this.selectedAnnotation === annotation && !rect.locked,
            singlekeyframe: continueInterpolation && !(nextIndex == 0 && closestIndex === null)
        }

        rect.appear(appear_params);

        //only set bounds when annotation is visible.
        if (!appear_params.real) return;

        // Don't mess up our drag
        if (rect.isBeingDragged()) return;
        rect.bounds = bounds;
    }

The meat of the function is this line if (!appear_params.real) return;

3.) I generated 3000 images of circles to test. In my test case the video sequence swaps between these two images

Code to generate test images is here. draw_circles.txt

4.) This step is time consuming. But I labled every 100 frames, until about frame 1200

This gives me 649 annotations

6.) Then I trigger a redraw event. either draw a rectangle, move a rectangle, go to another frame, or hit f5, with my console open and time how long it takes to redraw all the rectangles in the image.

With my early stop

//only set bounds when annotation is visible.
if (!appear_params.real) return;

without my early stop, where we go and set bounding boxes for every annotation

//only set bounds when annotation is visible.
// if (!appear_params.real) return;

[manishs17]

while annotating the above test set, did you annotate each object in each frame independently ? or you used a tracker to annotate objects in multiple frames at once. This is because the inherited design follows a track, multiple annotations become part of same annotation array, that's y this problem arises.

[jascase901]

I didn't use the tracker. When I tried to use the tracker the problem didn't come up. I think this is because

inherited design follows a track, multiple annotations become part of same annotation array, that's y this problem arises.

In my real example I was labeling vehicles on a small part of a busy street. New vehicles were constantly leaving and entering the area I was labeling. Which caused new annotations to be added to the annotation array.