libvmaf ERROR vmaf_predict_score_at_index()

[Yassine-Bouksaier]

I'll be scoring between each frame of 2 YUV raw videos. When I select 'vmaf_4k_v0.6.1.json' as the model, the scores are calculated. However, when I use version=vmaf_4k_v0.6.1 as the model, only the first score is calculated, and there's an error with the second score:

VMAF version b24c3d68 1 frame ⢀⠀ 0.00 FPS vmaf_4k_v0.6.1: 100.000000 2 frames ⡀⠀ 0.00 FPS libvmaf ERROR vmaf_predict_score_at_index(): no feature 'VMAF_integer_feature_motion2_score' at index1 Frame 2: issue generating pooled VMAF score

[nilfm99]

Hi @Yassine-Bouksaier, that is unexpected. Could you please share the sources and command you used so we can investigate this on our end?

[Yassine-Bouksaier]

for (picture_index = 0;; picture_index++) { if (c.frame_cnt && picture_index >= c.frame_cnt) // 0 break;

    VmafPicture pic_ref, pic_dist;
    int ret1 = fetch_picture(&vid_ref, &pic_ref, common_bitdepth);
    int ret2 = fetch_picture(&vid_dist, &pic_dist, common_bitdepth);

    if (ret1 && ret2)
    {
        break;
    }
    else if (ret1 < 0 || ret2 < 0)
    {
        fprintf(stderr, "\nproblem while reading pictures\n");
        break;
    }
    else if (ret1)
    {
        fprintf(stderr, "\n\"%s\" ended before \"%s\".\n",
                c.path_ref, c.path_dist);
        int err = vmaf_picture_unref(&pic_dist);
        if (err)
            fprintf(stderr, "\nproblem during vmaf_picture_unref\n");
        break;
    }
    else if (ret2)
    {
        fprintf(stderr, "\n\"%s\" ended before \"%s\".\n",
                c.path_dist, c.path_ref);
        int err = vmaf_picture_unref(&pic_ref);
        if (err)
            fprintf(stderr, "\nproblem during vmaf_picture_unref\n");
        break;
    }

    if (istty && !c.quiet)
    {
        if (picture_index > 0 && !(picture_index % 10))
        {
            fps = (picture_index + 1) /
                  (((float)clock() - t0) / CLOCKS_PER_SEC);
        }

        fprintf(stderr, "\r%d frame%s %s %.2f FPS\033[K",
                picture_index + 1, picture_index ? "s" : " ",
                spinner[picture_index % spinner_length], fps);
        fflush(stderr);
    }

    err = vmaf_read_pictures(vmaf, &pic_ref, &pic_dist, picture_index);
    if (err)
    {
        fprintf(stderr, "\nproblem reading pictures\n");
        break;
    }
    if (!c.no_prediction)
    {
        for (unsigned i = 0; i < c.model_cnt; i++)
        {
            err = vmaf_score_at_index(vmaf, model[i], &vmaf_score, picture_index);
            if (err)
            {
                fprintf(stderr, "frame %d : problem generating pooled VMAF score\n", picture_index + 1);
                return -1;
            }
            if (istty && (!c.quiet || !c.output_path))
            {
                fprintf(stderr, "%s: %f\n",
                        c.model_config[i].version ? c.model_config[i].version : c.model_config[i].path,
                        vmaf_score);
            }
        }
        for (unsigned i = 0; i < model_collection_cnt; i++)
        {
            VmafModelCollectionScore score = {0};
            for (int j = 0; j < picture_index; j++)
            {
                err = vmaf_score_at_index_model_collection(vmaf, model_collection[i], &score, j);
                if (err)
                {
                    fprintf(stderr, "problem generating pooled VMAF score\n");
                    return -1;
                }

                switch (score.type)
                {
                case VMAF_MODEL_COLLECTION_SCORE_BOOTSTRAP:
                    if (istty && (!c.quiet || !c.output_path))
                    {
                        fprintf(stderr, "%s: %f, ci.p95: [%f, %f], stddev: %f\n",
                                model_collection_label[i],
                                score.bootstrap.bagging_score, score.bootstrap.ci.p95.lo,
                                score.bootstrap.ci.p95.hi,
                                score.bootstrap.stddev);
                    }
                    break;
                default:
                    break;
                }
            }
        }
    }
} // command i use is : ./quality -r /home/yassine/quality/pipe5 -d /home/yassine/quality/pipe6 --model version=vmaf_4k_v0.6.1

[nilfm99]

Sorry, I don't understand what that code from vmaf.c has to do with the issue.

On the command you posted below: I tried the following:

➜  vmaf git:(master) ✗ vmaf -r ../inputs/ref5.y4m -d ../inputs/dis5.y4m
VMAF version 3.0.0
5 frames ⡂⠀ 0.00 FPS
vmaf_v0.6.1: 7.341517
➜  vmaf git:(master) ✗ vmaf -r ../inputs/ref5.y4m -d ../inputs/dis5.y4m --model version=vmaf_4k_v0.6.1
VMAF version 3.0.0
5 frames ⡂⠀ 0.00 FPS
vmaf_4k_v0.6.1: 28.612129

and both succeeded. It would be useful to know:

1. what is ./quality - is it an alias for the vmaf executable?
2. what is in /home/yassine/quality/pipe5 and /home/yassine/quality/pipe6 - how do you generate those files?

[Yassine-Bouksaier]

1. ./quality is an executable for a program named quality.c, which is very similar to vmaf.c but calculates the score between every two frames of the same index. The score calculation loop occurs right after the extraction of a single image from the video, as indicated in the previous code section.
2. /home/yassine/quality/pipe5 and /home/yassine/quality/pipe6 are raw videos in the yuv4mpeg format generated with ffmpeg.

[nilfm99]

Sorry I missed this. If you could provide the input files themselves, or anything that could help reproduce this on our end, that would be helpful.

./quality is an executable for a program named quality.c, which is very similar to vmaf.c but calculates the score between every two frames of the same index.

I see, so the main difference between this and vmaf.c is that you want to get the quality results for each frame as it is computed, rather than at the end, is that correct?

[nilfm99]

Hi @Yassine-Bouksaier, I was able to reproduce your issue (here is a branch that reproduces it with a simple change), and the root of it is that you are requesting vmaf_score_at_index for a picture index that cannot be computed yet (each frame has a temporal dependency on the next one due to the motion2 feature, see here). What you can do is delay the vmaf_score_at_index call by one frame: when you have just read frame 1, request the score for frame 0, and so on. When you read the last frame, and you call vmaf_read_pictures with two NULL inputs, the feature extractor will be flushed and you can get the final score. This will allow you to get the scores almost in real time, with just a single frame delay.

cc @kylophone and @christosbampis for any more observations

[nilfm99]

Closing this issue for now, please do not hesitate to reopen if you run into more problems.