VP9 Encoder crashes when --tune=ssim is set

[GoogleCodeExporter]

One of the recent commits in the VP9 master branch 
(http://git.chromium.org/webm/libvpx.git, SHA1 
d3a2e55af41ea85be57ec68870e1ccd81cb3020d) broke the VP9 encoder for 
--tune=ssim. It throws a segmentation fault after encoding 16 frames in the 
second pass. A week before, this was running perfectly fine. --tune=psnr still 
works, so this seems to be an issue confined to some ssim logic. I attached the 
console output ("Speicherzugriffsfehler" = German for segmentation fault).

Tried 2 machines (openSUSE 12.2 "Mantis" x64, Fedora 15 "Lovelock" x64), 
release and debug compilation, same behaviour.  Note that I needed to disable 
sse3, ssse3 and sse4_1 in order to compile and run it.

Here's the command line I used:

./vpxenc ~/mysequence.yuv --output=str.webm --width=800 --height=480 
--limit=200 --fps=30/1 --i420 --skip=0 --end-usage=vbr --passes=2 
--minsection-pct=5 --maxsection-pct=800 --tune=ssim --best --cpu-used=0 
--codec=vp9 --auto-alt-ref=1 --lag-in-frames=16 --kf-max-dist=360 
--kf-min-dist=0 --max-q=60 --min-q=0 --static-thresh=0 --threads=1 
--drop-frame=0 --buf-initial-sz=4000 --buf-optimal-sz=5000 --buf-sz=6000 
--resize-allowed=0 --verbose  --target-bitrate=4333

I noticed that --lag-in-frames has direct influence on the segfault, setting it 
to 10 makes vpxenc crash after 10 frames, not 16.

Additional information:
-------------------------

In detail, a segfault is thrown after about 16 frames in the second pass. GDB 
tells me:

VPX-27NovDebugVP9SSIMProblem Encoder [C/C++ Application]  
    /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vpxenc [19865] [cores: 2]  
        Thread [1] 19865 [core: 2] (Suspended : Signal : SIGSEGV:Segmentation fault)  
            vp9_fdct4x4_c() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_dct.c:89 0x538b92  
            vp9_encode_block_intra() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodemb.c:632 0x5564a7  
            foreach_transformed_block_in_plane() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/common/vp9_blockd.h:367 0x55682d  
            vp9_encode_intra_block_y() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodemb.c:658 0x55682d  
            vp9_encode_intra() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodemb.c:676 0x556bcb  
            alt_activity_measure() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodeframe.c:142 0x545d51  
            mb_activity_measure() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodeframe.c:155 0x545d9d  
            build_activity_map() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodeframe.c:311 0x545f02  
            encode_frame_internal() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodeframe.c:2,055 0x54dffb  
            vp9_encode_frame() at /home/springer/RDO/VPX-27NovDebugVP9SSIMProblem/vp9/encoder/vp9_encodeframe.c:2,360 0x54ef93  
            <...more frames...>  

It occurs in the round operation during the transform step, the last line here 
crashes:

      // Transform.
      step[0] = input[0] + input[3];
      step[1] = input[1] + input[2];
      step[2] = input[1] - input[2];
      step[3] = input[0] - input[3];
      temp1 = (step[0] + step[1]) * cospi_16_64;
      temp2 = (step[0] - step[1]) * cospi_16_64;
      out[0] = fdct_round_shift(temp1);

Regards,
Dominic

Original issue reported on code.google.com by dominic....@gmail.com on 27 Nov 2013 at 9:40

Attachments:

• Console.txt

[GoogleCodeExporter]

Original comment by ya...@google.com on 3 Dec 2013 at 5:09

[GoogleCodeExporter]

Hi Dominic,

The current VP9 top of tree code gave segfault while encoding with --tune=ssim.

You mentioned it worked a week ago. I reset to commit: Merge "Correct a couple 
of typos" (ee1e4e645acd0e997040281eb4b59ff7d20a860e). But it didn't work. I got 
the following error:

$ ./vpxenc --codec=vp9 -o test.webm test.I420 --limit=20 -p 2 --best 
--cpu-used=0 --target-bitrate=2000 --auto-alt-ref=1 --minsection-pct=10 
--maxsection-pct=2000 --lag-in-frames=25 --kf-min-dist=0 --kf-max-dist=99999 
--static-thresh=0 --min-q=0 --max-q=63  --arnr-maxframes=7 --arnr-strength=5 
--arnr-type=3 --i420 -w 1280 -h 720 --tune=ssim
Pass 1/2 frame   20/21      3192B    1276b/f   38304b/s 5148898 us (3.88 fps)
Pass 2/2 frame   20/0          0B 5160938 us 3.88 fps [ETA  unknown] vpxenc: 
../libvpx/vp9/common/vp9_reconintra.c:382: vp9_predict_intra_block: Assertion 
`bwl >= 0' failed.
Aborted (core dumped)

It seems this problem existed for quite some time.

Yunqing

Original comment by yunqingw...@google.com on 3 Dec 2013 at 10:51

• Changed state: Assigned

[GoogleCodeExporter]

Tuning for SSIM is not properly implemented for vp9 yet, propose to temporarily 
disable the option here: 
https://gerrit.chromium.org/gerrit/#/c/67993/

Original comment by ya...@google.com on 4 Dec 2013 at 8:42

[GoogleCodeExporter]

Hi Yunqing,

that's true, I was sure I had it working once...  not so sure any more. I
only recently decided to evaluate compression performance for SSIM tuning
in detail. I was just curious because recently a commit arrived in the
master branch allowing to activate Adaptive Quantizers for the Segment IDs,
which to my mind carries the possibility of bringing SSIM performance of
VP9 to a new level. So thats why I wanted to set tune=SSIM, just for fair
evaluation. In my old test, I probably did not set tune=SSIM because they
were focused on PSNR performance, but I evaluted SSIM anyhow, so maybe
that's where I got the impression that SSIM for VP9 works.

On the ICIP2013 VP9 workshop in Melbourne, a lot of SSIM values for VP9
were presented, I just wonder, where the rate points NOT calculated with
tune=SSIM? That's going to be VERY interesting, seeing VP9 perform with
some SSIM optimizer logic!

Thanks to Yaowu for proposing to disable this command line option for now,
otherwise it would be quite confusing.

Original comment by dominic....@gmail.com on 5 Dec 2013 at 12:21

[GoogleCodeExporter]

Original comment by ya...@google.com on 12 Dec 2013 at 5:52

• Changed state: Fixed