SIGILL (Illegal instruction) when transcoding with vp9 - using FFmpeg

[GoogleCodeExporter]

I updated to latest version of libvpx to test VP9 transcoding (using FFmpeg - 
also latest version from master)

With any video I try, the process receive a SIGILL (Illegal instruction) and 
aborts immediately.

The full `gdb` output can be seen here:
https://trac.ffmpeg.org/ticket/3370

When I tried with `valgrind`, the process seemed to work but rendering 0.2 fps 
(using a 2.4GHz Quad core intel processor for a 1280x720 video).

I don't know how I should test VP9 coding without FFmpeg.

Original issue reported on code.google.com by jbv...@gmail.com on 3 Feb 2014 at 7:11

[GoogleCodeExporter]

The error happens at the instruction marked by an arrow below:
File: libvpx/vp9/encoder/x86/vp9_quantize_ssse3.asm:191

.accumulate_eob:
  ; horizontally accumulate/max eobs and write into [eob] memory pointer
  mov                             r2, eobmp
  pshufd                          m7, m8, 0xe
  pmaxsw                          m8, m7
  pshuflw                         m7, m8, 0xe
  pmaxsw                          m8, m7
  pshuflw                         m7, m8, 0x1
  pmaxsw                          m8, m7
--->  pextrw                        [r2], m8, 0
  RET

If you're wondering, the /proc/cpuid (first CPU only):

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
stepping        : 11
microcode       : 0xb6
cpu MHz         : 1600.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm 
constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor 
ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dtherm tpr_shadow vnmi 
flexpriority
bogomips        : 4800.00
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Original comment by jbv...@gmail.com on 3 Feb 2014 at 8:41

[GoogleCodeExporter]

Original comment by ya...@google.com on 6 Feb 2014 at 11:29

• Changed state: Assigned

[GoogleCodeExporter]

Seems like the memory copy for this instruction is only available with SSE4.1
Prior to that (SSE), it could only use a general purpose register as first 
operand.

Original comment by jbv...@gmail.com on 7 Feb 2014 at 1:33

[GoogleCodeExporter]

You can test VP9 encoder using libvpx sample encoder: vpxenc

For building libvpx, please refer to 
http://www.webmproject.org/code/build-prerequisites/

Install Yasm. Clone libvpx to /libvpx directory, and create a build directory 
/build. Under /build, do configuration:
$../libvpx/configure --target=x86_64-linux-gcc
$make

For encoding, please refer to
http://www.webmproject.org/docs/encoder-parameters/

Use --codec=vp9 for VP9 encoder

Please test to see if you still have the problem.

Original comment by yunqingw...@google.com on 7 Feb 2014 at 5:49

[GoogleCodeExporter]

I assume you did 64bit build since quantize ssse3 functions are only included 
for x86_64.

Original comment by yunqingw...@google.com on 7 Feb 2014 at 6:12

[GoogleCodeExporter]

Yes, the error still occurs with this CPU (Core 2 Quad does not have SSE4).
On my Core i7 3rd gen it works.

Original comment by jbv...@gmail.com on 7 Feb 2014 at 6:38

[GoogleCodeExporter]

You could simply change that line for something like:

    pextrw   r3, m8, 0
    mov      [r2], r3

(Assuming r3 is free. I didn't read the rest of the code) 

Original comment by jbv...@gmail.com on 7 Feb 2014 at 6:50

[GoogleCodeExporter]

Please check out the fix. It was already in.

https://gerrit.chromium.org/gerrit/#/c/68694/

Thanks.

Original comment by yunqingw...@google.com on 7 Feb 2014 at 11:13

[GoogleCodeExporter]

Now it works. Thanks.

Original comment by jbv...@gmail.com on 8 Feb 2014 at 7:44

[GoogleCodeExporter]

Original comment by yunqingw...@google.com on 21 Apr 2014 at 5:53

• Changed state: Fixed