Unable to disassemble valid VXORPS instruction.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Try to disassemble a VEX-encoded AVX VXORPS instruction using
'distorm_decompose'. Currently, I'm seeing it fail on the following instruction:

C5 FC 57 C0      vxorps      ymm0,ymm0,ymm0

What is the expected output? What do you see instead?
I'd expect it to return DECRES_SUCCESS, and set usedInstructionCount to 1. 
Instead, I see DECRES_MEMORYERR. I attempted to follow the decoding, and it 
recognizes that its a VEX encoded instruction, but appears to derail in 
inst_get_info.

What version of the product are you using? On what operating system?
Using trunk r193 on Windows 7.

In the code I'm looking at, there's a sequence of these instructions:

C5 FC 57 C0      vxorps      ymm0,ymm0,ymm0 
C5 F4 57 C9      vxorps      ymm1,ymm1,ymm1 
C5 EC 57 D2      vxorps      ymm2,ymm2,ymm2 
C5 E4 57 DB      vxorps      ymm3,ymm3,ymm3 
C5 DC 57 E4      vxorps      ymm4,ymm4,ymm4 
C5 D4 57 ED      vxorps      ymm5,ymm5,ymm5 
C5 CC 57 F6      vxorps      ymm6,ymm6,ymm6 
C5 C4 57 FF      vxorps      ymm7,ymm7,ymm7 

The Visual Studio 2008 decoder has no problem with them.

Original issue reported on code.google.com by chri...@google.com on 26 Sep 2011 at 9:32

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

After digging through the code a bit more, this appears to only be an issue for 
the AVX 0F 57 operand, which corresponds to VXORPS. Specifically, it's only an 
issue when the VEX.V == 0, in which case the resulting index (0x944) into 
InstructionsTree is empty. However, the VXORPS instruction should be decoded in 
the same manner for any value of VEX.V, so adding a reference to _V_0F_57 
(value 0x4046) at this entry fixes the problem for me.

Original comment by chri...@google.com on 27 Sep 2011 at 6:49

Attachments:

• vxorps-v0.diff

[GoogleCodeExporter]

I will check it out when I get the time, thanks for the report.

Original comment by distorm@gmail.com on 28 Sep 2011 at 1:19

[GoogleCodeExporter]

Original comment by distorm@gmail.com on 29 Oct 2011 at 8:39

• Changed state: Accepted

[GoogleCodeExporter]

This bug affects more than just 'vxorps'. 

For example:
c5 7b 5e c0
should decode as:
VDIVSD  XMM8, XMM0, XMM0

but it comes back as undefined. 

If you change the VEX.V field to anything besides 0 (~0xf) though, it decodes 
fine:
c5 73 5e c0
VDIVSD XMM8, XMM1, XMM0

Either the insts.c table needs to be updated in a lot of places, or this part 
of the decoding of the vex prefix is wrong:
http://code.google.com/p/distorm/source/browse/trunk/src/instructions.c#242

        /* If the instruction is encoded using the vvvv field, fix the index into the Prefixed table. */
        if (v == 0) index += 4;

Original comment by spa...@rotateright.com on 8 Feb 2012 at 7:18

[GoogleCodeExporter]

Sanjay, the decoding is wrong. The case of zero V is tricky more than I thought.
I will have to think of a proper solution.
It concerns all AVX instructions.

Original comment by distorm@gmail.com on 26 Feb 2012 at 11:11

[GoogleCodeExporter]

Original comment by distorm@gmail.com on 27 Feb 2012 at 5:38

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Seems like I misread the documentation ;(
Removed that line.

Original comment by distorm@gmail.com on 28 Feb 2012 at 11:26

• Changed state: Fixed

[GoogleCodeExporter]

Looks like things are decoding correctly now with vvvv == 1111:
$ ./a.out c5 7b 5e c0
   1: (   0) c57b5ec0                 VDIVSD XMM8, XMM0, XMM0
$ ./a.out c5 fc 57 c0
   1: (   0) c5fc57c0                 VXORPS YMM0, YMM0, YMM0

and with vvvv != 1111:
$ ./a.out c5 0c 57 c0
decodedinst = 1
   1: (   0) c50c57c0                 VXORPS YMM8, YMM14, YMM0
$ ./a.out c5 0b 5e c0
decodedinst = 1
   1: (   0) c50b5ec0                 VDIVSD XMM8, XMM14, XMM0

Original comment by spa...@rotateright.com on 6 Mar 2012 at 5:32