Add `UD0_COMPAT` decoder mode

[flobernd]

Closes #436

[jpidancet]

Hi @flobernd, thanks a lot for working on this. I verified your changes give the expected results using this simple program:

#include <Zydis.h>
#include <assert.h>

int main(void)
{
    ZydisDecoder decoder;
    const char data[4] = "\x0f\xff\x00\x00";
    ZydisDecodedInstruction instruction;
    ZydisDecodedOperand operands[ZYDIS_MAX_OPERAND_COUNT];
    ZyanStatus status;

    ZydisDecoderInit(&decoder, ZYDIS_MACHINE_MODE_LONG_64,
             ZYDIS_STACK_WIDTH_64);
    ZydisDecoderEnableMode(&decoder, ZYDIS_DECODER_MODE_UD0_COMPAT, ZYAN_FALSE);
    status = ZydisDecoderDecodeFull(&decoder,
                    data, sizeof(data),
                    &instruction,
                    operands);
    assert(ZYAN_SUCCESS(status));
    assert(instruction.mnemonic == ZYDIS_MNEMONIC_UD0);
    assert(instruction.length == 3);

    ZydisDecoderEnableMode(&decoder, ZYDIS_DECODER_MODE_UD0_COMPAT, ZYAN_TRUE);
    status = ZydisDecoderDecodeFull(&decoder,
                    data, sizeof(data),
                    &instruction,
                    operands);

    assert(ZYAN_SUCCESS(status));
    assert(instruction.mnemonic == ZYDIS_MNEMONIC_UD0);
    assert(instruction.length == 2);

    return 0;   
}

[flobernd]

I think the encoder does not like this new mode. Maybe @mappzor can have a look 🙂

[mappzor]

Encoder itself is fine, it can encode everything based on mnemonics, signatures and encodings. ZydisFuzzEncoder starts with a random encoder request, so it needs to figure out how to configure decoder for verification purposes.

I would just add this piece of code here:

    if (request.mnemonic == ZYDIS_MNEMONIC_UD0 && request.operand_count == 0)
    {
        status = ZydisDecoderEnableMode(&decoder, ZYDIS_DECODER_MODE_UD0_COMPAT, ZYAN_TRUE);
        if (!ZYAN_SUCCESS(status))
        {
            fputs("Failed to enable UD0_COMPAT mode\n", ZYAN_STDERR);
            abort();
        }
    }