init_mb_mgr_avx() corrupts state on Windows

[thiagomacieira]

On Windows, registers XMM6 to XMM15 are callee-preserve (not scratch), so they must be preserved across function calls.

#include <assert.h>
#include <intel-ipsec-mb.h>
#include <immintrin.h>

int run()
{
    MB_MGR *mgr;
    volatile register __m128i xmm10 asm("xmm10");
    xmm10 = _mm_setzero_si128();

    mgr = alloc_mb_mgr(0);
    init_mb_mgr_avx(mgr);

    assert(_mm_testz_si128(xmm10, xmm10));
}

int main()
{
    run();
}

If you compile the above in debug mode and run it on Windows, the assertion will trigger. It is not a valid test case for Linux, where all SSE registers are scratch, and will not work with optimisations because the compiler will move the zeroing of XMM10 to after init_mb_mgr_avx().

Step-debugging this application reveals that registers XMM10 and XMM13 were not preserved.

[thiagomacieira]

I can show it's happening for the self-tests. The ciphers and hash tests appear to be fine, but something leaks from self_test_aead_gcm.

[thiagomacieira]

Specifically, self_test_aead_gcm. I can't tell which of the functions it is.

[tkanteck]

Thanks for reporting the problem! I'll prepare fix for it

[thiagomacieira]

Thanks, Tomász. Let me know if you have a patch you want me to test in the actual application.

[tkanteck]

init_mb_mgr_avx() issue was caused by GCM decrypt finalize operations not preserving/restoring all required registers.

I also see xmm14 being clobbered in init_mb_mgr_avx2(). I'll look into it as well.

[thiagomacieira]

:+1: