Cthulhu's proposed fix for studiorender.so crashes on Linux

[L1pE]

After discussing these crashes with Cthulhu, he has provided me a hex-editted studiorender.so to test a solution, which worked, fixed #315 and others. He has explained the instruction causing the crash (movntps) requires an aligned memory block (16-bytes aligned in this case). He then proposed the following fix to be implemented in the code of the mod:

Define the offsets in one of the headers

static constexpr std::array<unsigned long, 11> offsets
{
    0x0001649C, 0x0001649F, 0x000164A3, 0x000164A7,
    0x0001A6ED, 0x0001A6F0, 0x0001A6F4,
    0x00016B38, 0x00016B3B, 0x00016B3F, 0x00016B43
};

Add the following in a function

int ApplyStudioRenderFixes(struct dl_phdr_info* info, size_t size, void* data)
{
    if (!size || info == nullptr)
        return 0;

    if (info->dlpi_addr == nullptr || info->dlpi_name == nullptr)
        return 0;

    unsigned long base = (unsigned long)info->dlpi_addr;

    std::string library(info->dlpi_name);

    if (library.find('/') == string::npos)
    {
        if (library != "studiorender.so")
            return 0;
    }
    else
    {
        std::string studiorender("/studiorender.so");

        // Endswith: studiorender.so
        if (library.size() < studiorender.size())
            return 0;
        if (library.compare(library.size() - studiorender.size(), string::npos, studiorender) != 0)
            return 0;
    }

    for (auto offset: offsets)
    {
        byte* address = (byte*)(base + offset);

        // movntps -> movups
        if (*address == 0x2B && !mprotect(address, 1, PROT_READ | PROT_WRITE | PROT_EXEC))
        {
            *address = 0x11;
            mprotect(address, 1, PROT_READ | PROT_EXEC);
        }
    }

    return 1;
}

Then call, after loading studiorender.so and only once

dl_iterate_phdr(&ApplyStudioRenderFixes, nullptr);

This is extensible if there's still other crashes left in other addresses, you must load the coredump (or converted minidump file) in GDB, find the address of the crash, subtract it by the base address of the .so, then run x/i5 $pc and put all the movntps offsets on the array above, since movntps and movups share the same first byte within their instruction opcodes as well as the number and type of operands, only the address of the second byte is changed, from 0x2B to 0x11 (he explained), the loop will automatically apply the patch.

EDIT: code improvements.

[TotallyMehis]

Very cool if this works. I'll check this out at some point.

[L1pE]

Any news on this?

[TotallyMehis]

Nothing yet. I'll update the issue once there is.

[TotallyMehis]

I was able to stop the crashing (from what I tested) by recompiling the Pi model (https://github.com/zm-reborn/zmr-assets/commit/30849a8a12fb2ccaab4076f81ffcc0994faaa507).

I'm hesitant to start fixing this issue by patching binaries, especially when the cause is a bad decompile. I vividly remember having a similar issue on Windows with the Lawyer model, and that model doesn't have any of those flex definitions that Pi had.