Adding a 'best guess' scenario for the scanner to attempt to find and repair a pclntab with bad magic

[alphillips-lab]

Hello!

I've run into a recurring issue with packed/obfuscated samples, where the pclntab magic will be modified. This breaks GoReSym, particularly when trying to validate the table by loading it. This PR contains a fix that I've been using as a python script, but consolidating would be nice.

The fix does a couple things to accomplish a 'best guess' scan for the pclntab:

• Removed short-circuit when scanner thinks it finds valid pclntab via symbol, continue to do bytescans anyways
• Added scan for table by searching for first entry (internal/cpu.Initialize)
• For each signature in pclntab signatures, add a candidate with the valid signature magic bytes (encompass potential Go versions)

This had no major noticeable performance hits for valid Go binaries, although it did take a bit longer when trying to repair the packed/obfuscated files I've run across.

The reason for removing the short circuit is because even if the pclntab symbol may still exist, it points to the mangled pclntab which cannot be loaded by GoReSym. Removing this and just letting it go through the repair scan didn't cause any issues in my testing and still provided useful results on mangled binaries.

I am not so familiar with the Go runtime, nor the GoReSym codebase, so if you have a more optimal way to implement this I am all ears.

Please let me know if you have any comments, questions, or concerns. Thanks!

[stevemk14ebr]

Thanks for submitting a PR!

We have also seen Go binaries with destroyed pclntab magics. Typically though it is fairly easy to patch back the magic bytes to get GoReSym to parse correctly. I am open to adding support for additional scan techniques to support binaries with these destroyed magics, though I have an issue with the magic numbers subtraction this particular technique uses as implemented.

The pclntab (pcHeader) has changed it's members over time between the go versions. It is not safe and general to find some member and then subtract backwards to locate the start of the structure. For example, see this change between go 1.18 and 1.16:

1.18

type pcHeader struct {
    magic          uint32  // 0xFFFFFFF0
    pad1, pad2     uint8   // 0,0
    minLC          uint8   // min instruction size
    ptrSize        uint8   // size of a ptr in bytes
    nfunc          int     // number of functions in the module
    nfiles         uint    // number of entries in the file tab
    textStart      uintptr // base for function entry PC offsets in this module, equal to moduledata.text
    funcnameOffset uintptr // offset to the funcnametab variable from pcHeader
    cuOffset       uintptr // offset to the cutab variable from pcHeader
    filetabOffset  uintptr // offset to the filetab variable from pcHeader
    pctabOffset    uintptr // offset to the pctab variable from pcHeader
    pclnOffset     uintptr // offset to the pclntab variable from pcHeader
}

1.16

type pcHeader struct {
    magic          uint32  // 0xFFFFFFFA
    pad1, pad2     uint8   // 0,0
    minLC          uint8   // min instruction size
    ptrSize        uint8   // size of a ptr in bytes
    nfunc          int     // number of functions in the module
    nfiles         uint    // number of entries in the file tab.
    funcnameOffset uintptr // offset to the funcnametab variable from pcHeader
    cuOffset       uintptr // offset to the cutab variable from pcHeader
    filetabOffset  uintptr // offset to the filetab variable from pcHeader
    pctabOffset    uintptr // offset to the pctab varible from pcHeader
    pclnOffset     uintptr // offset to the pclntab variable from pcHeader
}

The addition of the textStart uintptr will cause this technique to fail. I would prefer a solution that arrives at the pcHeaders start via some other means. I suggest instead doing:

1. Signature the assembly runtime_modulesinit for the common architectures.
2. Within the signatured routine, locate the reference:

   for ( i = &runtime_firstmoduledata; i; i = i->next )

in x86-x64 asm:

.text:0000000000452618 48 8D 05 01 3A 28 00                    lea     rax, type_runtimeModuleData
.text:000000000045261F 90                                      nop
.text:0000000000452620 E8 9B C6 FB FF                          call    runtime_newobject            
.text:0000000000452625 48 89 44 24 50                          mov     [rsp+68h+var_18], modules_0
.text:000000000045262A 48 8D 0D CF 26 52 00                    lea     rcx, runtime_firstmoduledata
.text:0000000000452631 EB 0D                                   jmp     short loc_452640
.text:0000000000452633                         loc_452633:      
.text:0000000000452633 48 8B 89 30 02 00 00                    mov     rcx, [rcx+230h]           ; +0x230 will vary between go version
.text:000000000045263A 66 0F 1F 44 00 00                       nop     word ptr [rax+rax+00h]

An example signature might be:

48 8D 05 ? ? ? ? 90 E8 ? ? ? ? 48 89 44 24 ? 48 8D 0D (?<first_moduledata_offset> ? ? ? ?) EB 0D 48 8B 89 ? ? ? ? 66 0F 1F 44 00 ?

This would give us an offset to the runtime_moduledata which will hold a pointer to the start of the pcHeader. And the modulesinit routine is present since the begging of time in the Go world (though we may need a few different assembly signatures).

Would you be up for trying to implement that technique instead, or something similar that does not rely on magic numbers or signaturing fields within the pcHeader itself?

[alphillips-lab]

Sure, I'll take a look and see what I can do.

[alphillips-lab]

Hello again,

It's a bit of a larger commit, but I've gotten it working to where I would like it for my use case. See objfile/scanner.go for the signatures and scan logic.

A couple notes:

• I believe this method will not work on binaries using big endian (I can try to add that in some other time).
• Performance is as expected when dealing with binaries GoReSym could already handle, but when attempting to repair it takes much longer. I think this is not because of the scanner per se and seems to be the parser that deals with the candidates. I have not looked at this code but can potentially optimize in the future. Tested binaries needing repair brought time from around 5 seconds to 25 seconds.
• I only signatured recent files that I was dealing with (Go 1.20), but it should be easy to add more.

[stevemk14ebr]

Thanks for your contribution! I will review what you've written in a week or two and likely extend it to handle the cases you've mentioned and make it robust to all versions and configurations (this may take some time - don't fret if you don't see progress here for a bit). Thanks again!

[stevemk14ebr]

Hi! I've made the following changes:

• The scanner's signatures were all for the x64 architecture and not dependant on the OS - I've removed that parameter to simplify it.
• The scanner now returns a structure rather than anonymous array to improve readability
• The new technique is changed to always occur - but I reverted the changes done to the pre-existing techniques. The reason for this is they are independant techniques, the previous ones work well and I do not want to change that logic. So, all existing logic will occur exactly the same, the short circuit exit scan was removed, and a signature scan will occur over each section adding new candidates for this method
• The .Data() call in the Sections loop was duplicating an expensive read. The DataAfterSection() data is now used instead for the signature scan
• An additional loop through the sections was added to read out the pclntab data after the signature scan, this duplicated another expensive operation and also created an issue when the pclntab section is before the .text section (as only loop iterations through the sections beyond where the signature scan matching modulesinit would be checked to be in range of the found pointer) - I changed this to the much cheaper read_memory api which solves both issues.
• The candidates now patch their magic values for all supported magics which handles the case they were intentionally corrupted
• Big Endian and Little Endian scanner results are now both supported - candidates of each type are added from the signature scan.
• Commenting added

This seems to work really well for x64 targets on windows/mac/linux. There is still no support for other architectures such as x86, arm, or ppc. The scanner itself also does not support BigEndian, there is a single LittleEndian pointer read in there that must be changed in the future. All that's left to do is add signature and RVA resolution logic within scanner.go for all other architectures, then testing. The core of the candidate logic for each os should be complete.

I will continue to work on this!

[alphillips-lab]

Awesome!! Thanks so much for taking a look and working on changes! I've learned a lot reading through them.

And yeah I am aware that the scanner wasn't made for both BE and LE. I think you store that data in the file structs for elf and macho (File.FileHeader.ByteOrder), so I could try and either create a second scanner or add a case to the current scanner if the findModuleInitPCHeader function also receives the ByteOrder. Let me know if you'd like help, but I also don't want to make you have to refactor more code haha.

[stevemk14ebr]

I had not thought to use the ELF header for the endianess, that is smart I will modify this to do that! The scanner as it is now is capable of doing both big and little endian. The choice to use big/little endian is now handled in the callback that matches the signature, so we can choose which to use based on the arch being matched. I only expect us to see BE for arm and ppc really anyways.

I'll keep adding signatures and get this merged with some time!

[alphillips-lab]

Ah yes, I see how you use the callback. I like your implementation! I wouldn't have thought to do it that way, but it is smart and cuts down on needing to write separate implementations. Makes things much nicer!

[stevemk14ebr]

This PR is now complete! Thank you very much for your original issue and first pass at a PR. GoReSym now supports stomped PCLNTAB magic for the following architectures/os's

win x lin x mac x ( arm64 x arm x ppc64_be x x64 x x86)

There is a new dependency on LIBYARA. I have had to add this dependency as the custom written signature scanner even with manual optimizations be myself was much too slow, and did not support features I ended up needing such as logical OR and skip ranges. The new scanner is so fast there is no perceptible slowdown when falling back to this scanning mode for stomped magics, and allowed me to support more architectures!

Mission success!