Unsupported API stats to help contributors

re-fox commented 4 years ago

First off, thanks for releasing this tool.

I ran speakeasy against the Malpedia corpus (https://malpedia.caad.fkie.fraunhofer.de/) to get a rough estimate of how many samples successfully emulate with/without tossing errors. While running this test I gathered up a list of the count of unsupported API functions that were causing emulation to halt.

I understand a github issue may not be the best place to store this information. It may be useful for someone looking to contribute by going after the highly used API's first.

The result of ~4k samples (truncating results at 15 - the data has a long tail of one-off's) (Updated: 2022-02-17 running against c94bb629e8ecc1ab00b326b3e2919365c4bbc66d)

    150 advapi32.CryptImportKey
    128 advapi32.ConvertStringSecurityDescriptorToSecurityDescriptorA
    122 msvbvm60.ordinal_100
    102 user32.OpenInputDesktop
    100 kernel32.LocalFileTimeToFileTime
     99 msvcrt._wgetenv
     96 advapi32.EventRegister
     75 comctl32.ordinal_17
     71 gdi32.GetSystemPaletteEntries
     64 mfc42.ordinal_1576
     60 kernel32.HeapValidate
     56 advapi32.RegCreateKeyExA
     55 kernel32.GetThreadPreferredUILanguages
     54 advapi32.InitializeSecurityDescriptor
     52 shell32.SHGetSpecialFolderPathA
     45 advapi32.RegCreateKeyExW
     41 kernel32.GetTimeZoneInformation
     40 msvcrt.__p___initenv
     31 shlwapi.PathFileExistsW
     30 userenv.GetUserProfileDirectoryW
     30 kernel32.GetTempFileNameA
     29 user32.GetWindowRect
     27 kernel32.SetFileAttributesW
     27 kernel32.SetFileAttributesA
     27 iphlpapi.GetAdaptersInfo
     26 user32.MapVirtualKeyW
     25 oleaut32.SysAllocStringLen
     24 kernel32.InitializeSRWLock
     23 user32.RegisterClipboardFormatA
     22 shell32.SHGetSpecialFolderPathW
     22 ntdll.VerSetConditionMask
     21 user32.GetCursorInfo
     21 urlmon.ObtainUserAgentString
     20 user32.RegisterClassA
     20 kernel32.GetProcessAffinityMask
     19 wininet.HttpAddRequestHeadersA
     19 oleaut32.ordinal_2
     19 kernel32.RtlPcToFileHeader
     19 kernel32.GetSystemWow64DirectoryA
     19 advapi32.RegSetValueExA
     18 kernel32.SetFilePointerEx
     18 advapi32.SetEntriesInAclA
     17 msvcrt.atexit
     17 advapi32.RegisterServiceCtrlHandlerExW
     16 oleaut32.SysReAllocStringLen
     15 ntdll.RtlAdjustPrivilege
     15 msvcrt._ismbblead
     15 kernel32.SetProcessShutdownParameters
     15 kernel32.GlobalMemoryStatusEx
     15 kernel32.FreeResource
     15 gdiplus.GdiplusStartup

In addition to the above data, Malpedia publicly posts an API frequency graph: https://malpedia.caad.fkie.fraunhofer.de/stats/api_dll_frequencies

drewvis commented 4 years ago

Hey, thanks a lot that is super helpful! I'll be sure to keep this open.

re-fox commented 4 years ago

Quite a few changes were updated last night, I re-ran the analysis and posted up the new results. I'll keep editing this issue on a semi-regular basis to keep the list current.

Flerov commented 2 years ago

Could I get a hint for where I have to add undocumented functions for emulation support. I really need this to be working for the API ntoskrnl.ZwCreateKey

jtbennett-fe commented 2 years ago

@Flerov this blog has the hints you are looking for: https://www.mandiant.com/resources/blog/using-speakeasy-emulation-framework-programmatically-to-unpack-malware.

Flerov commented 2 years ago

@JohnLaTwC Thanks for the link do you have some further recommendations to get into the functionality of API hooks implemented in '/speakesy/winenv/api/kernelmode/' specifically for '[...]/ntoskrnl.py' since the function is not supported and a simple return 0x0 wont help for further emulation of the driver I'm facing. I would like to get into how to add support for the API 'ZwCreateKey' for a successfull emulation

From my report it seems that "ZwCreateKey" gets called and after that there is only one more api being called 'RtlInitUnicodeString' According to the listed errors my error type is: 'NoneType' object has no attribute 'to_bytes' raised by the emulation's attempt to call "ZwOpenKey" My Terminal output give me the following hint:"""Invalid memory read (UC_ERR_READ_UNMAPPED) Unsupported API: ntoskrnl.ZwCreateKey"""

I appreciate any further links to documentation and help regarding my problem many thanks in advance

jtbennett-fe commented 2 years ago

@Flerov I recommend reading the documentation on the API: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/nf-wdm-zwcreatekey

It is up to you to decide how much of the API you want to support in emulation to meet your needs.

Flerov commented 2 years ago

@jtbennett-fe I added support for ZwCreateKey as far as I was able to. Seems like it works better to change the create Flag in the code for ZwOpenKey.

TLDR: Finally it works now, I appreciate it alot thanks

jtbennett-fe commented 2 years ago

@Flerov awesome, glad to hear it!

mandiant / speakeasy

Unsupported API stats to help contributors #3