search

mandiant / capa

The FLARE team's open-source tool to identify capabilities in executable files.
Apache License 2.0
3.99k stars 501 forks source link

Missed detection of imported ELF symbols #783

Open TcM1911 opened 2 years ago

TcM1911 commented 2 years ago

There's still some issues with the "api" feature detection. I think this issue is also due to a bug in vivisect. See this file for example. 

md5                     3db3e55b16a7b1b1afb970d5e77c5d98
sha1                    bb44696c170e09b7708936ca3597002e3b8d93ff
sha256                  294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc
timestamp               2021-09-17T09:37:18.042198
capa version            v3.0.0-0-g5972d65
os                      linux
format                  elf
arch                    amd64
extractor               VivisectFeatureExtractor
base address            0x400000
rules                   /tmp/_MEIr6ntHQ/rules
function count          242
library function count  0
total feature count     7139

receive data
namespace    communication
author       william.ballenthin@fireeye.com
description  all known techniques for receiving data from a potential C2 server
scope        function
mbc          Command and Control::C2 Communication::Receive Data [B0030.002]
examples     BFB9B5391A13D0AFD787E87AB90F14F5:0x13145D60
function @ 0x40D640
  or:
    match: receive data on socket @ 0x40D640
      or:
        api: recv @ 0x40D753

receive data on socket
namespace  communication/socket/receive
author     moritz.raabe@fireeye.com, joakim@intezer.com
scope      function
mbc        Communication::Socket Communication::Receive Data [C0001.006]
examples   Practical Malware Analysis Lab 01-01.dll_:0x10001010
function @ 0x40D640
  or:
    api: recv @ 0x40D753

create UDP socket
namespace  communication/socket/udp/send
author     moritz.raabe@fireeye.com, joakim@intezer.com
scope      basic block
mbc        Communication::Socket Communication::Create UDP Socket [C0001.010]
examples   203BD48BCC18434314AD60F4C8BC21E3D3422EB0624B22B827410F9BC63B4082:0x401240
basic block @ 0x40C740
  and:
    count(number(0x2 = AF_INET/SOCK_DGRAM)): 2 or more = AF_INET/SOCK_DGRAM @ 0x40C745, 0x40C74F, 0x40C77B
    or:
      api: socket @ 0x40C767

resolve DNS
namespace  host-interaction/network/dns/resolve
author     william.ballenthin@fireeye.com, johnk3r, joakim@intezer.com
scope      function
mbc        Communication::DNS Communication::Resolve [C0011.001]
examples   17264e3126a97c319a6a0c61e6da951e:0x5FDC25D0
function @ 0x40E0D0
  or:
    api: gethostbyname @ 0x40E10B

create thread
namespace  host-interaction/thread/create
author     moritz.raabe@fireeye.com, michael.hunhoff@fireeye.com, joakim@intezer.com
scope      basic block
mbc        Process::Create Thread [C0038]
examples   946A99F36A46D335DEC080D9A4371940:0x10001DA0, B5F85C26D7AA5A1FB4AF5821B6B5AB9B:0x408020
basic block @ 0x409B49
  or:
    and:
      os: linux
      api: pthread_create @ 0x409B66

Few are detected but if we check the "imports" there should be a few more rules that got triggered. We can see that the sample imports both "uname" and "popen".

Symbol table '.dynsym' contains 150 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND ftell@GLIBC_2.2.5 (2)
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND d2i_RSA_PUBKEY@libcrypto.so.10 (3)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SSL_get_verify_result@libssl.so.10 (4)
     4: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND ERR_load_BIO_strings@libcrypto.so.10 (3)
     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND endmntent@GLIBC_2.2.5 (2)
     6: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sendto@GLIBC_2.2.5 (5)
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8__detail15_List_nod@GLIBCXX_3.4.15 (6)
     8: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND syscall@GLIBC_2.2.5 (2)
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND uname@GLIBC_2.2.5 (2)
    10: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND rewind@GLIBC_2.2.5 (2)
    11: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sem_timedwait@GLIBC_2.2.5 (5)
    12: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_ctrl@libcrypto.so.10 (3)
    13: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sprintf@GLIBC_2.2.5 (2)
    14: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs6appendEPKcm@GLIBCXX_3.4 (7)
    15: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSdD2Ev@GLIBCXX_3.4 (7)
    16: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SSL_CTX_free@libssl.so.10 (4)
    17: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs12_M_leak_hardEv@GLIBCXX_3.4 (7)
    18: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __res_query@GLIBC_2.2.5 (8)
    19: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNKSs4findEPKcmm@GLIBCXX_3.4 (7)
    20: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND OPENSSL_add_all_algorithm@libcrypto.so.10 (3)
    21: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_write@libcrypto.so.10 (3)
    22: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND socket@GLIBC_2.2.5 (2)
    23: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND rand@GLIBC_2.2.5 (2)
    24: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SSL_load_error_strings@libssl.so.10 (4)
    25: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8ios_baseC2Ev@GLIBCXX_3.4 (7)
    26: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND popen@GLIBC_2.2.5 (2)
    27: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND recv@GLIBC_2.2.5 (5)
    28: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8ios_baseD2Ev@GLIBCXX_3.4 (7)
    29: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND AES_set_decrypt_key@libcrypto.so.10 (3)
    30: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZSt17__throw_bad_allocv@GLIBCXX_3.4 (7)
    31: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSo9_M_insertIxEERSoT_@GLIBCXX_3.4.9 (9)
    32: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __cxa_begin_catch@CXXABI_1.3 (10)
    33: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getprotobyname@GLIBC_2.2.5 (2)
    34: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strlen@GLIBC_2.2.5 (2)
    35: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND inet_ntoa@GLIBC_2.2.5 (2)
    36: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZStrsIcSt11char_traitsIc@GLIBCXX_3.4 (7)
    37: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strncmp@GLIBC_2.2.5 (2)
    38: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND RSA_public_encrypt@libcrypto.so.10 (3)
    39: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getcwd@GLIBC_2.2.5 (2)
    40: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SSL_CTX_new@libssl.so.10 (4)
    41: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_read@libcrypto.so.10 (3)
    42: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memset@GLIBC_2.2.5 (2)
    43: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSsC1ERKSs@GLIBCXX_3.4 (7)
    44: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_create@GLIBC_2.2.5 (5)
    45: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND geteuid@GLIBC_2.2.5 (2)
    46: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND EVP_sha256@libcrypto.so.10 (3)
    47: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs4_Rep10_M_disposeERK@GLIBCXX_3.4 (7)
    48: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getnameinfo@GLIBC_2.2.5 (2)
    49: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZSt19__throw_logic_error@GLIBCXX_3.4 (7)
    50: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND localtime@GLIBC_2.2.5 (2)
    51: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sleep@GLIBC_2.2.5 (2)
    52: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND recvfrom@GLIBC_2.2.5 (5)
    53: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sem_post@GLIBC_2.2.5 (5)
    54: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs7reserveEm@GLIBCXX_3.4 (7)
    55: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs4_Rep10_M_destroyERK@GLIBCXX_3.4 (7)
    56: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (11)
    57: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_new_mem_buf@libcrypto.so.10 (3)
    58: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __cxa_atexit@GLIBC_2.2.5 (2)
    59: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND AES_cbc_encrypt@libcrypto.so.10 (3)
    60: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZdlPv@GLIBCXX_3.4 (7)
    61: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND srand@GLIBC_2.2.5 (2)
    62: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND setmntent@GLIBC_2.2.5 (2)
    63: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fclose@GLIBC_2.2.5 (2)
    64: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _Znwm@GLIBCXX_3.4 (7)
    65: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __xstat@GLIBC_2.2.5 (2)
    66: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND inet_addr@GLIBC_2.2.5 (2)
    67: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SHA256@libcrypto.so.10 (3)
    68: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_new@libcrypto.so.10 (3)
    69: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs6appendERKSs@GLIBCXX_3.4 (7)
    70: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fseek@GLIBC_2.2.5 (2)
    71: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sem_wait@GLIBC_2.2.5 (5)
    72: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSsC1EPKcRKSaIcE@GLIBCXX_3.4 (7)
    73: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND gethostbyname@GLIBC_2.2.5 (2)
    74: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSsC1EmcRKSaIcE@GLIBCXX_3.4 (7)
    75: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getpwuid@GLIBC_2.2.5 (2)
    76: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fopen@GLIBC_2.2.5 (2)
    77: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND signal@GLIBC_2.2.5 (2)
    78: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND free@GLIBC_2.2.5 (2)
    79: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND daemon@GLIBC_2.2.5 (2)
    80: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getmntent@GLIBC_2.2.5 (2)
    81: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_f_base64@libcrypto.so.10 (3)
    82: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs6assignERKSs@GLIBCXX_3.4 (7)
    83: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND malloc@GLIBC_2.2.5 (2)
    84: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND clock_gettime@GLIBC_2.2.5 (12)
    85: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt9basic_iosIcSt11char@GLIBCXX_3.4 (7)
    86: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND HMAC@libcrypto.so.10 (3)
    87: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNKSs5rfindEcm@GLIBCXX_3.4 (7)
    88: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strtol@GLIBC_2.2.5 (2)
    89: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt6localeD1Ev@GLIBCXX_3.4 (7)
    90: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pclose@GLIBC_2.2.5 (2)
    91: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SSLv23_client_method@libssl.so.10 (4)
    92: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZSt20__throw_out_of_rang@GLIBCXX_3.4 (7)
    93: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND tolower@GLIBC_2.2.5 (2)
    94: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __cxa_rethrow@CXXABI_1.3 (10)
    95: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getpid@GLIBC_2.2.5 (2)
    96: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND chdir@GLIBC_2.2.5 (2)
    97: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fread@GLIBC_2.2.5 (2)
    98: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND gethostname@GLIBC_2.2.5 (2)
    99: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND opendir@GLIBC_2.2.5 (2)
   100: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND AES_set_encrypt_key@libcrypto.so.10 (3)
   101: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt18basic_stringstream@GLIBCXX_3.4 (7)
   102: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND readdir@GLIBC_2.2.5 (2)
   103: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND feof@GLIBC_2.2.5 (2)
   104: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_new_ssl_connect@libssl.so.10 (4)
   105: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_push@libcrypto.so.10 (3)
   106: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSsC1ERKSsmm@GLIBCXX_3.4 (7)
   107: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strtok@GLIBC_2.2.5 (2)
   108: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fgets@GLIBC_2.2.5 (2)
   109: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND closedir@GLIBC_2.2.5 (2)
   110: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memmove@GLIBC_2.2.5 (2)
   111: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt8__detail15_List_nod@GLIBCXX_3.4.15 (6)
   112: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_new_connect@libcrypto.so.10 (3)
   113: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __cxa_end_catch@CXXABI_1.3 (10)
   114: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs4swapERSs@GLIBCXX_3.4 (7)
   115: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSolsEi@GLIBCXX_3.4 (7)
   116: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND freeifaddrs@GLIBC_2.3 (13)
   117: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sem_init@GLIBC_2.2.5 (5)
   118: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _Unwind_Resume@GCC_3.0 (14)
   119: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND setsockopt@GLIBC_2.2.5 (2)
   120: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND SSL_ctrl@libssl.so.10 (4)
   121: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_set_flags@libcrypto.so.10 (3)
   122: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND RSA_size@libcrypto.so.10 (3)
   123: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_free_all@libcrypto.so.10 (3)
   124: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
   125: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND ferror@GLIBC_2.2.5 (2)
   126: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
   127: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getifaddrs@GLIBC_2.3 (13)
   128: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSs4_Rep9_S_createEmmRK@GLIBCXX_3.4 (7)
   129: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fwrite@GLIBC_2.2.5 (2)
   130: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND close@GLIBC_2.2.5 (5)
   131: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND BIO_s_mem@libcrypto.so.10 (3)
   132: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNSt6localeC1Ev@GLIBCXX_3.4 (7)
   133: 000000000040f804     0 FUNC    GLOBAL DEFAULT   14 _fini
   134: 00000000004032b0     0 FUNC    WEAK   DEFAULT  UND __pthread_key_create@GLIBC_2.2.5 (5)
   135: 0000000000402b50     0 FUNC    GLOBAL DEFAULT   11 _init
   136: 00000000006154a0     0 NOTYPE  GLOBAL DEFAULT   26 __bss_start
   137: 00000000006159c8     0 NOTYPE  GLOBAL DEFAULT   26 _end
   138: 00000000006155e0    88 OBJECT  WEAK   DEFAULT   26 _ZTVN10__cxxabiv117__clas@CXXABI_1.3 (10)
   139: 0000000000615560   128 OBJECT  WEAK   DEFAULT   26 _ZTVSt15basic_streambufIc@GLIBCXX_3.4 (7)
   140: 0000000000403230     0 FUNC    GLOBAL DEFAULT  UND _ZNSsD1Ev@GLIBCXX_3.4 (7)
   141: 00000000004032a0     0 FUNC    GLOBAL DEFAULT  UND __gxx_personality_v0@CXXABI_1.3 (10)
   142: 00000000006154a0     0 NOTYPE  GLOBAL DEFAULT   25 _edata
   143: 00000000006154c0    32 OBJECT  GLOBAL DEFAULT   26 _ZNSs4_Rep20_S_empty_rep_@GLIBCXX_3.4 (7)
   144: 0000000000615640   128 OBJECT  WEAK   DEFAULT   26 _ZTVSt15basic_stringbufIc@GLIBCXX_3.4 (7)
   145: 00000000006154a0    32 OBJECT  WEAK   DEFAULT   26 _ZTVSt9basic_iosIcSt11cha@GLIBCXX_3.4 (7)
   146: 000000000040e670   112 FUNC    WEAK   DEFAULT   13 _ZNSt15basic_stringbufIcS
   147: 000000000040e600   104 FUNC    WEAK   DEFAULT   13 _ZNSt15basic_stringbufIcS
   148: 00000000006154e0   120 OBJECT  WEAK   DEFAULT   26 _ZTVSt18basic_stringstrea@GLIBCXX_3.4 (7)
   149: 00000000006156c0    80 OBJECT  WEAK   DEFAULT   26 _ZTTSt18basic_stringstrea@GLIBCXX_3.4 (7)

This is a snippet where the function calls popen on the last line.

0x00404e80      4155           push r13                                                                                
│           0x00404e82      4154           push r12                                                                                
│           0x00404e84      4989fc         mov r12, rdi                ; arg1                                                      
│           0x00404e87      55             push rbp                                                                                
│           0x00404e88      53             push rbx                                                                                
│           0x00404e89      4881ec580400.  sub rsp, 0x458              ; rsi ; rsi                                                 
│           0x00404e90      e84beaffff     call fcn.threadID           ;[1]                                                        
│           0x00404e95      89c7           mov edi, eax                                                                            
│           0x00404e97      e8b4a10000     call fcn.srand              ;[2]                                                        
│                                                                      ; void srand(-1)                                            
│           0x00404e9c      498d7c2418     lea rdi, [r12 + 0x18]                                                                   
│           0x00404ea1      ba0c000000     mov edx, 0xc                ; rdx                                                       
│           0x00404ea6      be94f84000     mov esi, str._2_1___exit    ; 0x40f894 ; " 2>&1 ; exit"                                 
│           0x00404eab      e8a0ddffff     call sym std::string::append(char const*, unsigned long) ;[3] ; sym.imp.std::string::ap
│           0x00404eb0      be01000000     mov esi, 1                  ; rsi                                                       
│           0x00404eb5      bf11000000     mov edi, 0x11               ; 17                                                        
│           0x00404eba      e881e1ffff     call sym.imp.signal         ;[4] ; void signal(int sig, void *func)                     
│                                                                      ; void signal(-1, -1)                                       
│           0x00404ebf      498b7c2418     mov rdi, qword [r12 + 0x18]                                                             
│           0x00404ec4      bebdf84000     mov esi, str.r              ; 0x40f8bd ; "r"                                            
│           0x00404ec9      e842deffff     call sym.imp.popen          ;[5] ; file*popen(const char *filename, const char *mode)

The rule create process on Linux is very simple and should have fired. I can't see any other reason than the symbols not being discovered correctly being at fault.

rule:
  meta:
    name: create process on Linux
    namespace: host-interaction/process/create
    author:
      - joakim@intezer.com
    scope: basic block
    mbc:
      - Process::Create Process [C0017]
    examples:
      - 7351f8a40c5450557b24622417fc478d:0x40236D
  features:
    - and:
      - os: linux
      - or:
        - api: execve
        - api: execl
        - api: execlp
        - api: execle
        - api: execv
        - api: execvp
        - api: execvpe
        - api: posix_spawn
        - api: posix_spawnp
        - api: popen

It's not caused by the os requirement. If I modify the rule to:

rule:
  meta:
    name: create process on Linux
    namespace: host-interaction/process/create
    author:
      - joakim@intezer.com
    scope: basic block
    mbc:
      - Process::Create Process [C0017]
    examples:
      - 7351f8a40c5450557b24622417fc478d:0x40236D
  features:
    - or:
      - api: execve
      - api: execl
      - api: execlp
      - api: execle
      - api: execv
      - api: execvp
      - api: execvpe
      - api: posix_spawn
      - api: posix_spawnp
      - api: popen

It still doesn't fire, while working fine on other samples:

$ capa --rules tools/capa-rules/host-interaction/process/create/create-process-on-linux.yml path/to/vermilion/294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc.sample 
loading : 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1339.18 rules/s]
matching: 100%|████████████████████████████████████████████| 242/242 [00:00<00:00, 258.36 functions/s, skipped 0 library functions]
+------------------------+----------------------------------------------------------------------------------------------------+
| md5                    | 3db3e55b16a7b1b1afb970d5e77c5d98                                                                   |
| sha1                   | bb44696c170e09b7708936ca3597002e3b8d93ff                                                           |
| sha256                 | 294b8db1f2702b60fb2e42fdc50c2cee6a5046112da9a5703a548a4fa50477bc                                   |
| os                     | linux                                                                                              |
| format                 | elf                                                                                                |
| arch                   | amd64                                                                                              |
+------------------------+----------------------------------------------------------------------------------------------------+

no capabilities found

$ capa --rules tools/capa-rules/host-interaction/process/create/create-process-on-linux.yml path/to/redxor/po1kitd-update-k 
loading : 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1263.73 rules/s]
matching: 100%|████████████████████████████████████████████| 143/143 [00:00<00:00, 224.15 functions/s, skipped 0 library functions]
+------------------------+------------------------------------------------------------------------------------+
| md5                    | 7351f8a40c5450557b24622417fc478d                                                   |
| sha1                   | 8766d7e0c943ea66ebe90030617881a899b2aa11                                           |
| sha256                 | 0423258b94e8a9af58ad63ea493818618de2d8c60cf75ec7980edcaa34dcc919                   |
| os                     | linux                                                                              |
| format                 | elf                                                                                |
| arch                   | amd64                                                                              |
+------------------------+------------------------------------------------------------------------------------+

+-----------------------------+-------------------------------------------------------------------------------+
| MBC Objective               | MBC Behavior                                                                  |
|-----------------------------+-------------------------------------------------------------------------------|
| PROCESS                     | Create Process:: [C0017]                                                      |
+-----------------------------+-------------------------------------------------------------------------------+

+------------------------------------------------------+------------------------------------------------------+
| CAPABILITY                                           | NAMESPACE                                            |
|------------------------------------------------------+------------------------------------------------------|
| create process on Linux (4 matches)                  | host-interaction/process/create                      |
+------------------------------------------------------+------------------------------------------------------+
williballenthin commented 2 years ago

thanks for the thorough report here. likewise, i have a suspicion thats its a bug in the underlying analysis engine (vivisect). we're awaiting some substantial improvements to be released in the next few days, after which we'll release v3.0.1.

i will see if capa behaves better using a snapshot of vivisect from master. if that doesn't work, i'll dig into why it doesn't identify the API calls.

incidentally, to confirm the features that capa extracts, you can use the script show-features.py.

williballenthin commented 2 years ago

using viv/master (https://github.com/vivisect/vivisect/commit/d5b895eeeb19a0304d965cbf4fc86a5f3a5183c5) doesn't help.

using show-features we do see the import of popen: image

but, actually, it doesn't look like capa identifies the function: image

looking in IDA, the function is referenced by pointer and then maybe added to a list: image

i suspect the viv analysis doesn't recognize that the pointer is to a function (rather than data) so capa doesn't have a chance to analyze this function.

williballenthin commented 2 years ago

the function in question has a pretty clear prolog: image

so i wonder if we can extend the viv analysis a bit further to recognize the function pointers.

williballenthin commented 2 years ago

in vivisect funcentries we see the brute forcing of function entry points using the i386 signatures from here. notably, these signatures do not match the prolog we see here.

some options:

  1. extend the byte signatures to identify this prolog (and other common prologs on amd64, which we'd need to collect)
  2. add an analysis pass that is more targeted:
    • for each pointer to executable section, does it look like code?

notably, we can implement these analysis passes outside of viv, within capa, while we evaluate their effectiveness.

TcM1911 commented 2 years ago

Looking at the sample, it appears that it's been compiled with the -fomit-frame-pointer flag.

-fomit-frame-pointer Don't keep the frame pointer in a register for functions that don't need one. This avoids the instructions to save, set up and restore frame pointers; it also makes an extra register available in many functions. It also makes debugging impossible on some machines.

On some machines, such as the VAX, this flag has no effect, because the standard calling sequence automatically handles the frame pointer and nothing is saved by pretending it doesn't exist. The machine-description macro FRAME_POINTER_REQUIRED controls whether a target machine supports this flag. See Register Usage.

Starting with GCC version 4.6, the default setting (when not optimizing for size) for 32-bit GNU/Linux x86 and 32-bit Darwin x86 targets has been changed to -fomit-frame-pointer. The default can be reverted to -fno-omit-frame-pointer by configuring GCC with the --enable-frame-pointer configure option.

Enabled at levels -O, -O2, -O3, -Os.

This can make it hard to detect functions via a signature because the beginning can be different. Here are some function prologs from the same file:

 |             0x00404e80      4155                   push r13                                                           
│           0x00404e82      4154                   push r12                                                           
│           0x00404e84      4989fc                mov r12, rdi                ; arg1                                 
│           0x00404e87      55                       push rbp                                                           
│           0x00404e88      53                       push rbx                                                           
│           0x00404e89      4881ec580400.  sub rsp, 0x458                                                     
│           0x004038f0      55                   push rbp                                                           
│           0x004038f1      53                   push rbx                                                           
│           0x004038f2      4889fb            mov rbx, rdi                ; arg1                                 
│           0x004038f5      4883ec18       sub rsp, 0x18                                                      
│           0x0040f060      53              push rbx                                                           
│           0x0040f061      89fb           mov ebx, edi                ; arg1

No prolog

│           0x0040ecc0      89f0            mov eax, esi                ; arg2                                 
│           0x0040ecc2      c1e818       shr eax, 0x18                                                      
│           0x0040ecc5      85d2           test edx, edx               ; arg3                                 
│           0x0040ecc7      7417           je 0x40ece0

It looks like it can be: callee saving registers, subtracting the stack pointer, or non of it. It looks like only short functions doesn't have the push or sub instructions. Maybe it can be assumed to be a function if it points into the text section and a first set of instructions are decoded. If it results into a subtraction of the stack pointer, we could lock it in as a function. If we don't hit a sub instruction, we can decode a maximum of bytes until a ret in case it's a short function that doesn't use the stack.

I'm just spitballing ideas for detecting these functions without a run-off decoding.

atlas0fd00m commented 2 years ago

@williballenthin we just discovered a bug in Viv's ELF parsing that may help with some of these undiscovered functions as well. stay tuned! i have to run it through the PR process and clean up the unittests that are bound to break.

@