search

NationalSecurityAgency / ghidra

Ghidra is a software reverse engineering (SRE) framework
https://www.nsa.gov/ghidra
Apache License 2.0
50.43k stars 5.77k forks source link

analyzeHeadless with processor option seems to ignore symbols #2293

Open jstucke opened 3 years ago

jstucke commented 3 years ago

Describe the bug

When running scripts on ARM binaries in headless mode and explicitly stating the processor with -processor ARM:LE:32:v8, Ghidra treats the binary as if it was stripped if any other ARM version than "v8" is used (tested with v4t, v5t, v7 and "Cortex"). This effect does not occur when running the script manually through the GUI (regardless of the selected processor/architecture).

To Reproduce

1. Compile Minimal Demo Binary

Create hello_world.c with content

#include <stdio.h>

int main() {
    printf("Hello, World!");
    return 0;
}
> arm-linux-gnueabi-gcc -O0 hello_world.c -o hello_world.elf
> arm-linux-gnueabi-objdump -x hello_world.elf | grep arch
architecture: armv5t, flags 0x00000112:

(binary is compiled for armv5t and not stripped)

2. Create Minimal Demo Script

Create test.py with content:

from __future__ import print_function

def iterate_functions():
    function = getFirstFunction()
    while function is not None:
        yield function
        function = getFunctionAfter(function)

def main():
    for function in iterate_functions():
        print("Name:", function.name)

if __name__ == '__main__':
    main()

3. Run Ghidra Headless with Script

> analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py -processor ARM:LE:32:v8
> analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py -processor ARM:LE:32:v5t

Expected Behavior

The script should print the function names (like it does with -processor ARM:LE:32:v8):

> analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py -processor ARM:LE:32:v8
(...)
Name: _init
Name: printf
Name: __libc_start_main
Name: abort
Name: _start
Name: call_weak_fn
Name: deregister_tm_clones
Name: register_tm_clones
Name: __do_global_dtors_aux
Name: frame_dummy
Name: main
Name: __libc_csu_init
Name: __libc_csu_fini
Name: _fini
Name: __gmon_start__
Name: printf
Name: abort
Name: __libc_start_main

Actual Behavior

When passing -processor ARM:LE:32:v5t Ghidra seems to behave as if the binary was stripped (not all functions were found and the symbols are gone) -- or maybe I'm just doing it wrong:

> analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py -processor ARM:LE:32:v5t
(...)
Name: _DT_INIT
Name: FUN_000002ec
Name: FUN_00000304
Name: FUN_0000034c
Name: _DT_FINI

Environment

ghidra1 commented 3 years ago

Could you please provide the full log output from the analyzeHeadless. Could you also please share your binary to speed things up for us. I tried one of our binaries and could not reproduce the problem.

jstucke commented 3 years ago

Could you please provide the full log output from the analyzeHeadless.

Here is the log for -processor ARM:LE:32:v5t:

> analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py -processor ARM:LE:32:v5t
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "ghidra.GhidraClassLoader"). To use archived non-system classes, this property must not be set
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7-Ubuntu-1ubuntu1)
OpenJDK 64-Bit Server VM (build 14.0.1+7-Ubuntu-1ubuntu1, mixed mode, sharing)
INFO  Using log config file: jar:file:/opt/ghidra_9.1.2_PUBLIC/Ghidra/Framework/Generic/lib/Generic.jar!/generic.log4j.xml (LoggingInitialization)  
INFO  Using log file: /home/jstucke/.ghidra/.ghidra_9.1.2_PUBLIC/application.log (LoggingInitialization)  
INFO  Loading user preferences: /home/jstucke/.ghidra/.ghidra_9.1.2_PUBLIC/preferences (Preferences)  
INFO  Class search complete (510 ms) (ClassSearcher)  
INFO  Initializing SSL Context (SSLContextInitializer)  
INFO  Initializing Random Number Generator... (SecureRandomFactory)  
INFO  Random Number Generator initialization complete: NativePRNGNonBlocking (SecureRandomFactory)  
INFO  Trust manager disabled, cacerts have not been set (ApplicationTrustManagerFactory)  
INFO  HEADLESS Script Paths:
    /home/jstucke/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Base/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/BytePatterns/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Decompiler/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/FileFormats/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/FunctionID/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/GnuDemangler/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Python/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/VersionTracking/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/8051/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/DATA/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/PIC/ghidra_scripts (HeadlessAnalyzer)  
INFO  HEADLESS: execution starts (HeadlessAnalyzer)  
INFO  Creating temporary project: /home/jstucke/test/ghidra/./test_project (HeadlessAnalyzer)  
INFO  Creating project: /home/jstucke/test/ghidra/./test_project (DefaultProject)  
INFO  REPORT: Processing input files:  (HeadlessAnalyzer)  
INFO       project: /home/jstucke/test/ghidra/./test_project (HeadlessAnalyzer)  
INFO  IMPORTING: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$2 (file:/opt/ghidra_9.1.2_PUBLIC/Ghidra/Framework/Generic/lib/cglib-nodep-2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO  REPORT: Import succeeded with language "ARM:LE:32:v5t" and cspec "default" for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  ANALYZING all memory and code: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  Packed database cache: /tmp/jstucke-Ghidra/packed-db-cache (PackedDatabaseCache)  
INFO  -----------------------------------------------------
    ARM Constant Reference Analyzer            0.019 secs
    ARM Pre-Pattern Analyzer                   0.003 secs
    ARM Symbol                                 0.001 secs
    ASCII Strings                              0.318 secs
    Apply Data Archives                        0.291 secs
    Call Convention Identification             0.015 secs
    Call-Fixup Installer                       0.001 secs
    Create Address Tables                      0.007 secs
    Create Function                            0.019 secs
    Data Reference                             0.001 secs
    Decompiler Switch Analysis                 0.093 secs
    Demangler                                  0.016 secs
    Disassemble                                0.009 secs
    Disassemble Entry Points                   0.044 secs
    Disassemble Entry Points - One Time        0.010 secs
    ELF Header Annotation                      0.140 secs
    Embedded Media                             0.010 secs
    External Entry References                  0.000 secs
    Function Start Search                      0.047 secs
    Function Start Search After Code           0.000 secs
    Function Start Search After Data           0.000 secs
    Function Start Search delayed - One Time   0.000 secs
    Non-Returning Functions - Discovered       0.007 secs
    Reference                                  0.001 secs
    Stack                                      0.005 secs
    Subroutine References                      0.002 secs
-----------------------------------------------------
     Total Time   1 secs
-----------------------------------------------------
 (AutoAnalysisManager)  
INFO  REPORT: Analysis succeeded for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  SCRIPT: /home/jstucke/test/ghidra/test.py (HeadlessAnalyzer)  
Name: _DT_INIT
Name: FUN_000002ec
Name: FUN_00000304
Name: FUN_0000034c
Name: _DT_FINI
INFO  ANALYZING changes made by post scripts: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  REPORT: Post-analysis succeeded for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  REPORT: Save succeeded for file: /hello_world.elf (HeadlessAnalyzer)

And here is the output for -processor ARM:LE:32:v8: 

analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py -processor ARM:LE:32:v8
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "ghidra.GhidraClassLoader"). To use archived non-system classes, this property must not be set
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7-Ubuntu-1ubuntu1)
OpenJDK 64-Bit Server VM (build 14.0.1+7-Ubuntu-1ubuntu1, mixed mode, sharing)
INFO  Using log config file: jar:file:/opt/ghidra_9.1.2_PUBLIC/Ghidra/Framework/Generic/lib/Generic.jar!/generic.log4j.xml (LoggingInitialization)  
INFO  Using log file: /home/jstucke/.ghidra/.ghidra_9.1.2_PUBLIC/application.log (LoggingInitialization)  
INFO  Loading user preferences: /home/jstucke/.ghidra/.ghidra_9.1.2_PUBLIC/preferences (Preferences)  
INFO  Class search complete (442 ms) (ClassSearcher)  
INFO  Initializing SSL Context (SSLContextInitializer)  
INFO  Initializing Random Number Generator... (SecureRandomFactory)  
INFO  Random Number Generator initialization complete: NativePRNGNonBlocking (SecureRandomFactory)  
INFO  Trust manager disabled, cacerts have not been set (ApplicationTrustManagerFactory)  
INFO  HEADLESS Script Paths:
    /home/jstucke/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Base/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/BytePatterns/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Decompiler/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/FileFormats/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/FunctionID/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/GnuDemangler/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Python/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/VersionTracking/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/8051/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/DATA/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/PIC/ghidra_scripts (HeadlessAnalyzer)  
INFO  HEADLESS: execution starts (HeadlessAnalyzer)  
INFO  Creating temporary project: /home/jstucke/test/ghidra/./test_project (HeadlessAnalyzer)  
INFO  Creating project: /home/jstucke/test/ghidra/./test_project (DefaultProject)  
INFO  REPORT: Processing input files:  (HeadlessAnalyzer)  
INFO       project: /home/jstucke/test/ghidra/./test_project (HeadlessAnalyzer)  
INFO  IMPORTING: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$2 (file:/opt/ghidra_9.1.2_PUBLIC/Ghidra/Framework/Generic/lib/cglib-nodep-2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO  REPORT: Import succeeded with language "ARM:LE:32:v8" and cspec "default" for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  ANALYZING all memory and code: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  Packed database cache: /tmp/jstucke-Ghidra/packed-db-cache (PackedDatabaseCache)  
INFO  REPORT: Analysis succeeded for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  SCRIPT: /home/jstucke/test/ghidra/test.py (HeadlessAnalyzer)  
Name: _init
Name: printf
Name: __libc_start_main
Name: abort
Name: _start
Name: call_weak_fn
Name: deregister_tm_clones
Name: register_tm_clones
Name: __do_global_dtors_aux
Name: frame_dummy
Name: main
Name: __libc_csu_init
Name: __libc_csu_fini
Name: _fini
Name: __gmon_start__
Name: printf
Name: abort
Name: __libc_start_main
INFO  ANALYZING changes made by post scripts: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  REPORT: Post-analysis succeeded for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  REPORT: Save succeeded for file: /hello_world.elf (HeadlessAnalyzer)

Could you also please share your binary to speed things up for us.

test_files.zip

ghidra1 commented 3 years ago

It is curious that the v8 import does not dump the analysis results. Was this removed from the v8 log output? Not sure why it would be missing.

jstucke commented 3 years ago

It is curious that the v8 import does not dump the analysis results. Was this removed from the v8 log output? Not sure why it would be missing.

I was also wondering about that. But analyses like function search shouldn't be necessary, because the binary is not stripped. I did not omit anything from the log. It is the same behaviour as without the -processor parameter:

> analyzeHeadless . test_project -deleteProject -import hello_world.elf -postScript test.py
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "ghidra.GhidraClassLoader"). To use archived non-system classes, this property must not be set
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7-Ubuntu-1ubuntu1)
OpenJDK 64-Bit Server VM (build 14.0.1+7-Ubuntu-1ubuntu1, mixed mode, sharing)
INFO  Using log config file: jar:file:/opt/ghidra_9.1.2_PUBLIC/Ghidra/Framework/Generic/lib/Generic.jar!/generic.log4j.xml (LoggingInitialization)  
INFO  Using log file: /home/jstucke/.ghidra/.ghidra_9.1.2_PUBLIC/application.log (LoggingInitialization)  
INFO  Loading user preferences: /home/jstucke/.ghidra/.ghidra_9.1.2_PUBLIC/preferences (Preferences)  
INFO  Class search complete (434 ms) (ClassSearcher)  
INFO  Initializing SSL Context (SSLContextInitializer)  
INFO  Initializing Random Number Generator... (SecureRandomFactory)  
INFO  Random Number Generator initialization complete: NativePRNGNonBlocking (SecureRandomFactory)  
INFO  Trust manager disabled, cacerts have not been set (ApplicationTrustManagerFactory)  
INFO  HEADLESS Script Paths:
    /home/jstucke/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Base/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/BytePatterns/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Decompiler/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/FileFormats/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/FunctionID/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/GnuDemangler/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/Python/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Features/VersionTracking/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/8051/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/DATA/ghidra_scripts
    /opt/ghidra_9.1.2_PUBLIC/Ghidra/Processors/PIC/ghidra_scripts (HeadlessAnalyzer)  
INFO  HEADLESS: execution starts (HeadlessAnalyzer)  
INFO  Creating temporary project: /home/jstucke/test/ghidra/./test_project (HeadlessAnalyzer)  
INFO  Creating project: /home/jstucke/test/ghidra/./test_project (DefaultProject)  
INFO  REPORT: Processing input files:  (HeadlessAnalyzer)  
INFO       project: /home/jstucke/test/ghidra/./test_project (HeadlessAnalyzer)  
INFO  IMPORTING: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$2 (file:/opt/ghidra_9.1.2_PUBLIC/Ghidra/Framework/Generic/lib/cglib-nodep-2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO  REPORT: Import succeeded with language "ARM:LE:32:v8" and cspec "default" for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  ANALYZING all memory and code: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  Packed database cache: /tmp/jstucke-Ghidra/packed-db-cache (PackedDatabaseCache)  
INFO  REPORT: Analysis succeeded for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  SCRIPT: /home/jstucke/test/ghidra/test.py (HeadlessAnalyzer)  
Name: _init
Name: printf
Name: __libc_start_main
Name: abort
Name: _start
Name: call_weak_fn
Name: deregister_tm_clones
Name: register_tm_clones
Name: __do_global_dtors_aux
Name: frame_dummy
Name: main
Name: __libc_csu_init
Name: __libc_csu_fini
Name: _fini
Name: __gmon_start__
Name: printf
Name: abort
Name: __libc_start_main
INFO  ANALYZING changes made by post scripts: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  REPORT: Post-analysis succeeded for file: /home/jstucke/test/ghidra/hello_world.elf (HeadlessAnalyzer)  
INFO  REPORT: Save succeeded for file: /hello_world.elf (HeadlessAnalyzer)