search

NationalSecurityAgency / ghidra

Ghidra is a software reverse engineering (SRE) framework
https://www.nsa.gov/ghidra
Apache License 2.0
50.85k stars 5.8k forks source link

Can't build swig bindings for lldb #6628

Open heartpunk opened 3 months ago

heartpunk commented 3 months ago

Tried following the instructions, tried everything could find in all the relevant issues reported here. Retried again in a fresh install, here's a transcript of what happened. Please help!!!

heartpunk@REDACTED-MacBook-Pro ghidra_11.1_PUBLIC % LLVM_HOME=/Users/heartpunk/llvm-project GHIDRA_INSTALL_DIR=`pwd` ./Ghidra/Debug/Debugger-swig-lldb/macos_debugger_lldb_build_from_brew.sh --verbose --debug
+ '[' -z /Users/heartpunk/ghidra_11.1_PUBLIC ']'
+ '[' '!' -z /Users/heartpunk/ghidra_11.1_PUBLIC ']'
+ pushd /Users/heartpunk/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb
~/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb ~/ghidra_11.1_PUBLIC
+ LLVM_VERSION=16
+ brew install llvm@16
Warning: llvm@16 16.0.6_1 is already installed and up-to-date.
To reinstall 16.0.6_1, run:
  brew reinstall llvm@16
++ mktemp -d
+ LLVM_TEMP_DIR=/var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R
+ brew unpack --patch --destdir /var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R llvm@16
==> Unpacking llvm@16 to: /var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R/llvm@16-16.0.6
==> Downloading https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.6/llvm-project-16.0.6.src.tar.xz
Already downloaded: /Users/heartpunk/Library/Caches/Homebrew/downloads/8cbb1836057b964f8f1bea2eee120bd27a78aa698ba2d5263996f3071234b47e--llvm-project-16.0.6.src.tar.xz
++ echo /var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R/llvm@16-16.0.6
+ export LLVM_HOME=/var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R/llvm@16-16.0.6
+ LLVM_HOME=/var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R/llvm@16-16.0.6
+ '[' '!' -d /var/folders/sc/zl1g6ny95zj9lv5zvn7s7dm40000gn/T/tmp.ASCr9z4B3R/llvm@16-16.0.6 ']'
++ brew --prefix llvm@16
+ BREW_LLVM=/opt/homebrew/opt/llvm@16
+ export 'LDFLAGS=-L/opt/homebrew/opt/llvm@16/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm@16/lib/c++,-L/opt/homebrew/opt/llvm@16/lib'
+ LDFLAGS='-L/opt/homebrew/opt/llvm@16/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm@16/lib/c++,-L/opt/homebrew/opt/llvm@16/lib'
+ export 'PATH=/opt/homebrew/opt/llvm@16/bin:/Users/heartpunk/llvm-bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2/bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2@global/bin:/Users/heartpunk/.rvm/rubies/ruby-3.1.2/bin:/Users/heartpunk/.local/bin:/Users/heartpunk/.cabal/bin:/Users/heartpunk/.ghcup/bin:/Users/heartpunk/.nvm/versions/node/v18.13.0/bin:/Users/heartpunk/.pyenv/shims:/opt/homebrew/opt/python@3.10/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.1.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/go/bin:/Users/heartpunk/.cargo/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/heartpunk/.local/bin:/Users/heartpunk/.rvm/bin:/Users/heartpunk/go/bin:/Users/heartpunk/.dirtybin/'
+ PATH='/opt/homebrew/opt/llvm@16/bin:/Users/heartpunk/llvm-bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2/bin:/Users/heartpunk/.rvm/gems/ruby-3.1.2@global/bin:/Users/heartpunk/.rvm/rubies/ruby-3.1.2/bin:/Users/heartpunk/.local/bin:/Users/heartpunk/.cabal/bin:/Users/heartpunk/.ghcup/bin:/Users/heartpunk/.nvm/versions/node/v18.13.0/bin:/Users/heartpunk/.pyenv/shims:/opt/homebrew/opt/python@3.10/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.1.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/go/bin:/Users/heartpunk/.cargo/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/heartpunk/.local/bin:/Users/heartpunk/.rvm/bin:/Users/heartpunk/go/bin:/Users/heartpunk/.dirtybin/'
+ export CPPFLAGS=-I/opt/homebrew/opt/llvm@16/include
+ CPPFLAGS=-I/opt/homebrew/opt/llvm@16/include
++ echo /opt/homebrew/opt/llvm@16
+ export LLVM_BUILD=/opt/homebrew/opt/llvm@16
+ LLVM_BUILD=/opt/homebrew/opt/llvm@16
+ gradle buildNatives

> Configure project :Debugger-swig-lldb
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Configure project :Decompiler
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Configure project :FileFormats
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Configure project :PDB
OS: Mac OS X
Arch: aarch64
Using platform: mac_arm_64

> Task :Debugger-swig-lldb:compileMainMac_arm_64SharedLibraryMainCpp FAILED
/Users/heartpunk/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb/src/main/cpp/LLDBWrapJava.cpp:317:10: fatal error: 'lldb/API/SBScriptObject.h' file not found
#include "lldb/API/SBScriptObject.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':Debugger-swig-lldb:compileMainMac_arm_64SharedLibraryMainCpp'.
> A build operation failed.
      C++ compiler failed while compiling LLDBWrapJava.cpp.
  See the complete log at: file:///Users/heartpunk/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-swig-lldb/build/tmp/compileMainMac_arm_64SharedLibraryMainCpp/output.txt
   > C++ compiler failed while compiling LLDBWrapJava.cpp.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 1s
2 actionable tasks: 2 executed
<-------------> 0% WAITING
> IDLE
heartpunk@REDACTED-MacBook-Pro ghidra_11.1_PUBLIC % system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 14.5 (23F79)
      Kernel Version: Darwin 23.5.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: REDACTED's MacBook Pro
      User Name: REDACTED (heartpunk)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 3 days, 6 hours, 55 minutes

heartpunk@REDACTED ghidra_11.1_PUBLIC %
heartpunk commented 3 months ago 
/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applica
tions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09) 
[Clang 15.0.0 (clang-1500.3.9.4)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-1316.0.21.2.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
(lldb) script import ghidralldb
(lldb) target create "REDACTED"
Current executable set to 'REDACTED' (arm64
).
(lldb) ghidra trace connect "127.0.0.1:62869"
(lldb) ghidra trace start
(lldb) ghidra trace sync-enable
(lldb) process launch --stop-at-entry
add listener for process failed
add initial events for cli failed
add initial events for target failed
add initial events for process failed
Process 59513 exited with status = -1 (0xffffffff) attach failed (Not allowed to at
tach to process.  Look in the console messages (Console.app), near the debugserver 
entries, when the attach failed.  The subsystem that denied the attach permission w
ill likely have logged an informative message about why it was denied.)
Process 59513 launched: 'REDACTED' (arm64)
(lldb)
d-millar commented 3 months ago

Yep re redaction. At this point, I'd see if you can repeat the tutorial results. Possibly before tackling your own target.

heartpunk commented 3 months ago

Ahh, good idea. I'll do that. The above paste is from my target.

heartpunk commented 3 months ago

(Ahh, come to think of it, the above is almost certainly just SIP being on still.)

d-millar commented 3 months ago

AHHH, that looks great! The permissions thing is standard Apple security stuff. Basically, lldb has to have debugging permissions (which it should unless you built it from scratch) and the target has to have "allow debugging".

d-millar commented 3 months ago

Yes, re SIP - although if I remember correctly, you can enable per-target debugger with SIP still enabled. Will do some googling.

heartpunk commented 3 months ago

Oh if you find a pointer for that it'd be great. I'd prefer to not disable SIP unless necessary!

d-millar commented 3 months ago

OK, took me a while but....

Copy your target somewhere not in a system/system-related directory. If it's a package, you may want to extract the arm64 or x64 executable from it. Make sure the copy is still executable. Then "codesign -s lldb_codesign -f target" (where lldb_codesign is a certificate I'd already made with debug permissions - details here https://stackoverflow.com/questions/58356844/what-are-the-ways-or-technologies-to-sign-an-executable-application-file-in-mac ). Should (fingers-crossed) be debuggable then.

heartpunk commented 3 months ago

This is probably starting to get into a new issue ... oh right I forgot to do the test targets first but, since I've got the info anyway I'll share it!

The trace REDACTED.3 has no mapping to its program AARCH64-64-cpu0x0
---------------------------------------------------
Build Date: 2024-Jun-07 1416 EDT
Ghidra Version: 11.1
Java Home: /opt/homebrew/Cellar/openjdk/22.0.1/libexec/openjdk.jdk/Contents/Home
JVM Version: Homebrew 22.0.1
OS: Mac OS X 14.5 aarch64

It seems like the program is partly launching to debug but partly not? Unclear!

The ghidra terminal output for completness' sake:

/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applicat
ions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09) 
[Clang 15.0.0 (clang-1500.3.9.4)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-1316.0.21.2.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
(lldb) script import ghidralldb
(lldb) target create "/Users/heartpunk/REDACTED"
Current executable set to '/Users/heartpunk/REDACTED' (arm64).
(lldb) ghidra trace connect "127.0.0.1:64173"
(lldb) ghidra trace start
(lldb) ghidra trace sync-enable
add listener for process failed
add initial events for cli failed
add initial events for target failed
add initial events for process failed
(lldb) process launch
Process 6231 launched: '/Users/heartpunk/REDACTED' (arm64)
2024-06-12 11:48:41.032869-0700 REDACTED[6231:19698698] Failed to check bundle ID.
Process 6231 exited with status = 173 (0x000000ad) 
(lldb) Couldn't record page with PC: Cannot convert '$pc' to address: unable to eval
uate expression while the process is exited: the process must be stopped because the
 expression might require allocating memory.
Couldn't record page with SP: Cannot convert '$sp' to address: unable to evaluate ex
pression while the process is exited: the process must be stopped because the expres
sion might require allocating memory.
heartpunk commented 3 months ago

(the codesign thing worked! wouldn't be here w/o it.)

d-millar commented 3 months ago

OK, so looks like your process terminated - did you use the "process launch --stop-at-entry" option?

heartpunk commented 3 months ago

It seems to quite self evidently be working now!!! This looks just like a debugger should.

heartpunk commented 3 months ago

It does die as soon as I resume it though, huh.

d-millar commented 3 months ago

hmmm, well, first thing I woould check - does it behave that way in lldb w/o Ghidra or jusst in Ghidra? If just in Ghidra, one of two (?) things is happening. Either something went wrong in the Ghidra logic (possible) or we haven't registered a break for some exception that lldb has. It's been a while since I've looked at the lldb commands for handling execptions and events. I thought we default to break for most of them, but would have to check. All of that would also have to be handled through the command-line. Am sure we don't have GUI elements for those yet.

If Ghidra went south, there should be errors in either the Debug Console in the GUI or the ghidraDebug shell if you're using it.

If lldb's behavior is the same, you either need to set up the exception handling or some set of breakpoints. Also, could be that the target has detected the debugger and is punting (if it's that kind of app).

d-millar commented 3 months ago

"Use breakpoint set -E c++ to break on all exceptions and breakpoint set -F std::range_error to break on a specific exception." from stackoverflow

heartpunk commented 3 months ago

Giving this a try now!

heartpunk commented 3 months ago

two things:

heartpunk@REDACTED ~ % lldb
(lldb) script import ghidralldb
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ModuleNotFoundError: No module named 'ghidralldb'
(lldb) ^D
d-millar commented 3 months ago

Regarding the missing mapping, there could be a lot of reasons for that. Basically, the error means there's a mismatch between the name known to the OS while executing and the name of the program in Ghidra. Could be because the program was renamed after it was compiled. Could have been renamed on import. Could have been renamed in Ghidra after it was imported. In all of these cases, you can open the Modules window, find the module that is a match, and right-click to "Map Module to....", which will bring up a dialog. Select the entry in there, hit OK, and that association will be memoized, allowing the debugger to translate between dynamic and static addresses.

On the second point, I thought we solved that, no? Or are you running lldb from the command line? I can't quite tell from your post. In any case, you have to make sure that either the PYTHONPATH environment variable contains the paths to pypkg/src in Debugger-rmi-trace and Debugger-agent-lldb, or you have to install those packages in lldb's version of python using "python -m pip install" or the equivalent.

teta2k commented 3 months ago

I had a similar issue where psutil and protobuf weren't installed. What's interesting is that while psutil shouted ModuleNotFound, protobuf didn't and rather returned ghidra is not a valid command.

heartpunk commented 2 months ago

Regarding the missing mapping, there could be a lot of reasons for that. Basically, the error means there's a mismatch between the name known to the OS while executing and the name of the program in Ghidra. Could be because the program was renamed after it was compiled. Could have been renamed on import. Could have been renamed in Ghidra after it was imported. In all of these cases, you can open the Modules window, find the module that is a match, and right-click to "Map Module to....", which will bring up a dialog. Select the entry in there, hit OK, and that association will be memoized, allowing the debugger to translate between dynamic and static addresses.

ok, cool

On the second point, I thought we solved that, no? Or are you running lldb from the command line? I can't quite tell from your post. In any case, you have to make sure that either the PYTHONPATH environment variable contains the paths to pypkg/src in Debugger-rmi-trace and Debugger-agent-lldb, or you have to install those packages in lldb's version of python using "python -m pip install" or the equivalent.

that was from the terminal, the paste you're responding to. when I run it from ghidra, the ghidra-internal terminal gives output like:

/Applications/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-rmi-trace/pypkg/src:/Applica
tions/ghidra_11.1_PUBLIC/Ghidra/Debug/Debugger-agent-lldb/pypkg/src:
3.9.6 (default, Mar 29 2024, 10:51:09) 
[Clang 15.0.0 (clang-1500.3.9.4)]
3.11.0b5 (main, Aug 16 2022, 09:17:48) [Clang 13.1.6 (clang-1316.0.21.2.5)]
(lldb) version
lldb-1500.0.404.7
Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
(lldb) script import ghidralldb
(lldb) target create "/Users/heartpunk/REDACTED"
Current executable set to '/Users/heartpunk/REDACTED' (arm64).
(lldb) ghidra trace connect "127.0.0.1:61685"
(lldb) ghidra trace start
(lldb) ghidra trace sync-enable
(lldb) process launch --stop-at-entry
add listener for process failed
add initial events for cli failed
add initial events for target failed
add initial events for process failed
Process 59991 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0x0000000100044b70 dyld`_dyld_start
dyld`:
->  0x100044b70 <+0>:  mov    x0, sp
    0x100044b74 <+4>:  and    sp, x0, #0xfffffffffffffff0
    0x100044b78 <+8>:  mov    x29, #0x0
    0x100044b7c <+12>: mov    x30, #0x0
Target 0: (REDACTED) stopped.
Process 59991 launched: '/Users/heartpunk/REDACTED' (arm64)
(lldb)

Perhaps this is just anti reversing stuff, and that's my own problem to look into, perhaps not.

d-millar commented 2 months ago

I don’t see an obvious Ghidra-related error in your last post. Above, the “module not found” error after “script import ghidralldb” is a problem, but that doesn’t appear in your last set of messages.

I think from a Ghidra perspective you’re ok. The two tests I might try: (1) debug something innocuous (like xclock) that will be obviously visible on launch and doesn’t terminate immediately, and (2) debug REDACTED in lldb using “target create” and “process launch —stop-at-entry” and see if you get the same SIGSTOP messages.