Troubleshooting: Installation on macOS

[bdemick]

I'm running into some issues installing pyhidra on macOS.

Environment

• OS: macOS 14.3
• JDK: Corretto-17.0.6.10.1
• Python: 3.12.0 (using pyenv)
• Ghidra 11

Troubleshooting

1. Followed the instructions in readme. GHIDRA_INSTALL_DIR is pointing to my local Ghidra 11 install folder. pip install pyhidra succeeds. pyhidraw command runs, returns 0, and I briefly see the Java/Ghidra icon bounce in the dock, but the UI is never launched
2. Run an interactive Python session. Can import pyhidra, but pyhidra.start() fails with:

   Traceback (most recent call last):
   File "<frozen runpy>", line 198, in _run_module_as_main
   File "<frozen runpy>", line 88, in _run_code
   File "/Users/usr/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pyhidra/install_plugins.py", line 9, in <module>
   pyhidra.DeferredPyhidraLauncher().start()
   File "/Users/ben/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pyhidra/launcher.py", line 233, in start
   from ghidra import GhidraLauncher
   ImportError: cannot import name 'GhidraLauncher' from 'ghidra' (unknown location)

3. Attempted reinstall with python -m pyhidra.uninstall_plugin pyhidra and python -m pyhidra.install_plugins. The install command fails with the same ImportError as above
4. Ran the install command via VSCode's Python debugger, and it failed much later on in the install process when running this way for some reason: pyhidra/javac.py raises ModuleNotFoundError for java.lang

Seems like something might be messed up in my Java/JDK configuration, but I'm having quite a time tracking it down. Ghidra 11 seems to be running just fine on its own, and launching it via ghidraRun it does detect the pyhidra extension.

Does anyone have other troubleshooting suggestions that I can try out?

[dc3-tsd]

If you haven't already, try updating jpype to 1.5.0 (https://github.com/jpype-project/jpype/releases/tag/v1.5.0). That release added support for Python 3.12

[bdemick]

Thanks! Looks like I'm already running that version, unfortunately. I did re-configure and point $GHIDRA_INSTALL_DIR to my 10.4 installation, and I'm now able to run headless analysis and use the library in the python console, but I'm still is unable to successfully run pyhidraw.

[loverics]

I have the same issue as @bdemick but if I go into my user's ghidra_scripts directory it works.

I have the official ghidra 11 release, with python 3.12.1 from pyenv on macos 14.3. I also have jpype 1.5. Finally pyhidraw fails to open ghidra.

[bdemick]

I just replicated the ghidra_scripts behavior with Ghidra 11. Then I went back and tested against a build > 10.4 but before the 11 release, and that seems to work just fine. Then I went back and created a virtual environment for pyhidra and pointed GHIDRA_INSTALL_DIR back to Ghidra 11, and within the virtual environment I can use pyhidra from any folder just fine (although it still won't launch pyhidraw).

Curiously, it now seems to be working via the interpreter just fine outside of the virtual environment using my pyenv-installed Python.

[dc3-tsd]

We just pushed Pyhidra 1.0.0 which may address this issue. If not, can you try starting up the gui with pyhidra -g instead? This should help to show any error logs coming from the Ghidra side.

[bdemick]

TLDR; new version seems to work as expected when installed in a virtual environment. When attempting to use it via pyenv's global Python, it still only partially works.

Uninstalled old version, installed 1.0.0 (using pyenv with Python 3.12). pyhidraw still fails in the same way. pyhidra -g launches Ghidra with the following log:

INFO  Using log config file: jar:file:/Users/ben/ghidra/ghidra_11.0_PUBLIC/Ghidra/Framework/Generic/lib/Generic.jar!/generic.log4j.xml (LoggingInitialization)
INFO  Using log file: /Users/ben/.ghidra/.ghidra_11.0_PUBLIC/application.log (LoggingInitialization)
INFO  Loading user preferences: /Users/ben/.ghidra/.ghidra_11.0_PUBLIC/preferences (Preferences)
INFO  Loading previous preferences: /Users/ben/.ghidra/.ghidra_10.4_PUBLIC/preferences (Preferences)
INFO  Searching for classes... (ClassSearcher)
INFO  Class search complete (553 ms) (ClassSearcher)
INFO  Initializing SSL Context (SSLContextInitializer)
INFO  Initializing Random Number Generator... (SecureRandomFactory)
INFO  Random Number Generator initialization complete: NativePRNGNonBlocking (SecureRandomFactory)
INFO  Trust manager disabled, cacerts have not been set (ApplicationTrustManagerFactory)
INFO  User ben started Ghidra. (GhidraRun)
INFO  Opening project: /Users/ben/ghidra_projects/test_pyhidra (DefaultProject)
INFO  restoring data storage index... (IndexJournal)

and I can open the Pyhidra interpreter window in the GUI. However, in the main (terminal) Python interpreter, I'm now seeing:

>>> import pyhidra
>>> pyhidra.start()
An error occured launching Ghidra: cannot import name 'GhidraApplicationLayout' from 'ghidra' (unknown location)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ben/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pyhidra/core.py", line 26, in start
    launcher.start()
  File "/Users/ben/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pyhidra/launcher.py", line 305, in start
    self._report_fatal_error("An error occured launching Ghidra", str(e), e)
  File "/Users/ben/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pyhidra/launcher.py", line 216, in _report_fatal_error
    raise cause
  File "/Users/ben/.pyenv/versions/3.12.0/lib/python3.12/site-packages/pyhidra/launcher.py", line 271, in start
    from ghidra import GhidraApplicationLayout
ImportError: cannot import name 'GhidraApplicationLayout' from 'ghidra' (unknown location)

I see the same traceback when running python -m pyhidra.install_plugins.

From a virtual environment, I can use the API successfully, and pyhidraw launches the GUI successfully. So as it stands, it appears to be fixed as long as I use it in the context of a virtual environment (which is probably a best practice, anyway). It's still unclear to me why it's failing at the pyenv/global level.

[dc3-tsd]

Could you try wrapping the import on line 49 in an if statement that checks platform.system() for OS X? This import was created for debugging purposes and appears to cause an issue in Mac environments. If making this change works we would appreciate if you could supply a pull request so we could test it on our side to verify and then merge it in if it works across all of our environments.

[bdemick]

Are you referring to the import at launcher.py:271 that's throwing the error? Not seeing any imports at line 49 in any of the files in the package. I did try wrapping that particular import in if platform.system() != 'Darwin' and launcher.py:_uninstall_old_plugin throws an error resolving the extension path (line 145, RuntimeError: extension_path cannot be obtained until launcher starts.).

[dc3-tsd]

Sorry for the earlier confusion. We were referring to https://github.com/dod-cyber-crime-center/pyhidra/blob/439e112fe2785e551328b52572199a202f7a221c/pyhidra/gui.py#L49

[bdemick]

Not sure that the release in PyPI matches what's in the repo. The gui.py in my site-packages looks nothing like that one (and not quite like the previous commit of that file in Github either). Just to sanity check, I ran pip download pyhidra and unzipped the pyhidra-1.0.0-py3-none-any.whl, and it matches what's in my site-packages. For reference, he's the gui.py that's coming from PyPI (see below). Something is off here.

import logging
import os
import platform
import sys
import warnings

from pyhidra import get_current_interpreter as _get_current_interpreter

logger = logging.getLogger(__name__)

def _gui():
    if platform.system() == 'Windows':
        # gui_script works like it is supposed to on windows
        gui()
        return

    pid = os.fork()
    if pid != 0:
        # original process can exit
        return

    fd = os.open(os.devnull, os.O_RDWR)
    # redirect stdin, stdout and stderr to /dev/null so the jvm can't use the terminal
    # this also prevents errors from attempting to write to a closed sys.stdout #21
    os.dup2(fd, sys.stdin.fileno(), inheritable=False)
    os.dup2(fd, sys.stdout.fileno(), inheritable=False)
    os.dup2(fd, sys.stderr.fileno(), inheritable=False)

    # run the application
    gui()

def gui():
    """
    Starts the Ghidra GUI
    """
    if "GHIDRA_INSTALL_DIR" not in os.environ:
        import tkinter.messagebox
        msg = "GHIDRA_INSTALL_DIR not set.\nPlease see the README for setup instructions"
        tkinter.messagebox.showerror("Improper Setup", msg)
        sys.exit()
    import pyhidra
    pyhidra.GuiPyhidraLauncher().start()

def get_current_interpreter():
    warnings.warn(
        "get_current_interpreter has been moved. Please use pyhidra.get_current_interpreter",
        DeprecationWarning
    )
    return _get_current_interpreter()

[dc3-tsd]

Thank you for catching this. It looks like we ran into an error in our pipeline earlier when tagging and pushing to GitHub which then pushes to PyPi. The 1.0.0 tag has been fixed on GitHub and removed from PyPi to avoid confusion. A 1.0.1 release has been pushed and is now available on both GitHub and PyPi.

[bdemick]

Just tested it out - no change in behavior, but adding the macOS guard around that tkinter.messagebox import does result in pyhidraw working as expected in the "global" Python context. Virtual environment-based installs still work fine with and without the change. I'll go ahead and fire up a PR.