ImportError: cannot import name 'KS_ARCH_ARM64' from 'keystone' in Apple Silicon M1

rustammendel commented 8 months ago

What is the problem? (Here is where you provide a complete Traceback.)

Any ofrak command fails with the error below:

File <...>/lib/python3.9/site-packages/ofrak/service/assembler/assembler_service_keystone.py", line 3, in <module>
    from keystone import (
ImportError: cannot import name 'KS_ARCH_ARM64' from 'keystone' <...>

Please provide some information about your environment. At minimum we would like the following information on your platform and Python environment:

Platform: macOS-14.3-arm64-arm-64bit
Python environment: python3.9

If you've discovered it, what is the root cause of the problem? not sure How often does the issue happen? always What are the steps to reproduce the issue?

pip install ofrak pip install ofrak_gdhira ofrak deps //throw keystone not found pip install keystone // openssl/aes.h not found brew install openssl export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include" pip install keystone ofrak deps //throws the error above about AARCH64 brew install cmake // didn't fix anything

How would you implement this fix?

Are there any (reasonable) alternative approaches?

Are you interested in implementing it yourself?

whyitfor commented 8 months ago

Is it possible that keystone is not installed? What is the output of pip freeze?

rustammendel commented 8 months ago

I installed it by pip install keystone ( as I mentioned in What are the steps to reproduce the issue? section.)

pip freeze out: absl-py==2.0.0 ailment==9.2.6 aiohttp==3.8.6 aiohttp-cors==0.7.0 aiosignal==1.3.1 alembic==1.13.1 amqp==5.2.0 angr==9.2.6 aniso8601==9.0.1 anyio==4.2.0 appnope==0.1.3 archinfo==9.2.6 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 astcheck==0.4.0 asttokens==2.4.1 async-lru==2.0.4 async-timeout==4.0.3 attrs==23.2.0 Babel==2.14.0 bcrypt==4.1.2 beartype==0.12.0 beautifulsoup4==4.12.3 bitarray==2.9.2 bitstring==4.1.4 black==23.3.0 bleach==6.1.0 blinker==1.7.0 bokeh==3.3.4 cachetools==5.3.2 capstone==4.0.2 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 chipwhisperer==5.7.0 claripy==9.2.6 cle==9.2.6 click==8.1.7 cloudpickle==3.0.0 cmake==3.28.3 colorcet==3.0.1 comm==0.2.1 configobj==5.0.8 contourpy==1.2.0 coverage==6.5.0 CppHeaderParser==2.7.4 cryptography==42.0.4 cstruct==5.3 cycler==0.12.1 Cython==3.0.8 dask==2024.1.1 datashader==0.16.0 debtcollector==2.5.0 debugpy==1.8.0 decorator==5.1.1 defusedxml==0.7.1 dm-tree==0.1.8 dnspython==2.6.1 dogpile.cache==1.3.2 dpkt==1.9.8 ECPy==1.2.5 elementpath==4.3.0 entrypoints==0.4 eventlet==0.35.2 exceptiongroup==1.2.0 executing==2.0.1 fastdtw==0.3.4 fasteners==0.19 fastjsonschema==2.19.1 fdt==0.3.3 fixtures==4.1.0 Flask==3.0.2 Flask-RESTful==0.3.10 fonttools==4.47.2 fqdn==1.5.1 freetype-py==2.4.0 frozenlist==1.4.1 fsspec==2023.12.2 fun-coverage==0.2.0 future==1.0.0 futurist==2.4.1 gitdb==4.0.11 GitPython==3.1.32 greenlet==3.0.3 h5py==3.10.0 holoviews==1.18.1 hsluv==5.0.4 hypothesis==6.39.6 idna==3.6 immutabledict==2.2.0 importlib-metadata==4.13.0 importlib-resources==6.1.1 iniconfig==2.0.0 intervaltree==3.1.0 ipykernel==6.29.0 ipython==8.18.1 ipython-genutils==0.2.0 ipywidgets==8.1.1 iso8601==2.1.0 isoduration==20.11.0 itanium-demangler==1.1 itsdangerous==2.1.2 jedi==0.19.1 jefferson==0.4.5 Jinja2==3.1.3 joblib==1.3.2 json5==0.9.14 jsonpointer==2.4 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 jupyter==1.0.0 jupyter-bokeh==3.0.7 jupyter-console==6.6.3 jupyter-contrib-core==0.4.2 jupyter-contrib-nbextensions==0.7.0 jupyter-events==0.9.0 jupyter-highlight-selected-word==0.2.0 jupyter-lsp==2.2.2 jupyter-nbextensions-configurator==0.6.3 jupyter_client==8.6.0 jupyter_core==5.7.1 jupyter_server==2.12.5 jupyter_server_terminals==0.5.2 jupyterlab==4.0.12 jupyterlab-widgets==3.0.9 jupyterlab_pygments==0.3.0 jupyterlab_server==2.25.2 keras==3.0.2 keystone-engine==0.9.2 keystoneauth1==5.5.0 keystonemiddleware==10.5.0 kiwisolver==1.4.5 kombu==5.3.5 lascar==1.1 libusb1==3.1.0 lief==0.12.3 linkify-it-py==2.0.2 llvmlite==0.42.0 locket==1.0.0 lxml==5.1.0 lzallright==0.2.4 Mako==1.3.2 Markdown==3.5.2 markdown-it-py==3.0.0 MarkupSafe==2.1.5 matplotlib==3.8.2 matplotlib-inline==0.1.6 mdit-py-plugins==0.4.0 mdurl==0.1.2 mistune==3.0.2 mpmath==1.3.0 msgpack==1.0.7 mulpyplexer==0.9 multidict==6.0.5 multipledispatch==1.0.0 mypy==0.942 mypy-extensions==1.0.0 namex==0.0.7 nampa==0.1.1 nbclassic==1.0.0 nbclient==0.9.0 nbconvert==7.14.2 nbformat==5.9.2 nbparameterise==0.6 nbstripout==0.6.1 nest-asyncio==1.6.0 netaddr==1.2.1 netifaces==0.11.0 networkx==3.2.1 notebook==7.0.7 notebook_shim==0.2.3 numba==0.59.0 numpy==1.26.3 oauthlib==3.2.2 ofrak==3.2.0.post0 ofrak_angr==1.0.1 ofrak_capstone==1.0.0 ofrak_ghidra==0.0.1 ofrak_io==1.1.0 ofrak_patch_maker==4.0.2 ofrak_type==2.2.0 orjson==3.8.14 os-service-types==1.7.0 oslo.cache==3.6.0 oslo.concurrency==5.3.0 oslo.config==9.3.0 oslo.context==5.3.0 oslo.db==14.1.0 oslo.i18n==6.2.0 oslo.log==5.4.0 oslo.messaging==14.6.0 oslo.metrics==0.7.0 oslo.middleware==6.0.0 oslo.policy==4.2.1 oslo.serialization==5.3.0 oslo.service==3.3.0 oslo.upgradecheck==2.2.0 oslo.utils==7.0.0 osprofiler==4.1.0 overrides==7.7.0 packaging==23.2 pandas==2.2.0 pandocfilters==1.5.1 panel==1.3.8 param==2.0.2 parso==0.8.3 partd==1.4.1 passlib==1.7.4 Paste==3.7.1 PasteDeploy==3.1.0 pathspec==0.12.1 pbr==6.0.0 pefile==2023.2.7 pexpect==4.9.0 phoenixAES==0.0.5 pillow==10.2.0 platformdirs==4.2.0 plotly==5.18.0 pluggy==1.3.0 plumbum==1.8.2 ply==3.11 prettytable==3.10.0 progressbar2==4.3.2 prometheus_client==0.20.0 prompt-toolkit==3.0.43 protobuf==3.20.3 psutil==5.9.8 ptyprocess==0.7.0 pure-eval==0.2.2 pycadf==3.1.1 pycdlib==1.12.0 pycparser==2.21 pycryptodome==3.20.0 pyct==0.5.0 pyelftools==0.30 Pygments==2.17.2 PyJWT==2.8.0 pyOpenSSL==24.0.0 pyparsing==3.1.1 pysaml2==7.5.0 pyserial==3.5 PySMT==0.9.6.dev53 pytest==7.4.4 pytest-asyncio==0.19.0 pytest-cov==4.1.0 python-dateutil==2.8.2 python-json-logger==2.0.7 python-keystoneclient==5.3.0 python-magic==0.4.27 python-utils==3.8.1 pytz==2024.1 pyvex==9.2.6 pyviz_comms==3.0.1 PyYAML==6.0.1 pyzmq==25.1.2 qtconsole==5.5.1 QtPy==2.4.1 reedsolo==1.7.0 referencing==0.33.0 repoze.lru==0.7 requests==2.31.0 rfc3339-validator==0.1.4 rfc3986==2.0.0 rfc3986-validator==0.1.1 rich==13.7.0 Routes==2.5.1 rpds-py==0.18.0 rpyc==5.3.1 scikit-learn==1.3.2 scipy==1.12.0 scrypt==0.8.20 Send2Trash==1.8.2 six==1.16.0 smmap==5.0.1 sniffio==1.3.0 sortedcontainers==2.2.2 soupsieve==2.5 SQLAlchemy==2.0.27 stack-data==0.6.3 statsd==4.0.1 stevedore==5.1.0 sympy==1.12 synthol==0.1.1 tenacity==8.2.3 termcolor==1.1.0 terminado==0.18.0 terminaltables==3.1.10 testresources==2.0.1 testscenarios==0.5.0 testtools==2.7.1 threadpoolctl==3.2.0 tinycss2==1.2.1 tomli==2.0.1 toolz==0.12.1 tornado==6.4 tqdm==4.66.1 traitlets==5.14.1 typeguard==2.13.3 types-python-dateutil==2.8.19.20240106 typing-inspect==0.7.1 typing_extensions==4.9.0 tzdata==2024.1 ubi_reader==0.8.5 uc-micro-py==1.0.2 unicorn==1.0.2rc4 uri-template==1.3.0 urllib3==2.2.1 vine==5.1.0 vispy==0.14.1 wcwidth==0.2.13 webcolors==1.13 webencodings==0.5.1 WebOb==1.8.7 websocket-client==1.7.0 Werkzeug==3.0.1 widgetsnbextension==4.0.9 wrapt==1.16.0 xarray==2024.1.1 xattr==0.10.1 xmlschema==2.5.1 xyzservices==2023.10.1 yappi==1.6.0 yarl==1.9.4 z3-solver==4.12.5.0 zipp==3.17.0

whyitfor commented 8 months ago

Thanks for the pip freeze output.

It seems possible that keystone-engine is not fully installed -- this is something that we've observed happening on systems (such as arm64 Macs) where keystone install involves a build step and cmake is not available the first time around.

Can you try the following: pip uninstall -y keystone-engine; pip install keystone-engine?

Let me know if this works!

rustammendel commented 8 months ago

Thanks for suggestion,

I did as you suggested, however now I get another error:

<...> File "", line 1030, in _gcd_import File "", line 1007, in _find_and_load File "", line 986, in _find_and_load_unlocked File "", line 680, in _load_unlocked File "", line 855, in exec_module File "", line 228, in _call_with_frames_removed File "<...>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/ofrak/service/assembler/assembler_service_keystone.py", line 3, in from keystone import ( File "<...>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/keystone/init.py", line 4, in from .keystone import Ks, ks_version, ks_arch_supported, version_bind, debug, KsError, version File "<...>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/keystone/keystone.py", line 74, in raise ImportError("ERROR: fail to load the dynamic library.") ImportError: ERROR: fail to load the dynamic library.

whyitfor commented 8 months ago

@rustammendel. This error is related to keystone-engine not being properly installed.

On machines without pre-built keystone shared-objects bundled in pip, the pip install keystone-engine step tries to build those libraries.

There are two general paths:

Figure out how to install keystone-engine, either working through pip issues or trying to install from source. I'd recommend this path, as it is a useful library for reverse engineering and used with some OFRAK features.
If you are really excited to try to use OFRAK, you could try to just remove the keystone imports in /.pyenv/versions/3.9.5/lib/python3.9/site-packages/ofrak/service/assembler/assembler_service_keystone.py file. Most of OFRAK should work if you get rid of the import errors; some of the instruction patching will not.

I'd recommend the first path.

whyitfor commented 6 months ago

Closing this issue, since there is no feedback since https://github.com/redballoonsecurity/ofrak/issues/428#issuecomment-1973328759.

redballoonsecurity / ofrak