qilingframework / qiling

A True Instrumentable Binary Emulation Framework
https://qiling.io
GNU General Public License v2.0
5.07k stars 739 forks source link

ImportError: ERROR: fail to load the dynamic library (shown from unicorn.py). #1220

Open mrghubba opened 2 years ago

mrghubba commented 2 years ago

Hi there,

I was trying to run a Python code using Qiling. I downloaded Qiling along with the Qiling repository, Unicorn and Keystone.

I got the error: "fail to load the dynamic library" once "unicorn.py" appeared.

(The code is on the left and the error message is on the right): (Screenshot) https://paste.pics/IEAO0

I expected the program to decompile the bytecode of the specified smart contract as per this article from Qiling: https://www.shielder.com/blog/2022/04/a-sneak-peek-into-smart-contracts-reversing-and-emulation/

mrghubba commented 2 years ago

Is anyone available to help?

chinggg commented 2 years ago

Sorry, I don't have a Windows environment now. I guess that's caused by incomplete installation of unicorn. You can pip list to ensure you have installed the latest version unicorn2. Then check whether you have the unicorn lib (DLL file) in the corresponding folder

mrghubba commented 2 years ago

Thanks for your message. I went to this page to download unicorn 2: https://github.com/unicorn-engine/unicorn/blob/master/docs/COMPILE.md
Then I followed these instructions: https://paste.pics/IGAIX and I got this error message: https://paste.pics/IGAL5

chinggg commented 2 years ago

@mrghubba These instructions are used for building unicorn, so they should be done in unicorn source code folder instead of qiling folder. You may try pip install unicorn==2.0.0 to directly install the python binding of unicorn2, since https://pypi.org/project/unicorn/2.0.0/#files has listed the built distributions for Windows

mrghubba commented 2 years ago

I tried "pip install unicorn==2.0.0" and it was already installed globally along with cmake: https://paste.pics/IGLF7 The Import Error is now coming from utils.py

I also tried to build on cmake within the Unicorn Repository that I downloaded but I got this error: https://paste.pics/IGM1K

elicn commented 2 years ago

I think you over-complicated this and got stuck.. I recommend to remove Unicorn altogether [installation and residuals], and just install it via pip. Funny thing, EVM doesn't even use Unicorn...

mrghubba commented 2 years ago

Hi Eli,

Are you saying not to use the Qiling repository (https://github.com/qilingframework) at all? Get rid of it? Are you suggesting this alone(?): **One line installer Installation using pip (stable version)

pip3 install qiling**

I'm relieved to now say that the ImportError is now gone! At least for my Windows 7. I followed this on the Installation Instruction page: **Due to distribution restriction, Qiling Framework will not bundle Microsoft Windows DLL files and registry. Please copy respective DLLs and registry from Microsoft Windows System. For Windows 10 usually found in C:\Windows\system32 (64bit dll) and C:\Windows\SysWOW64 (32bits dll) and place them in $rootfs/dlls

We also included a script named dllscollector.bat. Run this on Windows, under Administrator privilege, to collect all the necessary dlls and registries.

examples/scripts/dllscollector.bat**

But I'm still getting errors coming from utils.py (which is part of the Qiling repo): "Import yaml could not be resolved from source pylance....."

I also haven't seen a solution for Ubuntu 20 (ARM64)

If I need to remove the repo, I'll give that a try.

Please confirm. Thanks very much.

elicn commented 2 years ago

No, I am afraid you got it wrong. I recommend using Qiling from git, specifically the dev branch. As for dependencies. yaml is not needed unless you emulate MCU. The unicorn module is heavily used around Qiling so it appears on many modules as an import; however, it is not used in EVM.

We could do a better job isolating dependencies, but that is the status as of now. To be able to use EVM, just install unicorn using pip. That should do.

mrghubba commented 2 years ago

Okay. Thanks for clearing that up. Unfortunately, the issue then appears to be the initial pip install for Qiling. I tried this on 2 different operating systems: Windows 7 - Needs MSVC - Microsoft Visual Studio C++ version 14 or higher (I installed Version 17 but they claimed that the software was not intended for Windows 7 even though the installation was a success)

Ubuntu 20.04 ARM 64 (trying the pip install git dev for Qiling): https://paste.pics/IIUDV (Error message - I couldn't even get Unicorn installed).

From what you're saying, it should all work once Qiling, its repo, the EVM and Unicorn are installed. But getting there is the issue as I can't properly install Qiling on either operating system.

elicn commented 2 years ago

You cannot install Qiling dev branch, you need to clone it: git clone https://github.com/qilingframework/qiling.git --branch dev --recurse-submodules

Then install Unicorn2 from pip: python3 -m pip install unicorn

When you run a Qiling script, make sure the Qiling folder is on your Python path by prepending with PYTHONPATH var, e.g.: PYTHONPATH=/path/to/your/qiling/dir python3 your_qiling_script.py

mrghubba commented 2 years ago

Okay good. Any suggestions for installing the EVM? I tried it on BOTH the ARM64 Ubuntu 20 and Windows 7 by following this site: https://docs.qiling.io/en/latest/evm/ where they say to enter this: pip3 install -e .[evm] . Windows 7 gave the same error of Microsoft Visual Studio C++ 14.0 required. The Ubuntu ARM 64 gave this error: https://paste.pics/IKKLC

So in summary, the dev branch and Unicorn instructions worked on Windows 7. The library errors are gone (just the error 'Qiling is undefined' I'm assuming that is because the EVM is missing?). Is there a way to get the EVM or Windows or Ubuntu?

Thanks Eli.

elicn commented 2 years ago

As far as I remember installing EVM dependencies requires a building toolchain. In Ubuntu that would work with installing the build-essential package: sudo apt install build-essential (be warned that it is going to consume a non negligble amount of disk space)

After that you can proceed with the python3 -m pip install -e .[evm] command.

mrghubba commented 2 years ago

Okay I tried this on the Ubuntu 20 ARM64 (here's the command and result): $ sudo apt install build-essential Reading package lists... Done Building dependency tree... Done Reading state information... Done build-essential is already the newest version (12.9ubuntu3). 0 upgraded, 0 newly installed, 0 to remove and 48 not upgraded.

I then tried the python3 -m pip install -e .[evm] command right after but got the same error as before: https://paste.pics/IKKLC "× Preparing metadata (pyproject.toml) did not run successfully."

I researched the last line of the error message: "This is an issue with the package mentioned above, not pip." They say that I need to present a virtual env of python as a downgrade (from Python 3.10 to 3.6). Example: sudo apt-get install python3.6-dev python3.6-venv

Should I try this?

Is there a way to get build-essential for Windows 7?

Thanks for all your efforts.

elicn commented 2 years ago

Well.. I don't know the blake library, so I cannot comment on that. EVM emulation requires a bunch of additional libraries. Sicne I hate trashing my system with unnecessary libraries and installation, I created a Docker container for it. Setting up a Docker container and using it is a bit cumbersome, so I rarely use it. Another method is indeed creating a virtual environment, which is awesome if you know how to use it..

If you are not a containers expert neither a virtual environment kind of guy, the simplest way to go would be setting up a virtual machine with a system you are convinient with. When you're done with your experiments, you can just discard the entire thing and your system will remain clean of residuals and unnecessary installations. Although this might seem like an overkill, it is far simpler to handle and set up.

To your last question, build-essential is a Debian package bundling GNU building toolchain. If you want GNU toolchain on Windows, you'll need to set up Cygwin, and I do not recommend that.

mrghubba commented 1 year ago

I'm stuck at the moment. But thanks Eli for your efforts. I've taken all that you have mentioned in account.