search

VirusTotal / yara-python

The Python interface for YARA
http://virustotal.github.io/yara/
Apache License 2.0
648 stars 179 forks source link

Module "magic" Not Working On macOS #153

Closed utkonos closed 1 year ago

utkonos commented 4 years ago

When installing yara-python using pip on macOS, the following exception is raised:

$ python test.py                                                     
Traceback (most recent call last):                                                             
  File "test.py", line 3, in <module>                                                          
    rules = yara.compile(filepath='test.yar')                                                  
yara.SyntaxError: test.yar(6): invalid field name "mime_type"

This is the YARA test file:

import "magic"

rule test
{
condition:
    magic.mime_type() == "application/vnd.ms-excel"
}

This is the Python testing script:

import yara

rules = yara.compile(filepath='test.yar')
matches = rules.match('test.txt')

print(matches)

If yara-python is installed from source, the following exception is raised:

$ python test.py 
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import yara
ImportError: dlopen(/Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg/yara.cpython-38-darwin.so, 2): Symbol not found: _yr_compiler_
add_file
  Referenced from: /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg/yara.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg/yara.cpython-38-darwin.so

Here are the steps I used to install from pip:

$ python3 -m venv venv                                                                                                                                                      
$ source venv/bin/activate                                                                                                                                                  
(venv) $ pip install -U pip setuptools                                                                                                                                      
Collecting pip                                                                                                                                                                                 
  Using cached pip-20.2-py2.py3-none-any.whl (1.5 MB)                                                                                                                                          
Collecting setuptools                                                                                                                                                                          
  Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)                                                                                                                                     
Installing collected packages: pip, setuptools                                                                                                                                                 
  Attempting uninstall: pip                                                                                                                                                                    
    Found existing installation: pip 20.1.1                                                                                                                                                    
    Uninstalling pip-20.1.1:                                                                                                                                                                   
      Successfully uninstalled pip-20.1.1                                                                                                                                                      
  Attempting uninstall: setuptools                                                                                                                                                             
    Found existing installation: setuptools 47.1.0                                                                                                                                             
    Uninstalling setuptools-47.1.0:                                                                                                                                                            
      Successfully uninstalled setuptools-47.1.0                                                                                                                                               
Successfully installed pip-20.2 setuptools-49.2.1                                                                                                                                              
(venv) $ pip install yara-python                                                                                                                                            
Collecting yara-python                                                                                                                                                                         
  Using cached yara-python-4.0.2.tar.gz (405 kB)                                                                                                                                               
Using legacy 'setup.py install' for yara-python, since package 'wheel' is not installed.                                                                                                       
Installing collected packages: yara-python                                                                                                                                                     
    Running setup.py install for yara-python ... done                                                                                                                                          
Successfully installed yara-python-4.0.2

Here are the steps I used to install from source:

$ python3 -m venv venv                                                                                                                                                      
$ source venv/bin/activate                                                                                                                                                  
(venv) $ pip install -U pip setuptools                                                                                                                                      
Collecting pip                                                                                                                                                                                 
  Using cached pip-20.2-py2.py3-none-any.whl (1.5 MB)                                                                                                                                          
Collecting setuptools                                                                                                                                                                          
  Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)                                                                                                                                     
Installing collected packages: pip, setuptools                                                                                                                                                 
  Attempting uninstall: pip                                                                                                                                                                    
    Found existing installation: pip 20.1.1                                                                                                                                                    
    Uninstalling pip-20.1.1:                                                                                                                                                                   
      Successfully uninstalled pip-20.1.1                                                                                                                                                      
  Attempting uninstall: setuptools                                                                                                                                                             
    Found existing installation: setuptools 47.1.0                                                                                                                                             
    Uninstalling setuptools-47.1.0:                                                                                                                                                            
      Successfully uninstalled setuptools-47.1.0                                                                                                                                               
Successfully installed pip-20.2 setuptools-49.2.1                                                                                                                                              
(venv) $ git clone git@github.com:VirusTotal/yara-python.git                                                                                                                       
Cloning into 'yara-python'...                                                                                                                                                                  
remote: Enumerating objects: 24, done.                                                                                                                                                         
remote: Counting objects: 100% (24/24), done.                                                                                                                                                  
remote: Compressing objects: 100% (18/18), done.                                                                                                                                               
remote: Total 969 (delta 10), reused 18 (delta 6), pack-reused 945                                                                                                                             
Receiving objects: 100% (969/969), 231.01 KiB | 17.77 MiB/s, done.                                                                                                                             
Resolving deltas: 100% (570/570), done.                                                                                                                                
(venv) $ python setup.py build                                                                                                                                       
/Users/username/Desktop/test/venv/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit
 undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.                                                                                 
  warnings.warn(                                                                                                                                                                               
running build                                                                                                                                                                                  
running build_ext                                                                                                                                                                              
building 'yara' extension                                                                                                                                                                      
creating build                                                                                                                                                                                 
creating build/temp.macosx-10.15-x86_64-3.8                                                                                                                                                    
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/Library/Dev
eloper/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/op
t/openssl@1.1/include -D_GNU_SOURCE=1 -DUSE_MACH_PROC=1 -DHAVE_MEMMEM=1 -Iyara/libyara/include -Iyara/libyara/ -I. -I/usr/local/opt/openssl/include -I/opt/local/include -I/usr/local/include -
I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/username/Desktop/test/venv/include -I/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.fram
ework/Versions/3.8/include/python3.8 -c yara-python.c -o build/temp.macosx-10.15-x86_64-3.8/yara-python.o -std=c99                                                                             
yara-python.c:442:27: warning: comparison of integers of different signs: 'int64_t' (aka 'long long') and 'unsigned long long' [-Wsign-compare]                                                
      if (object->value.i != YR_UNDEFINED)                                                                                                                                                     
          ~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~~~                                                                                                                                                      
yara-python.c:993:16: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                                                
    identifier = PY_STRING_TO_C(key);                                                                                                                                                          
               ^ ~~~~~~~~~~~~~~~~~~~                                                                                                                                                           
yara-python.c:1022:13: warning: initializing 'char *' with an expression of type 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                         
      char* str = PY_STRING_TO_C(value);                                                                                                                                                       
            ^     ~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                        
yara-python.c:1063:16: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                                               
    identifier = PY_STRING_TO_C(key);                                                                                                                                                          
               ^ ~~~~~~~~~~~~~~~~~~~                                                                                                                                                           
yara-python.c:1092:13: warning: initializing 'char *' with an expression of type 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                         
      char* str = PY_STRING_TO_C(value);                                                                                                                                                       
            ^     ~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                        
yara-python.c:2089:18: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                                               
          source = PY_STRING_TO_C(value);                                                                                                                                                      
                 ^ ~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                       
yara-python.c:2090:14: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                                               
          ns = PY_STRING_TO_C(key);                                                                                                                                                            
             ^ ~~~~~~~~~~~~~~~~~~~                                                                                                                                                             
yara-python.c:2122:20: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                                               
          filepath = PY_STRING_TO_C(value);                                                                                                                                                    
                   ^ ~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                     
yara-python.c:2123:14: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]                                               
          ns = PY_STRING_TO_C(key);                                                                                                                                                            
             ^ ~~~~~~~~~~~~~~~~~~~                                                                                                                                                             
9 warnings generated.                                                                                                                                                                          
creating build/lib.macosx-10.15-x86_64-3.8                                                                                                                                                     
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -L/usr/local/opt/openssl@1.1/lib -I/usr/local/opt/openssl@1.1/include build/temp.mac
osx-10.15-x86_64-3.8/yara-python.o -L/opt/local/lib -L/usr/local/lib -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -o build/lib.macosx-10.15-x86_64-3.8/yara.cp
ython-38-darwin.so                                                                                                                                                                             
ld: warning: directory not found for option '-L/opt/local/lib'                                                                                                                                 
(venv) $ python setup.py install                                                                                                                                     
/Users/username/Desktop/test/venv/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit
 undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.                                                                                 
  warnings.warn(                                                                                                                                                                               
running install                                                                                                                                                                                
running bdist_egg                                                                                                                                                                              
running egg_info                                                                                                                                                                               
creating yara_python.egg-info                                                                                                                                                                  
writing yara_python.egg-info/PKG-INFO                                                                                                                                                          
writing dependency_links to yara_python.egg-info/dependency_links.txt                                                                                                                          
writing top-level names to yara_python.egg-info/top_level.txt                                                                                                                                  
writing manifest file 'yara_python.egg-info/SOURCES.txt'                                                                                                                                       
reading manifest file 'yara_python.egg-info/SOURCES.txt'                                                                                                                                       
reading manifest template 'MANIFEST.in'                                                                                                                                                        
warning: no files found matching 'yara/libyara/modules/module_list'                                                                                                                            
warning: no files found matching '*.c' under directory 'yara'                                                                                                                                  
warning: no files found matching '*.h' under directory 'yara'                                                                                                                                  
writing manifest file 'yara_python.egg-info/SOURCES.txt'                                                                                                                                       
installing library code to build/bdist.macosx-10.15-x86_64/egg
running install_lib
running build_ext
creating build/bdist.macosx-10.15-x86_64
creating build/bdist.macosx-10.15-x86_64/egg
copying build/lib.macosx-10.15-x86_64-3.8/yara.cpython-38-darwin.so -> build/bdist.macosx-10.15-x86_64/egg
creating stub loader for yara.cpython-38-darwin.so
byte-compiling build/bdist.macosx-10.15-x86_64/egg/yara.py to yara.cpython-38.pyc
creating build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/PKG-INFO -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/SOURCES.txt -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/dependency_links.txt -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/not-zip-safe -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/top_level.txt -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
writing build/bdist.macosx-10.15-x86_64/egg/EGG-INFO/native_libs.txt
creating dist
creating 'dist/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg' and adding 'build/bdist.macosx-10.15-x86_64/egg' to it
removing 'build/bdist.macosx-10.15-x86_64/egg' (and everything under it)
Processing yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg
creating /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg
Extracting yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg to /Users/username/Desktop/test/venv/lib/python3.8/site-packages
Adding yara-python 4.0.2 to easy-install.pth file

Installed /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg
Processing dependencies for yara-python==4.0.2
Finished processing dependencies for yara-python==4.0.2
utkonos commented 4 years ago

Everything works if the build is dynamically linked to the installed libyara: python setup.py build --dynamic-linking

wxsBSD commented 4 years ago

When installing yara-python using pip on macOS, the following exception is raised:

This is expected. The magic module is not on by default, and you did not tell pip to build with that option.

If yara-python is installed from source, the following exception is raised:

This part is a local problem and is likely because you built yara-python from src without having the yara submodule, so it used whatever old version of libyara it found laying around your system which does not include the necessary symbols (yr_compiler_add_file). When you clone yara-python clone it with the --recursive flag.

utkonos commented 2 years ago

The above problem is likely the same occurrence as #8

Just to make it clear for other people who encounter this problem, yara-python needs to be installed from source rather than using pip.

It would be nice to craft a method where a pip installed yara-python just worked without building from source, but until that time:

git clone --recursive git@github.com:VirusTotal/yara-python.git
python setup.py build --dynamic-linking
python setup.py install