custom module not loading

phate1 commented 2 years ago

Description

ive tried my hand a writing a custom module but seems im doing something stupid as i can't get fame to load it. i tried adding it into the community folder and didnt see it, tried adding a custom folder in the modules folder which also didn't work. ive now uploaded to github and adding it as a repo it clones the repo fine but still doesn't pick up the module. im not sure what criteria decides if it gets processed or not maybe ive messed something up on the folder structure or something?

any pointers welcome folder structure pete@fame:~/fame/fame/modules/private $ tree . . ├── init.py ├── processing │ ├── init.py │ ├── pycache │ └── yara_proc │ ├── details.html │ ├── init.py │ ├── pycache │ │ └── yara_proc.cpython-38.pyc │ ├── requirements.txt │ └── yara_proc.py └── pycache

Steps to Reproduce

add custom repo with processing module

Expected behavior

module available in fame

Actual behavior

no mention of the module in worker logs while starting or reloading from ui not available to enable in ui

Debug

seem to have a issue with the mongo auth: had to add: "from fame.core import fame_init" and fame_init() otherwise i got an auth error running the script

pete@fame:~/fame$ utils/run.sh utils/troubleshoot.py [+] Using existing virtualenv.

Traceback (most recent call last): File "utils/troubleshoot.py", line 7, in from fame.core import fame_init ModuleNotFoundError: No module named 'fame' pete@fame:~/fame$ vi utils/troubleshoot.py pete@fame:~/fame$ utils/run.sh utils/troubleshoot.py [+] Using existing virtualenv.

########## VERSION ##########

OS: Linux-5.4.0-124-generic-x86_64-with-glibc2.29 Python: 3.8.10

########## DEPENDENCIES ###########

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly. alabaster==0.7.12 amqp==2.6.1 androguard==3.3.5 appdirs==1.4.4 asn1crypto==1.5.1 asttokens==2.0.7 Babel==2.10.3 backcall==0.2.0 backports.zoneinfo==0.2.1 beautifulsoup4==4.11.1 billiard==3.6.4.0 bs4==0.0.1 capstone==4.0.2 celery==4.4.7 certifi==2022.6.15 cffi==1.15.1 chardet==4.0.0 click==8.0.3 colorama==0.3.7 colorclass==2.2.2 compressed-rtf==1.0.6 cryptography==37.0.4 cxxfilt==0.2.2 cycler==0.11.0 decorator==4.4.2 defang==0.5.3 distlib==0.3.5 docker==4.4.0 docutils==0.16 easygui==0.98.3 ebcdic==1.1.1 executing==0.9.1 extract-msg==0.36.1 fasttext==0.9.2 filelock==3.8.0 flare-capa==3.0.2 Flask==2.1.3 Flask-Classful==0.14.2 Flask-Login==0.5.0 flask-paginate==0.7.1 fonttools==4.34.4 funcy==1.16 future==0.18.2 gitdb==4.0.9 GitPython==3.1.27 googleplay-api==0.1.0 halo==0.0.31 hatching-triage==0.1.7 hexdump==3.3 ida-netnode==3.0 ida-settings==2.1.0 idna==2.10 ijson==3.1.4 imagesize==1.4.1 IMAPClient==2.3.1 importlib-metadata==4.12.0 intervaltree==3.1.0 ipython==8.4.0 itsdangerous==2.1.2 javaobj-py3==0.4.3 jbxapi==3.18.0 jedi==0.18.1 Jinja2==3.0.3 joblib==0.16.0 jsbeautifier==1.6.2 kiwisolver==1.4.4 kombu==4.6.11 lark-parser==0.12.0 libvirt-python==7.1.0 lief==0.11.0 lightgbm==3.3.0 log-symbols==0.0.14 lxml==4.9.1 malwareconfig==1.0.4 markdown2==2.3.10 MarkupSafe==2.1.1 matplotlib==3.5.3 matplotlib-inline==0.1.3 msgpack==1.0.4 msoffcrypto-tool==4.11.0 networkx==2.5.1 numpy==1.23.1 olefile==0.46 oletools==0.56 packaging==21.3 parso==0.8.3 pbkdf2==1.3 pcodedmp==1.2.6 peepdf==0.4.2 pefile==2021.9.3 pexpect==4.8.0 pickleshare==0.7.5 Pillow==3.2.0 platformdirs==2.5.2 prompt-toolkit==3.0.30 protobuf==4.21.5 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.10.0 pycparser==2.21 pycrypto==2.6.1 pycryptodomex==3.15.0 pydot==1.4.2 pyelftools==0.27 Pygments==2.12.0 pymongo==3.11.4 pyparsing==2.4.7 python-dateutil==2.8.1 python-flirt==0.6.3 python-magic==0.4.27 pythonaes==1.0 pytz==2022.1 pytz-deprecation-shim==0.1.0.post0 PyYAML==5.4.1 pyzipper==0.3.6 requests==2.25.1 RTFDE==0.0.2 ruamel.yaml==0.17.16 ruamel.yaml.clib==0.2.6 scikit-learn==0.23.2 scipy==1.9.0 six==1.16.0 smda==1.6.2 smmap==5.0.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 soupsieve==2.3.2.post1 Sphinx==3.2.1 sphinx-rtd-theme==0.5.2 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-httpdomain==1.7.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 spinners==0.0.24 stack-data==0.3.0 stringsifter==2.20201202 tabulate==0.8.9 termcolor==1.1.0 threadpoolctl==3.1.0 tqdm==4.62.3 traitlets==5.3.0 typing==3.7.4.3 tzdata==2022.1 tzlocal==4.2 urllib3==1.25.11 vine==1.3.0 virtualenv==20.13.4 virustotal-api==1.1.11 viv-utils==0.6.6 vivisect==1.0.5 volatility3==2.0.1 wcwidth==0.2.5 websocket-client==1.3.3 Werkzeug==2.0.3 yara-python==4.0.2 zipp==3.8.1 zxcvbn==4.4.28

########## MongoDB ##########

Version: 6.0.0 Authorization check: True

########## Configuration ##########

types: True comments: True extracted: True email: False malware_config: False volatility: True

Modules:

McAfee Antivirus Disabled Configured Sophos Antivirus Disabled Configured Symantec Antivirus Disabled Not Configured virustotal_download Preloading Enabled Configured cuckoo Processing Disabled Configured cuckoo_modified Processing Disabled Configured cutthecrap Processing Disabled Not Configured document_preview Processing Enabled Configured email_headers Processing Enabled Configured eml Processing Enabled Configured exiftool Processing Enabled Configured extract Processing Enabled Configured zip Processing Disabled Configured flare_capa Processing Enabled Configured triage Processing Enabled Configured joe Processing Disabled Not Configured marcher_config Processing Disabled Configured msg Processing Enabled Configured office_macros Processing Enabled Configured office_password Processing Enabled Configured peepdf Processing Enabled Configured stringsifter Processing Enabled Configured url_download Processing Enabled Configured url_preview Processing Enabled Configured virustotal_public Processing Enabled Configured mem_yara Processing Disabled Not Configured xlm_deobfuscator Processing Enabled Configured legacyzip Processing Disabled Configured mattermost Reporting Disabled Not Configured slack Reporting Disabled Not Configured Google Safe Browsing (Lookup API) Threat Intelligence Disabled Not Configured Google Safe Browsing (Update API) Threat Intelligence Disabled Not Configured SEKOIA.IO Threat Intelligence Disabled Not Configured URLhaus Threat Intelligence Disabled Not Configured Yeti Threat Intelligence Disabled Not Configured kvm Virtualization Disabled Configured virtualbox Virtualization Disabled Configured

Augustin-FL commented 2 years ago

Hi,

Few things here :

In order to import a module, you need to use the add module feature on the web interface (or you need to use do_clone() in a python script...see install.py for details). Editing the file structure directly is not supported, and will not work
This is because some additional operations are being carried out when cloning/updating a repo. More specifically:
- After the clone/pull, FAME will try to (re)import all python files of the repo, looking for instances of Module classes in them.
- If it finds any, it will register it in the database. This will cause the module to appear on the web interface, to be considered by the worker, etc...
- Note that python files that are invalid (syntax error, import error, etc..) are silently ignored. This is the reason why most modules have try/except blocks when importing external dependencies (the search for Module classes is done before installing requirements.txt)
- The code carrying out the imports/search of Module classes can be found here.

What likely happens is that your python file yara_proc.py has an error, and that FAME is not able to import it (why? I can't tell without seeing your python module). You could possibly tune the try/except in module_dispatcher to reveal & analyse the error?

This issue make me think of a new feature to be added : to display a warning message on the web interface when FAME is not being able to import a python file.

Also, regarding troubleshoot.py : indeed, thanks for the report. This will be corrected.

phate1 commented 2 years ago

Hey, thanks for the swift reply! that was indeed the problem it relied on a lib that needed installing, I wrongly assumed the requirements would be processed before trying to run it. I can now see the module in the UI ... it still doesn't work but I can figure that out from here :)

certsocietegenerale / fame