search

fkie-cad / FACT_core

Firmware Analysis and Comparison Tool
https://fkie-cad.github.io/FACT_core
GNU General Public License v3.0
1.26k stars 225 forks source link

Internal Server Error when trying to access upload page #892

Open yhtiee opened 2 years ago

yhtiee commented 2 years ago

i get an internal server error when trying to access the upload page from the home page

jstucke commented 2 years ago

Hi and sorry for the late response. What do you see in the command line log? Are there any error messages / stack traces?

code2319 commented 2 years ago

I have the same error, log:

[2022-11-11 20:16:21,890] ERROR in app: Exception on /upload [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_restx/api.py", line 672, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/user/Downloads/FACT_core-master/src/./web_interface/security/decorator.py", line 11, in decorated_view
    return fn(*args, **kwargs)
  File "/home/user/Downloads/FACT_core-master/src/./web_interface/components/io_routes.py", line 43, in get_upload
    analysis_plugins = sc.get_available_analysis_plugins()
  File "/home/user/Downloads/FACT_core-master/src/./intercom/front_end_binding.py", line 34, in get_available_analysis_plugins
    raise Exception('No available plug-ins found. FACT backend might be down!')
Exception: No available plug-ins found. FACT backend might be down!
[2022-11-11 20:16:21][app][ERROR]: Exception on /upload [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_restx/api.py", line 672, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/user/Downloads/FACT_core-master/src/./web_interface/security/decorator.py", line 11, in decorated_view
    return fn(*args, **kwargs)
  File "/home/user/Downloads/FACT_core-master/src/./web_interface/components/io_routes.py", line 43, in get_upload
    analysis_plugins = sc.get_available_analysis_plugins()
  File "/home/user/Downloads/FACT_core-master/src/./intercom/front_end_binding.py", line 34, in get_available_analysis_plugins
    raise Exception('No available plug-ins found. FACT backend might be down!')
Exception: No available plug-ins found. FACT backend might be down!
[pid: 22581|app: 0|req: 12/63] 127.0.0.1 () {42 vars in 717 bytes} [Fri Nov 11 20:16:21 2022] GET /upload => generated 265 bytes in 63 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 1)

it's clear ubuntu 22.04 lts with minimal installation

jstucke commented 2 years ago

Was there some other error earlier in the logs? It seems the web frontend is missing data here that should be passed from the backend but somehow that never happened, hence the error. If not, is the backend displayed as "online" on the /system_health page and can you analyze firmware using the "Run additional analysis" feature from the analysis page?

code2319 commented 2 years ago

Was there some other error earlier in the logs? - no internal routing support, rebuild with pcre support (after detected binary path). is the backend displayed as "online" on the /system_health page - Internal server error

[2022-11-11 20:16:21][app][ERROR]: Exception on /system_health [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_restx/api.py", line 672, in error_router
    return original_handler(e)
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/user/Downloads/FACT_core-master/src/./web_interface/security/decorator.py", line 11, in decorated_view
    return fn(*args, **kwargs)
  File "/home/user/Downloads/FACT_core-master/src/./web_interface/components/io_routes.py", line 43, in get_upload
    analysis_plugins = sc.get_available_analysis_plugins()
  File "/home/user/Downloads/FACT_core-master/src/./intercom/front_end_binding.py", line 34, in get_available_analysis_plugins
    raise Exception('No available plug-ins found. FACT backend might be down!')
Exception: No available plug-ins found. FACT backend might be down!

"Run additional analysis" feature from the analysis page - where is it?

jstucke commented 2 years ago

"Run additional analysis" feature from the analysis page - where is it?

It's the page with the analysis and meta data that you see when you click on a firmware / file from the start page or from the search results. On the left where you can select the analysis at the bottom there is a button with "Run additional analysis" written on it. If you click in it, you can run analysis plugins just for this single file.

Exception: No available plug-ins found. FACT backend might be down!

Either the backend seems to have problems when starting or the communication between the frontend and the backend over Redis does not work.

Can you post the full log of the backend that you see when you start it separately with src/start_fact_backend.py? There may still be an error hidden somewhere.

Do service redis status and service postgresql@14-main status say "active (running)"?

code2319 commented 2 years ago

Can you post the full log of the backend that you see when you start it separately with src/start_fact_backend.py?

user@user-virtual-machine:~/Downloads/FACT_core-master/src$ ./start_fact_backend.py 
[2022-11-15 19:54:06][YaraPluginBase][ERROR]: Signature file /home/user/Downloads/FACT_core-master/src/analysis/signatures/crypto_hints.yc not found. Did you run "compile_yara_signatures.py"?
[2022-11-15 19:54:06][start_fact_backend][CRITICAL]: Error during initialization of plugin crypto_hints. Shutting down FACT backend
[2022-11-15 19:54:06][process][CRITICAL]: SHUTTING DOWN SYSTEM
Killed
user@user-virtual-machine:~/Downloads/FACT_core-master/src$ ./compile_yara_signatures.py 
Create signature directory /home/user/Downloads/FACT_core-master/src/analysis/signatures
Compile signatures in /home/user/Downloads/FACT_core-master/src/plugins/analysis/known_vulnerabilities/signatures
Compile signatures in /home/user/Downloads/FACT_core-master/src/plugins/analysis/crypto_hints/signatures
Compile signatures in /home/user/Downloads/FACT_core-master/src/plugins/analysis/software_components/signatures
Compile signatures in /home/user/Downloads/FACT_core-master/src/plugins/analysis/crypto_material/signatures
user@user-virtual-machine:~/Downloads/FACT_core-master/src$ ./start_fact_backend.py 
[2022-11-15 19:56:01][cwe_checker][INFO]: Version is cwe_checker 0.6.0

[2022-11-15 19:56:02][analysis][ERROR]: Could not import plugin ip_and_uri_finder due to exception
Traceback (most recent call last):
  File "/home/user/Downloads/FACT_core-master/src/scheduler/analysis.py", line 181, in _load_plugins
    plugin = source.load_plugin(plugin_name)
  File "/usr/local/lib/python3.10/dist-packages/pluginbase.py", line 301, in load_plugin
    return __import__(self.base.package + '.' + name,
  File "/usr/local/lib/python3.10/dist-packages/pluginbase.py", line 438, in plugin_import
    return self._system_import(import_name, globals, locals,
  File "/home/user/Downloads/FACT_core-master/src/plugins/analysis/ip_and_uri_finder/code/ip_and_uri_finder.py", line 7, in <module>
    import geoip2.database
  File "/usr/local/lib/python3.10/dist-packages/pluginbase.py", line 438, in plugin_import
    return self._system_import(import_name, globals, locals,
ModuleNotFoundError: No module named 'geoip2'
[2022-11-15 19:56:02][analysis][ERROR]: Could not import plugin software_components due to exception
Traceback (most recent call last):
  File "/home/user/Downloads/FACT_core-master/src/scheduler/analysis.py", line 181, in _load_plugins
    plugin = source.load_plugin(plugin_name)
  File "/usr/local/lib/python3.10/dist-packages/pluginbase.py", line 301, in load_plugin
    return __import__(self.base.package + '.' + name,
  File "/usr/local/lib/python3.10/dist-packages/pluginbase.py", line 438, in plugin_import
    return self._system_import(import_name, globals, locals,
  File "/home/user/Downloads/FACT_core-master/src/plugins/analysis/software_components/code/software_components.py", line 13, in <module>
    from plugins.analysis.software_components.bin import OS_LIST
  File "/usr/local/lib/python3.10/dist-packages/pluginbase.py", line 438, in plugin_import
    return self._system_import(import_name, globals, locals,
ModuleNotFoundError: No module named 'plugins.analysis.software_components.bin'
[2022-11-15 19:56:02][analysis][INFO]: Analysis System online...
[2022-11-15 19:56:02][analysis][INFO]: Plugins available: ['binwalk', 'cpu_architecture', 'crypto_hints', 'crypto_material', 'cve_lookup', 'cwe_checker', 'device_tree', 'dummy_plugin_for_testing_only', 'elf_analysis', 'exploit_mitigations', 'file_hashes', 'file_system_metadata', 'file_type', 'hardware_analysis', 'hashlookup', 'information_leaks', 'init_systems', 'input_vectors', 'interesting_uris', 'kernel_config', 'known_vulnerabilities', 'printable_strings', 'qemu_exec', 'source_code_analysis', 'string_evaluator', 'tlsh', 'users_and_passwords']
[2022-11-15 19:56:02][unpacking_scheduler][INFO]: Unpacker Module online
[2022-11-15 19:56:03][unpacking_scheduler][INFO]: Queue Length (Analysis/Unpack): 0 / 0
[2022-11-15 19:56:04][compare][INFO]: Plug-ins available: dict_keys(['File_Coverage', 'File_Header', 'Software'])
[2022-11-15 19:56:04][comparison_scheduler][INFO]: Comparison Scheduler online...
[2022-11-15 19:56:04][back_end_binding][INFO]: InterCom started
[2022-11-15 19:56:04][binary_service][INFO]: binary service online
[2022-11-15 19:56:04][binary_service][INFO]: binary service online
[2022-11-15 19:56:04][binary_service][INFO]: binary service online
[2022-11-15 19:56:55][unpacking_scheduler][INFO]: Queue Length (Analysis/Unpack): 0 / 0
[2022-11-15 19:57:33][fail_safe_file_operations][ERROR]: Could not write file: [Errno 13] Permission denied: '/media/data/fact_fw_data'
Traceback (most recent call last):
  File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/data/fact_fw_data/01'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/common_helper_files/fail_safe_file_operations.py", line 61, in write_binary_to_file
    create_dir_for_file(file_path)
  File "/usr/local/lib/python3.10/dist-packages/common_helper_files/file_functions.py", line 50, in create_dir_for_file
    Path(file_path).resolve().parent.mkdir(parents=True, exist_ok=True)
  File "/usr/lib/python3.10/pathlib.py", line 1179, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/media/data/fact_fw_data'
Process Unpacking-Worker-3:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/user/Downloads/FACT_core-master/src/helperFunctions/process.py", line 57, in run
    raise exception
  File "/home/user/Downloads/FACT_core-master/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/Downloads/FACT_core-master/src/scheduler/unpacking_scheduler.py", line 66, in unpack_worker
    extracted_objects = unpacker.unpack(fo)
  File "/home/user/Downloads/FACT_core-master/src/unpacker/unpack.py", line 41, in unpack
    file_path = self._generate_local_file_path(current_fo)
  File "/home/user/Downloads/FACT_core-master/src/unpacker/unpack.py", line 124, in _generate_local_file_path
    local_path = self.file_storage_system.generate_path(file_object.uid)
  File "/home/user/Downloads/FACT_core-master/src/storage/fsorganizer.py", line 31, in generate_path
    return self.generate_path_from_uid(file_object.uid)
AttributeError: 'str' object has no attribute 'uid'
[2022-11-15 19:57:35][process][ERROR]: Exception in Unpacking process:
Traceback (most recent call last):
  File "/home/user/Downloads/FACT_core-master/src/helperFunctions/process.py", line 52, in run
    Process.run(self)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/Downloads/FACT_core-master/src/scheduler/unpacking_scheduler.py", line 66, in unpack_worker
    extracted_objects = unpacker.unpack(fo)
  File "/home/user/Downloads/FACT_core-master/src/unpacker/unpack.py", line 41, in unpack
    file_path = self._generate_local_file_path(current_fo)
  File "/home/user/Downloads/FACT_core-master/src/unpacker/unpack.py", line 124, in _generate_local_file_path
    local_path = self.file_storage_system.generate_path(file_object.uid)
  File "/home/user/Downloads/FACT_core-master/src/storage/fsorganizer.py", line 31, in generate_path
    return self.generate_path_from_uid(file_object.uid)
AttributeError: 'str' object has no attribute 'uid'

[2022-11-15 19:57:35][process][WARNING]: restarting Unpacking 3 process
[2022-11-15 19:57:47][unpacking_scheduler][INFO]: Queue Length (Analysis/Unpack): 0 / 0
[2022-11-15 19:58:39][unpacking_scheduler][INFO]: Queue Length (Analysis/Unpack): 0 / 0
[2022-11-15 19:59:31][unpacking_scheduler][INFO]: Queue Length (Analysis/Unpack): 0 / 0
[2022-11-15 20:00:23][unpacking_scheduler][INFO]: Queue Length (Analysis/Unpack): 0 / 0

error given that I tried to upload a file for analysis http://localhost:5000/rest/status -> current analyses: {}

jstucke commented 2 years ago

Those are a lot of errors. It seems some dependencies are missing. How did you install FACT? Did you follow the INSTALL.md? I would suggest rerunning python3 install.py from the src directory.

chenjianquan7 commented 1 year ago

it because code has a lot of error