jstucke
commented
3 years ago Does is show up in the web interface? The UID is really odd, though: It consists of a sha256 hash and the size in bytes and a size of 0 should not be correct.
Closed jen199 closed 2 years ago
jstucke
commented
3 years ago Does is show up in the web interface? The UID is really odd, though: It consists of a sha256 hash and the size in bytes and a size of 0 should not be correct.
jen199
commented
3 years ago Does is show up in the web interface? The UID is really odd, though: It consists of a sha256 hash and the size in bytes and a size of 0 should not be correct.
No, it never appears in the web interface. Also, I received the same uid each time I uploaded firmware file. Does it mean the file was not uploaded successfully? Thanks!
jstucke
commented
3 years ago Usually a response with a UID from the /rest/firmware endpoint should mean a successful upload. There seems to be something wrong with the initital upload request, though. It seems the binary content that was received is just the empty string, so no file was actually uploaded. Does the sample curl firmware upload snippet from https://github.com/fkie-cad/FACT_core/wiki/Rest-API work?
curl http://localhost:5000/rest/firmware -X PUT -H "Content-Type: application/json" -d '{"vendor": "AVM", "device_class": "Router", "file_name": "rest_test.txt", "requested_analysis_systems": ["file_type", "file_hashes"], "binary": "dGVzdDEyMzQgdGhpcyBpcyBzb21lIHRlc3QgZmlsZQ==", "device_name": "rest_test", "firmware_version": "1", "release_date": "2011-01-01", "tags": "tag1,tag2"}'I tried with another file rest_test.txt that I created, and received the same response.
Besides, I use "base64 MyFile" to obtain the binary. Wondering if it matters?
jstucke
commented
3 years ago Besides, I use "base64 MyFile" to obtain the binary. Wondering if it matters?
This should work, yes.
As far as I can tell, you did nothing wrong. Are there any errors showing up in the backend or frontend log of FACT? Are you using the latest master build or the stable release version?
jen199
commented
3 years ago The following information is the status of my FACT_Core, and the version according to what is shown in the response is 3.2-dev. I did not see any errors on the web interface. May I ask where can I see the logs? Thanks!!
curl $FACT_Core_server_url/rest/status --insecure -X GET | json_pp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6007 100 6007 0 0 279k 0 --:--:-- --:--:-- --:--:-- 279k { "plugins" : { "binwalk" : { "description" : "binwalk signature and entropy analysis", "version" : "0.5.3" }, "cpu_architecture" : { "description" : "identify CPU architecture", "version" : "0.3.2" }, "crypto_hints" : { "description" : "find indicators of specific crypto algorithms", "version" : "0.1" }, "crypto_material" : { "description" : "detects crypto material like SSH keys and SSL certificates", "version" : "0.5.2" }, "cve_lookup" : { "description" : "lookup CVE vulnerabilities", "version" : "0.0.4" }, "cwe_checker" : { "description" : "This plugin checks ELF binaries for several CWEs (Common Weakness Enumeration) likeCWE-243 (Creation of chroot Jail Without Changing Working Directory) andCWE-676 (Use of Potentially Dangerous Function).Due to the nature of static analysis, this plugin may run for a long time.", "version" : "0.5.0" }, "elf_analysis" : { "description" : "Analyzes and tags ELF executables and libraries", "version" : "0.3" }, "exploit_mitigations" : { "description" : "analyses ELF binaries within a firmware for present exploit mitigation techniques", "version" : "0.1.3" }, "file_hashes" : { "description" : "calculate different hash values of the file", "version" : "1.1" }, "file_system_metadata" : { "description" : "extract file system metadata (e.g. owner, group, etc.) from file system images contained in firmware", "version" : "0.2" }, "file_type" : { "description" : "identify the file type", "version" : "1.0" }, "init_systems" : { "description" : "detect and analyze auto start services", "version" : "0.4.1" }, "input_vectors" : { "description" : "Determines possible input vectors of an ELF executable like stdin, network, or syscalls.", "version" : "0.1.1" }, "interesting_uris" : { "description" : "This plugin filters all URIs identified inside the file based on relevance.The resulting list of URIs has a higher probability of representing important resources.", "version" : "0.1" }, "ip_and_uri_finder" : { "description" : "Search file for IP addresses and URIs based on regular expressions.", "version" : "0.4.2" }, "known_vulnerabilities" : { "description" : "Rule based detection of known vulnerabilities like Heartbleed", "version" : "0.2" }, "malware_scanner" : { "description" : "scan for known malware", "version" : "0.3.1" }, "printable_strings" : { "description" : "extracts strings and their offsets from the files consisting of printable characters", "version" : "0.3.4" }, "qemu_exec" : { "description" : "test binaries for executability in QEMU and display help if available", "version" : "0.5.1" }, "software_components" : { "description" : "identify software components", "version" : "0.4.1" }, "source_code_analysis" : { "description" : "This plugin implements static code analysis for multiple scripting languages", "version" : "0.5" }, "string_evaluator" : { "description" : "Tries to sort strings based on usefulness", "version" : "0.2.1" }, "tlsh" : { "description" : "find files with similar tlsh and calculate similarity value", "version" : "0.1" }, "users_and_passwords" : { "description" : "search for UNIX, httpd, and mosquitto password files, parse them and try to crack the passwords", "version" : "0.5.0" } }, "request_resource" : "/rest/status", "status" : 0, "system_status" : { "backend" : { "_id" : "backend", "analysis" : { "analysis_main_scheduler" : 0, "current_analyses" : { "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855_0" : { "analyzed_count" : 0, "start_time" : 1618199053.90555, "total_count" : 1, "unpacked_count" : 1 } }, "plugins" : { "binwalk" : { "active" : 0, "queue" : 0 }, "cpu_architecture" : { "active" : 0, "queue" : 0 }, "crypto_hints" : { "active" : 0, "queue" : 0 }, "crypto_material" : { "active" : 0, "queue" : 0 }, "cve_lookup" : { "active" : 0, "queue" : 0 }, "cwe_checker" : { "active" : 0, "queue" : 0 }, "dummy_plugin_for_testing_only" : { "active" : 0, "queue" : 0 }, "elf_analysis" : { "active" : 0, "queue" : 0 }, "exploit_mitigations" : { "active" : 0, "queue" : 0 }, "file_hashes" : { "active" : 0, "queue" : 0 }, "file_system_metadata" : { "active" : 0, "queue" : 0 }, "file_type" : { "active" : 0, "queue" : 0 }, "init_systems" : { "active" : 0, "queue" : 0 }, "input_vectors" : { "active" : 0, "queue" : 0 }, "interesting_uris" : { "active" : 0, "queue" : 0 }, "ip_and_uri_finder" : { "active" : 0, "queue" : 0 }, "known_vulnerabilities" : { "active" : 0, "queue" : 0 }, "malware_scanner" : { "active" : 0, "queue" : 0 }, "printable_strings" : { "active" : 0, "queue" : 0 }, "qemu_exec" : { "active" : 0, "queue" : 0 }, "software_components" : { "active" : 0, "queue" : 0 }, "source_code_analysis" : { "active" : 0, "queue" : 0 }, "string_evaluator" : { "active" : 0, "queue" : 0 }, "tlsh" : { "active" : 0, "queue" : 0 }, "users_and_passwords" : { "active" : 0, "queue" : 0 } }, "recently_finished_analyses" : {} }, "last_update" : 1618199056.54315, "name" : "backend", "platform" : { "fact_version" : "3.2-dev", "os" : "Ubuntu 20.04", "python" : "3.8.5" }, "status" : "online", "system" : { "cpu_cores" : 10, "cpu_percentage" : 22.8, "disk_percent" : 48, "disk_total" : 1967925690368, "disk_used" : 896237637632, "load_average" : "4.11, 5.19, 5.66", "memory_percent" : 57.3, "memory_total" : 134934016000, "memory_used" : 75404087296, "virtual_cpu_cores" : 20 }, "unpacking" : { "unpacking_queue" : 0 } }, "database" : { "_id" : "database", "last_update" : 1617158063.18218, "name" : "database", "platform" : { "fact_version" : "3.2-dev", "os" : "Ubuntu 20.04", "python" : "3.8.5" }, "status" : "online", "system" : { "cpu_cores" : 10, "cpu_percentage" : 5.8, "disk_percent" : 52.9, "disk_total" : 1967925690368, "disk_used" : 987295391744, "load_average" : "0.91, 1.08, 2.38", "memory_percent" : 24.7, "memory_total" : 134934016000, "memory_used" : 31390384128, "virtual_cpu_cores" : 20 } }, "frontend" : { "_id" : "frontend", "last_update" : 1617158060.2106, "name" : "frontend", "platform" : { "fact_version" : "3.2-dev", "os" : "Ubuntu 20.04", "python" : "3.8.5" }, "status" : "online", "system" : { "cpu_cores" : 10, "cpu_percentage" : 5.5, "disk_percent" : 52.9, "disk_total" : 1967925690368, "disk_used" : 987295391744, "load_average" : "0.9, 1.08, 2.39", "memory_percent" : 24.7, "memory_total" : 134934016000, "memory_used" : 31421833216, "virtual_cpu_cores" : 20 } } }, "timestamp" : 1618217904 }
jstucke
commented
3 years ago May I ask where can I see the logs? Thanks!!
Apart from the terminal where you are running FACT, it should also log to the file /tmp/fact_main.log (but not remotely accessible through REST).
What happens if you upload the file using the web frontend? Does that work for you?
jen199
commented
3 years ago It works fine when I uploaded file using the web front-end, the only issue is that I cannot download PDF file.
This is the latest content of the log file:
[2021-04-12 11:36:33][rest_firmware][WARNING]: Submission not according to API guidelines! (data could not be parsed)
[2021-04-12 11:40:14][rest_firmware][WARNING]: Submission not according to API guidelines! (data could not be parsed)
[2021-04-12 11:41:02][rest_firmware][WARNING]: Submission not according to API guidelines! (data could not be parsed)
[2021-04-12 11:43:40][rest_firmware][WARNING]: Submission not according to API guidelines! (data could not be parsed)
[2021-04-12 11:44:16][process][ERROR]: [91mException in Unpacking process:
Traceback (most recent call last):
File "/home/security/FACT_core/src/helperFunctions/process.py", line 52, in run
Process.run(self)
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/security/FACT_core/src/scheduler/Unpacking.py", line 71, in unpack_worker
self.post_unpack(fo)
File "/home/security/FACT_core/src/scheduler/Analysis.py", line 85, in start_analysis_of_object
self._schedule_analysis_tasks(fo, fo.scheduled_analysis, mandatory=True)
File "/home/security/FACT_core/src/scheduler/Analysis.py", line 94, in _schedule_analysis_tasks
scheduled_analysis = self._add_dependencies_recursively(copy(scheduled_analysis) or [])
File "/home/security/FACT_core/src/scheduler/Analysis.py", line 410, in _add_dependencies_recursively
new_dependencies = self._get_cumulative_remaining_dependencies(scheduled_analyses_set)
File "/home/security/FACT_core/src/scheduler/Analysis.py", line 417, in _get_cumulative_remaining_dependencies
return {
File "/home/security/FACT_core/src/scheduler/Analysis.py", line 420, in
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/dist-packages/flask_restful/init.py", line 272, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1948, in full_dispatch_request
rv = self.preprocess_request()
File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2242, in preprocess_request
rv = func()
File "/usr/local/lib/python3.8/dist-packages/flask_principal.py", line 477, in _on_before_request
identity = loader()
File "/usr/local/lib/python3.8/dist-packages/flask_security/core.py", line 245, in _identity_loader
if not isinstance(current_user._get_current_object(), AnonymousUserMixin):
File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 307, in _get_current_object
return self.local()
File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 26, in
jstucke
commented
3 years ago File "/home/security/FACT_core/src/scheduler/Analysis.py", line 420, in for dependency in self.analysis_plugins[plugin].DEPENDENCIES KeyError: 'l'
It seems like there was an error in a previous request where the plugins were not provided as list
sqlite3.OperationalError: no such table: user
Are you running FACT with authentication enabled? In this case you need to create users using src/manage_users.py (from the CLI).
binascii.Error: Incorrect padding
It seems there also was a problem with the padding of the base64 encoded "binary" data field. I'm not sure if the output of base64 is always compatible with the python library but I would expect it to be. You could try this in case you encounter the error again:
python3 -c "import sys, pathlib, base64; print(base64.b64encode(pathlib.Path(sys.argv[1]).read_bytes()).decode())" <file_path>Normally the worker processes should be restarted in case of errors but maybe you should restart FACT to make sure all workers work correctly
Is the "firmware" you uploaded successfully through the web interface showing up correctly?
jen199
commented
3 years ago It seems like there was an error in a previous request where the plugins were not provided as list
sqlite3.OperationalError: no such table: user
Are you running FACT with authentication enabled? In this case you need to create users using
src/manage_users.py(from the CLI).May I ask where and how can I set the authentication? It seems that there is no instruction on the GitHub or the official website. Or, I have to run src/manage_users.py by myself? Highly appreciate if there is any details about how to setup.
binascii.Error: Incorrect padding
It seems there also was a problem with the padding of the base64 encoded "binary" data field. I'm not sure if the output of
base64is always compatible with the python library but I would expect it to be. You could try this in case you encounter the error again:python3 -c "import sys, pathlib, base64; print(base64.b64encode(pathlib.Path(sys.argv[1]).read_bytes()).decode())" <file_path>Thanks! I will use the python library instead. Normally the worker processes should be restarted in case of errors but maybe you should restart FACT to make sure all workers work correctly
Got it.
Is the "firmware" you uploaded successfully through the web interface showing up correctly?
Yes, the one I uploaded through the web page works well, and I can see the results of scanning and everything.
By the way, I would like to ask what is "device_part" in Rest API /rest/firmware[/
jstucke
commented
3 years ago May I ask where and how can I set the authentication? It seems that there is no instruction on the GitHub or the official website. Or, I have to run src/manage_users.py by myself? Highly appreciate if there is any details about how to setup.
Per default, authentication is disabled. You can enable it by setting authentication = true in src/config/main.cfg
Documentation regarding this can be found here. But since you don't know about it, you probably didn't enable it and it should not be required to run FACT. That being said, could you try adding a user as documented (but without enabling authentication) to see if this at least fixes your issue?
By the way, I would like to ask what is "device_part" in Rest API /rest/firmware[/]? Since there's no explanation of how to use and how to obtain it from the firmware file.
This was meant as an option to upload different parts of a firmware individually but is not really necessary. You could simply set it to "complete" (the default that is set through the web interface upload) and leave it at that.
jstucke
commented
3 years ago Did you use the REST endpoint with an API key? I took another look at the code and the authentication issue seems to be caused by the "Authorization" field in the HTML header being set without having enabled authentication and setting up users.
The normal way to use REST with authentication in FACT is to use the API key that is provided for the user. If you did not set an API key then maybe the Authorization field is set by something else? Maybe the --insecure parameter of curl? We probably need to implement some error handling for that.
jen199
commented
3 years ago Did you use the REST endpoint with an API key? I took another look at the code and the authentication issue seems to be caused by the "Authorization" field in the HTML header being set without having enabled authentication and setting up users.
The normal way to use REST with authentication in FACT is to use the API key that is provided for the user. If you did not set an API key then maybe the Authorization field is set by something else? Maybe the
--insecureparameter of curl? We probably need to implement some error handling for that.
So far I have added two users who are admin and worker for the FACT_Core by following the instruction in https://github.com/fkie-cad/FACT_core/wiki/Authentication. I also obtained API key for both of the users. Besides, I have shut down and restarted FACT_Core.
However, the firmware file still does not appear even I have added --insecure and --header with the API key when using curl to upload the firmware file via Rest API.
Since there's no domain name for our FACT_Core server, only IP address without SSL certification, --insecure is required. Thus, I am not sure whether the problem is caused by it or not. Wondering what else caused this issue that I always receive the same uid and status 0, but cannot find the firmware file in the database when using Rest API to do the uploading.
jen199
commented
3 years ago By the way, when I use python3 -c "import sys, pathlib, base64; print(base64.b64encode(pathlib.Path(sys.argv[1]).read_bytes()).decode())" <file_path> to get the binary string if my firmware, curl throws an error which is Argument list is too long. Could I provide a file to the binary field instead of a string? Thanks!
jstucke
commented
3 years ago Wondering what else caused this issue that I always receive the same uid and status 0, but cannot find the firmware file in the database when using Rest API to do the uploading.
The uid should be the same if you upload the same file but not for different files and it should never end in "_0" (which would mean the file is empty). Could you please reset your log file (e.g. by renaming it), restart FACT and try the example with the text file again (without API key) and post the contents of the log here?
Could I provide a file to the binary field instead of a string? Thanks!
It should be fine to continue using the base64 cli tool, since the problem seems to lie somewhere else.
jen199
commented
3 years ago Hi, these three files are the log files that I have gather so far. I have tried to shutdown and restart the FACT_Core, and upload a test.txt file.
jstucke
commented
3 years ago It seems that in fact_main.log you erroneously submitted a string instead of a list of strings for requested_analysis_systems. This error will hopefully be caught once #557 is merged. Could you retry and make sure that the field is set correct?
There were some pdf report errors in fact_main_bk.log but these were recently fixed. You can get the fixed version with docker pull fkiecad/fact_pdf_report:latest. If you see ServerSelectionTimeoutErrors It means that the frontend or backend could not connect to the database (i.e. MongoDB is not running). Trying to remove X from current analysis of Y but it is not included usually means that you started the analysis of a firmware again before it was finished.
The error in the "cwe_checker" plugin is new, though and not yet fixed. It could be helpful if you could provide one of the binaries that caused the error.
rhelmke
commented
2 years ago As #557 has been merged a few months ago and the issuer did not provide any more context for us to reproduce the problem with our cwe_checker plugin, I'm going to close this as for now.
Hi, I recently uploaded a firmware file to the FACT_Core which I install on localhost through Rest API /rest/firmware[/]. The response and status code are as shown in the image below:
However, when I requested the binary by giving the uid of the firmware file that I uploaded, I received error message which says "No firmware with UID xxx found in database". The response of my request is as shown in the picture below:
Does it mean that the file successfully uploaded? Besides, why the error message shows "no firmware with the uid in the database"? Since the previous return status is 0, which is success according to the document.
Thanks!