Software components are missing in the /firmware API endpoint

[IoT-junkrat]

Dear FACT_core Team,

I am missing the 'software_components' item with the found libraries in the /rest/firmware GET response, which was there before.

It would be very kind if you guys could add it again. Cheers

[dorpvom]

Hi, can you post the rest response here? (Just replace long results with [...] and remove private information if needed) The structure might help to understand the issue.

Thx

[IoT-junkrat]

Below are the snippets of two JSON files. So I found another JSON file by chance, which I saved locally in July that contains the Libraries and the longer JSON below is the one with the actual release of FACT. I hope that helps.

... "~{&bfromVHli", "~{IHb/>", "~|Q{}P"]}, "software_components": {"OpenSSL": {"matches": true, "meta": {"description": "SSL library", "open_source": true, "software_name": "OpenSSL", "version": [""], "website": "https://www.openssl.org"}, "rule": "OpenSSL", "strings": [[407889, "$a", "T3BlblNTTA=="], [835361, "$a", "T1BFTlNTTA=="], [1074279, "$a", "T3BlblNTTA=="], [1215374, "$a", "T3BlblNTTA=="], [1224013, "$a", "T3BlblNTTA=="], [1233278, "$a", "T3BlblNTTA=="], [1264181, "$a", "T3BlblNTTA=="], [1295736, "$a", "T3BlblNTTA=="], [1304074, "$a", "T3BlblNTTA=="], [1335948, "$a", "T3BlblNTTA=="], [1370279, "$a", "T3BlblNTTA=="]]}, "analysis_date": 1532097426.4743593, "plugin_version": "0.3", "summary": ["OpenSSL "]}, "string_evaluator": {"analysis_date": 1532097463.1255865, "plugin_version": "0.2", ...

---

{"firmware": {"analysis": {"base64_decoder": {"analysis_date": 1537966400.9217114, "plugin_version": "0.1.3", "summary": []}, "binwalk": {"analysis_date": 1537966339.0955791, "entropy_analysis_graph": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAYAAAA10dzkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzanPmjNBn2bJl1Lp1axKJRPwzMIyxJkNEVE93SzPGGGOMsWaB3wTCGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbCcAHIGGOMMdbC/D98Y+hPkDvl6QAAAABJRU5ErkJggg==", "plugin_version": "0.5.2", "signature_analysis": "\nDECIMAL HEXADECIMAL DESCRIPTION\n--------------------------------------------------------------------------------\n0 0x0 RPM v3 bin i386 \"hp-firmware-ilo4-2.22-1.1\"\n2460 0x99C Unix path: /var/lib/rpm-state/hp-firmware/\n2498 0x9C2 Unix path: /var/lib/rpm-state/hp-firmware/hp-firmware-ilo4-2.22-1.1\n2589 0xA1D Unix path: /usr/lib/i386-linux-gnu/hp-firmware-ilo4-2.22-1.1\" ] ; then\n2661 0xA65 Unix path: /usr/lib/i386-linux-gnu/hp-firmware-ilo4-2.22-1.1\"\n2724 0xAA4 Unix path: /etc/hp-firmware/postrm.d ]; then\n2771 0xAD3 Unix path: /etc/hp-firmware/postrm.d/*.sh; do\n2975 0xB9F XML document, version: \"1.0\"\n221212 0x3601C Unix path: /usr/lib/i386-linux-gnu/\n221402 0x360DA Unix path: /var/lib/rpm-state/hp-firmware/hp-firmware-ilo4-2.22-1.1\" ] ; then\n221480 0x36128 Unix path: /var/lib/rpm-state/hp-firmware/hp-firmware-ilo4-2.22-1.1\"\n221558 0x36176 Unix path: /var/lib/rpm-state/hp-firmware -type f)\" ] ; then\n221705 0x36209 Unix path: /etc/hp-firmware/postinst.d ]; then\n221758 0x3623E Unix path: /etc/hp-firmware/postinst.d/*.sh; do\n222129 0x363B1 gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)\n\n", "summary": ["Unix path: /etc/hp-firmware/postinst.d ]; then", "Unix path: /var/lib/rpm-state/hp-firmware/", "Unix path: /etc/hp-firmware/postrm.d ]; then", "Unix path: /etc/hp-firmware/postinst.d/*.sh; do", "Unix path: /var/lib/rpm-state/hp-firmware/hp-firmware-ilo4-2.22-1.1", "Unix path: /usr/lib/i386-linux-gnu/", "Unix path: /var/lib/rpm-state/hp-firmware -type f)\" ] ; then", "Unix path: /var/lib/rpm-state/hp-firmware/hp-firmware-ilo4-2.22-1.1\"", "RPM v3 bin i386 \"hp-firmware-ilo4-2.22-1.1\"", "Unix path: /usr/lib/i386-linux-gnu/hp-firmware-ilo4-2.22-1.1\" ] ; then", "Unix path: /etc/hp-firmware/postrm.d/*.sh; do", "Unix path: /usr/lib/i386-linux-gnu/hp-firmware-ilo4-2.22-1.1\"", "Unix path: /var/lib/rpm-state/hp-firmware/hp-firmware-ilo4-2.22-1.1\" ] ; then", "XML document", "gzip compressed data"]}, "cpu_architecture": {"analysis_date": 1537966388.6327174, "plugin_version": "0.3.2", "summary": ["x86, little endian (M)"], "x86, little endian (M)": "Detection based on metadata"}, "crypto_material": {"analysis_date": 1537966350.3784475, "plugin_version": "0.5.2", "summary": []}, "exploit_mitigations": {"analysis_date": 1537966380.365649, "plugin_version": "0.1.2", "skipped": "blacklisted file type", "summary": []}, "file_hashes": {"analysis_date": 1537966432.9256623, "imphash": null, "md5": "9fdf473cfd1652f8636ab1eda4164894", "plugin_version": "1.0", "ripemd160": "c24ab26e140065ee5d0b2eae53489a1638bc566c", "sha1": "b49153425cb575726d3153e9f3c981b1c44726ca", "sha256": "688d7994ffad070396961018fb6c82f1ac40dbd68f4d2aa6ee1b7f23a9edb742", "sha512": "9501eb89c265572eaa774b3fe197d50d7ae75c440ac1a36831bf5c8c9c6ac090492680616698defec69fca17e87a84f933428f74297742f33fbbe689bbb7ed62", "ssdeep": "393216:deQMLTK0lZvDfp4de0GwcGoGGO3QTyEXmiiH:deQMLT5LB4df6yEXmiiH", "whirlpool": "85f91cbd8cf6846c0b141cad07ad9a672ef0d8e118b867efda425ce65d5dcb3c88b7abe2ccc5053eca8cd665869362e592ec8d36734ce91556ac4e9db72dd8f6"}, "file_type": {"analysis_date": 1537965984.2664123, "full": "RPM v3.0 bin i386/x86_64", "mime": "application/x-rpm", "plugin_version": "1.0", "summary": ["application/x-rpm"]}, "init_systems": {"analysis_date": 1537966355.6027596, "plugin_version": "0.4.1", "summary": []}, "ip_and_uri_finder": {"analysis_date": 1537966345.2269373, "ips_v4": [["2.0.1.0", "47.4563, -0.2711"], ["4.5.0.0", "37.751, -97.822"], ["4.4.0.0", "37.751, -97.822"]], "ips_v6": [["::c", ""], ["::9", ""], ["::6", ""], ["::", ""], ["::d", ""], ["b::", ""], ["0::A", ""], ["::A", ""], ["C::", ""], ["2::A", ""], ["6::", ""], ["::4", ""], ["3::", ""], ["::8", ""], ["::B", ""], ["::3", ""], ["f::", ""], ["8::", ""], ["::1", ""], ["2::", ""], ["E::", ""], ["::2", ""], ["::5", ""]], "plugin_version": "0.4.1", "summary": ["http://www.hp.com/go/ilo", "http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp", "http://h20000.www2.hp.com/", "http://www.hp.com/go/proliantlinux", "http://name", "http://h20000.www2.hp.com/bizsupport/TechSu", "http://www.hp.com/jp/servers/ilo", "2.0.1.0", "4.5.0.0", "4.4.0.0", "::c", "::9", "::6", "::", "::d", "b::", "0::A", "::A", "C::", "2::A", "6::", "::4", "3::", "::8", "::B", "::3", "f::", "8::", "::1", "2::", "E::", "::2", "::5"], "uris": ["http://www.hp.com/go/ilo", "http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp", "http://h20000.www2.hp.com/", "http://www.hp.com/go/proliantlinux", "http://name", "http://h20000.www2.hp.com/bizsupport/TechSu", "http://www.hp.com/jp/servers/ilo"]}, "known_vulnerabilities": {"analysis_date": 1537966438.796647, "plugin_version": "0.2", "summary": []}, "malware_scanner": {"analysis_date": 1537966381.9623334, "md5": "9fdf473cfd1652f8636ab1eda4164894", "number_of_scanners": 1, "plugin_version": "0.3.1", "positives": 0, "scanners": ["ClamAV"], "scans": {"ClamAV": {"detected": false, "result": "clean", "version": "ClamAV 0.100.1/24983/Wed Sep 26 04:39:15 2018\n"}}, "summary": [], "system_version": "0.2.6"}, "printable_strings": {"analysis_date": 1537966363.2203603, "offsets": [[10, "hp-firmware-ilo4-2.22-1.1"], [518, "9}b7ef27022950fb1abb69dfe29423e9443ef02c4e"], [2066, "hp-firmware-ilo4"], [2092, "HP Integrated Lights-Out 4 firmware"], [2128, "This package contains the HP Integrated Lights-Out 4 firmware"], [2196, "bldeb6u3x64003.SDG"], [2220, "Hewlett-Packard Company"], [2244, "2012 Hewlett-Packard Development Company, L.P."], [2291, "Hewlett-Packard Company"], [2315, "Applications/System"], [2335,"~s:7dN[\u000b$?hd", "~sQz-M", "~sXJzJ,bv", "~sXTU-", "~tZ\ttv?g", "~tts5x\u000b", "~tu3$?l", "~uxZc-", "~v&7O)m", "~vHda~}", "~v_yi\tt", "~vrL]1o", "~wp/hv", "~x1H{.W", "~xC 6R", "~xsA,{Tl", "~y\u000b\r\\Ok", "~y;{Ma", "~yL/h#ZO", "~yXQ<pP", "~y\n$\f", "~ya =$", "~z#9<v", "~z<c<$", "~zEYY!", "~zPt~W2?0", "~z_O\\_", "~zgxQ9", "~zmx<{Q", "~{T\"DBG", "~{~Qi,", "~|xzI_%i", "~}5zq(", "~}6XgJ", "~}VOu!", "~~V}J]", "~~f\f/?"]}, "software_components": {"analysis_date": 1537966423.3171716, "plugin_version": "0.3.1", "summary": []}, "string_evaluator": {"analysis_date": 1537966416.271086, "plugin_version": "0.2.1", "string_eval": ["<br />]]>\n</revision_enhancements_xlate_part>\n </revision_enhancements_xlate>\n </revision>\n </revision_history>\n", "_N\u000bUK}", "_P?tE^", "_Q?^Qo", "_\\<^3\u000b", "_]{e-C", "_^^\u000bD\n", "_^e?5#", "_gA{?A", "_t=]*", "_u#E^(", "_}5-Xl", "_}_\"\tn", "51j\\{8^}", "^{}3*aN ", "]6{M%u", "^{t>>6", "}^kEC1", "^u=_}", "}{A^F;", ".^XbX>[DZ", ".){_n{", ".^~Sv{ ", ".zu^{QR", "_:^\"{_\"", ".@}:J", "_Bp{V", ".\t@s{}"]}, "unpacker": {"analysis_date": 1537965970.9848382, "entropy": 0.8885296494586213, "number_of_unpacked_files": 1, "output": "patool: Extracting /mnt/FACT/fact_fw_data/68/688d7994ffad070396961018fb6c82f1ac40dbd68f4d2aa6ee1b7f23a9edb742_13698369 ...\npatool: running /usr/local/bin/7z x -o/tmp/faf_unpack_ykmixaq6 -- /mnt/FACT/fact_fw_data/68/688d7994ffad070396961018fb6c82f1ac40dbd68f4d2aa6ee1b7f23a9edb742_13698369\npatool: ... /mnt/FACT/fact_fw_data/68/688d7994ffad070396961018fb6c82f1ac40dbd68f4d2aa6ee1b7f23a9edb742_13698369 extracted to /tmp/faf_unpack_ykmixaq6'.\n", "plugin_used": "PaTool", "plugin_version": "0.5", "size packed -> unpacked": "13.06 MiB -> 16.66 MiB", "summary": ["no data lost"]}, "users_and_passwords": {"analysis_date": 1537966372.7730966, "plugin_version": "0.4.1", "summary": []}}, "meta_data": {"device_class": "X", "device_name": "Y", "device_part": "", "hid": "Z", "release_date": "", "size": 13698369, "vendor": "P", "version": "99"}}, "request": {"uid": "688d7994ffad070396961018fb6c82f1ac40dbd68f4d2aa6ee1b7f23a9edb742_13698369"}, "request_resource": "/rest/firmware", "status": 0, "timestamp": 1537979756}

[dorpvom]

Hi,

we recently introduced a new blacklisting feature to automatically skip certain files in an analysis. The software_components plugin for example skips media and filesystem files among others. This is not the case here though, as the skipping is noted in the result [1]. As there are results for other plugins, I guess the problem has less to do with the REST API than the plugin itself.

If you previously had results for this file, there might be something wrong with the signature files. Can you check if you get results for other files?

Other than that, if the given file is a firmware, probably the software is detected in the included files rather than on the outside container. If that's the case there might be a problem with the propagation. Check that summary=true is present in your request URL if querying a firmware object to enforce propagation. I recognize that's not in the wiki, and I will look to add it soon.

[1] Example:

    "software_components": {
        "analysis_date": 1537451418.159038,
        "plugin_version": "0.3.1",
        "skipped": "blacklisted file type",
        "summary": []
    },

[IoT-junkrat]

Hey dorpvom,

Thanks for your reply. Using "summary=true" did the job! Yes it would be very nice to put that into the wiki.

Thank you very much!

Cheers