Deepfence Threatmapper Integration Request

[shodanwashere]

Scanner Name Deepfence Threatmapper is an open source cloud native security observability platform, focused on verifying vulnerabilities, exposed secrets, malware and security compliance on platforms like AWS Fargate, Kubernetes, Linux, etc. https://github.com/deepfence/ThreatMapper

Sample File Threatmapper has several types of reports. Currently, there isnt a proper export format for its reports, but I've converted some of their reports into CSV for easier consumption.

Compliance Report

@timestamp,compliance_check_type,count,doc_id,host_name,cloud_account_id,masked,node_id,node_name,node_type,status,test_category,test_desc,test_info,test_number
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,149c4791fc6502e5a30f738d4eaba982,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,pass,Docker Files,3.6 - PASS,Ensure that /fenced/mnt/host/etc/docker directory permissions are set to 755 or more restrictively (Automated),gdpr_3.6
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,47edf84375c0bb90f48fa61684883b04,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,info,Docker Files,3.12 - INFO,Ensure that the Docker server certificate file permissions are set to 444 or more restrictively (Automated),gdpr_3.12
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,ad1965efb22e226df8a95a361a30cbc3,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,info,Docker Files,3.2 - INFO,Ensure that docker.service file permissions are appropriately set (Automated),gdpr_3.2
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,1db7418dc73082cdfc1c9e0d5ba5f6e0,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,warn,Audit,1.1.12 - WARN,1.1.12 Ensure auditing is configured for Dockerfiles and directories - /fenced/mnt/host/etc/containerd/config.toml (Automated),gdpr_1.1.12
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,2c3f915f3e72d6e16d192ae9aa71c704,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,pass,Docker Files,3.16 - PASS,Ensure that the Docker socket file permissions are set to 660 or more restrictively (Automated),gdpr_3.16
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,d158a60b1c623d11ce88cf68555e08af,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,info,Docker Files,3.4 - INFO,Ensure that docker.socket file permissions are set to 644 or more restrictive (Automated),gdpr_3.4
2024-01-25 11:17:30.272 +0000 UTC,gdpr,,,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,182758849647,False,4d1d5b7a279ce57b0f76be61b461d22c,cf-ngm-dev-cicd-ip-xxx-xxx-xxx-xxx.eu-central-1.compute.internal,host,info,Docker Files,3.14 - INFO,Ensure that the Docker server certificate key file permissions are set to 400 (Automated),gdpr_3.14

Malware Report

Rule Name,Class,File Name,Summary,Severity,Node Name,NodeType,Container Name,Kubernetes Cluster Name
MD5_Constants,Crypto Mining,/tmp/Deepfence/YaraHunter/df_db09257b02e615049e0aecc05be2dc2401735e67db4ab74225df777c62c39753/usr/sbin/mkfs.cramfs,The matched rule file's  author  is phoul (@phoul) .The file has a rule match that  It is a crypto signature.Look for MD5 constants .The matched rule file's  Date  is 2014-01 .The matched rule file's  version  is 0.2 .,low,portal / secpipe-core-prd-ip-zzz-zzz-zzz-zzz.eu-west-1.compute.internal,container,portal,secpipe-core-prd
MD5_Constants,Crypto Mining,/tmp/Deepfence/YaraHunter/df_80ffd64c318595cf17a9ea482315b0c2a03572fb6e41f7ee53ec27786158c27c/usr/sbin/mkfs.cramfs,The matched rule file's  author  is phoul (@phoul) .The file has a rule match that  It is a crypto signature.Look for MD5 constants .The matched rule file's  Date  is 2014-01 .The matched rule file's  version  is 0.2 .,low,portal / secpipe-core-prd-ip-uuu-uuu-uuu-uuu.eu-west-1.compute.internal,container,portal,secpipe-core-prd
CRC32_table,Crypto Mining,/tmp/Deepfence/YaraHunter/df_0dfa48a10ee6ca92c7d910ecd72a6207978f7f1bdc36870bf1587625f0270d37/lib/libz.so.1.2.13,The matched rule file's  author  is _pusher_ .The file has a rule match that  It is a crypto signature.Look for CRC32 table .The matched rule file's  Date  is 2015-05 .The matched rule file's  version  is 0.1 .,low,nginx / secpipe-core-prd-ip-kkk-kkk-kkk-kkk.eu-west-1.compute.internal,container,nginx,secpipe-core-prd
CRC32_poly_Constant,Crypto Mining,/tmp/Deepfence/YaraHunter/df_0dfa48a10ee6ca92c7d910ecd72a6207978f7f1bdc36870bf1587625f0270d37/lib/libz.so.1.2.13,The matched rule file's  author  is _pusher_ .The file has a rule match that  It is a crypto signature.Look for CRC32 [poly] .The matched rule file's  Date  is 2015-05 .The matched rule file's  version  is 0.1 .,low,nginx / secpipe-core-prd-ip-kkk-kkk-kkk-kkk.eu-west-1.compute.internal,container,nginx,secpipe-core-prd
MD5_Constants,Crypto Mining,/tmp/Deepfence/YaraHunter/df_cc54a20c0e1cee5e4951d047e13f69551cfddedbd67a05cc4e3de61939b10e7a/usr/sbin/mkfs.cramfs,The matched rule file's  author  is phoul (@phoul) .The file has a rule match that  It is a crypto signature.Look for MD5 constants .The matched rule file's  Date  is 2014-01 .The matched rule file's  version  is 0.2 .,low,portal / secpipe-core-prd-ip-yyy-yyy-yyy-yyy.eu-west-1.compute.internal,container,portal,secpipe-core-prd
MD5_Constants,Crypto Mining,/tmp/Deepfence/YaraHunter/df_5e10a8e665e9def9227c98ec630c80d8c8b441c389c3d2b25d7c8d3b07c94eb4/sbin/mkfs.cramfs,The matched rule file's  author  is phoul (@phoul) .The file has a rule match that  It is a crypto signature.Look for MD5 constants .The matched rule file's  Date  is 2014-01 .The matched rule file's  version  is 0.2 .,low,rabbitmq / secpipe-core-prd-ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal,container,rabbitmq,secpipe-core-prd
BASE64_table,Crypto Mining,/tmp/Deepfence/YaraHunter/df_5e10a8e665e9def9227c98ec630c80d8c8b441c389c3d2b25d7c8d3b07c94eb4/lib/x86_64-linux-gnu/libresolv-2.31.so,The matched rule file's  author  is _pusher_ .The file has a rule match that  It is a crypto signature.Look for Base64 table .The matched rule file's  Date  is 2015-07 .The matched rule file's  version  is 0.1 .,low,rabbitmq / secpipe-core-prd-ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal,container,rabbitmq,secpipe-core-prd
BASE64_table,Crypto Mining,/tmp/Deepfence/YaraHunter/df_5e10a8e665e9def9227c98ec630c80d8c8b441c389c3d2b25d7c8d3b07c94eb4/opt/bitnami/erlang/lib/erlang/erts-13.1.3/bin/beam.smp,The matched rule file's  author  is _pusher_ .The file has a rule match that  It is a crypto signature.Look for Base64 table .The matched rule file's  Date  is 2015-07 .The matched rule file's  version  is 0.1 .,low,rabbitmq / secpipe-core-prd-ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal,container,rabbitmq,secpipe-core-prd
CRC32_table,Crypto Mining,/tmp/Deepfence/YaraHunter/df_5e10a8e665e9def9227c98ec630c80d8c8b441c389c3d2b25d7c8d3b07c94eb4/lib/x86_64-linux-gnu/libz.so.1.2.11,The matched rule file's  author  is _pusher_ .The file has a rule match that  It is a crypto signature.Look for CRC32 table .The matched rule file's  Date  is 2015-05 .The matched rule file's  version  is 0.1 .,low,rabbitmq / secpipe-core-prd-ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal,container,rabbitmq,secpipe-core-prd

Secret Report

Filename,Content,Name,Rule,Severity,Node Name,Container Name,Kubernetes Cluster Name,Signature
usr/share/doc/curl-8.3.0/TheArtOfHttpScripting.md,"""\n    curl http://user:password@example.org/""",Username and password in URI,110,high,fluent-bit / secpipe-core-prd-ip-xxx-xxx-xxx-xxx.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,"([\w+]{1,24})(://)([^$<]{1})([^\s"";]{1,}):([^$<]{1})([^\s"";/]{1,})@[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,24}([^\s]+)"
usr/share/doc/curl-8.3.0/TheArtOfHttpScripting.md,"""\n    curl http://user:password@example.org/""",Username and password in URI,110,high,fluent-bit / secpipe-core-prd-ip-yyy-yyy-yyy-yyy.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,"([\w+]{1,24})(://)([^$<]{1})([^\s"";]{1,}):([^$<]{1})([^\s"";/]{1,})@[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,24}([^\s]+)"
var/lib/yum/history/history-2023-10-12.sqlite,""".sqlite""",SQLite database file,12,low,fluent-bit / secpipe-core-prd-ip-yyy-yyy-yyy-yyy.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,
usr/share/mime/magic,"""\n\u003e0=\u0000%-----BEGIN PGP PRIVATE KEY BLOCK-----""",Contains a private key,127,medium,fluent-bit / secpipe-core-prd-ip-yyy-yyy-yyy-yyy.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,-----BEGIN (EC|RSA|DSA|OPENSSH|PGP) PRIVATE KEY
usr/share/doc/curl-8.3.0/TheArtOfHttpScripting.md,"""\n    curl http://user:password@example.org/""",Username and password in URI,110,high,fluent-bit / secpipe-core-prd-ip-zzz-zzz-zzz-zzz.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,"([\w+]{1,24})(://)([^$<]{1})([^\s"";]{1,}):([^$<]{1})([^\s"";/]{1,})@[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,24}([^\s]+)"
var/lib/yum/history/history-2023-10-12.sqlite,""".sqlite""",SQLite database file,12,low,fluent-bit / secpipe-core-prd-ip-zzz-zzz-zzz-zzz.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,
usr/share/mime/magic,"""\n\u003e0=\u0000%-----BEGIN PGP PRIVATE KEY BLOCK-----""",Contains a private key,127,medium,fluent-bit / secpipe-core-prd-ip-zzz-zzz-zzz-zzz.eu-west-1.compute.internal,fluent-bit,secpipe-core-prd,-----BEGIN (EC|RSA|DSA|OPENSSH|PGP) PRIVATE KEY

I also wanted to include the vulnerability report but the way it was converted doesn't seem to follow CSV standards, so I may have to include it on this issue or open a new one with a request for that report type at a later date.

[manuel-sommer]

You can add it also here. I will take a look at it.

[shodanwashere]

Here's also the

Vulnerability Report

@timestamp,cve_attack_vector,cve_caused_by_package,cve_container_image,scan_id,cve_container_image_id,cve_cvss_score,cve_description,cve_fixed_in,cve_id,cve_link,cve_severity,cve_overall_score,cve_type,host_name,cloud_account_id,masked
2024-02-22 15:54:17.939 +0000 UTC,cvss:3.1/av:l/ac:l/pr:n/ui:r/s:u/c:n/i:n/a:l,libsepol:2.5-8.1.amzn2.0.2,aws-node / secpipe-core-prd-ip-10-xxx-xx-xx.eu-west-1.compute.internal,8031c90cd679ae9fb4d2689e645205d1b403e970b2fbcc19249a8b851996bacf-1708612867,8031c90cd679ae9fb4d2689e645205d1b403e970b2fbcc19249a8b851996bacf,3.3,The CIL compiler in SELinux 3.2 has a use-after-free in __cil_verify_classperms (called from __cil_verify_classpermission and __cil_pre_verify_helper).,2.5-10.amzn2.0.1,CVE-2021-36084,https://www.cve.org/CVERecord?id=CVE-2021-36084,low,3.3,base,secpipe-core-prd-ip-10-xxx-xx-xx.eu-west-1.compute.internal,,False
2024-02-07 16:03:35.325 +0000 UTC,cvss:3.1/av:l/ac:l/pr:l/ui:n/s:u/c:h/i:n/a:n,libcurl3-gnutls:7.74.0-1.3+deb11u7,celery / secpipe-core-prd-ip-10-xxx-xx-xx.eu-west-1.compute.internal,6e86b864c9e8c64b006074335819dbfad83a183ddda541edf3e755586d25870c-1707316590,6e86b864c9e8c64b006074335819dbfad83a183ddda541edf3e755586d25870c,5.5,"An authentication bypass vulnerability exists in libcurl prior to v8.0.0 where it reuses a previously established SSH connection despite the fact that an SSH option was modified, which should have prevented reuse. libcurl maintains a pool of previously used connections to reuse them for subsequent transfers if the configurations match. However, two SSH settings were omitted from the configuration check, allowing them to match easily, potentially leading to the reuse of an inappropriate connection.",8.0.1-1.amzn2.0.1,CVE-2023-27538,https://www.cve.org/CVERecord?id=CVE-2023-27538,medium,5.5,base,secpipe-core-prd-ip-10-xxx-xx-xx.eu-west-1.compute.internal,,False
2024-02-07 16:10:15.317 +0000 UTC,av:l/ac:h/au:n/c:c/i:n/a:n,openssl:1.1.1w-0+deb11u1,deepfence-agent / secpipe-core-prd-ip-10-xxx-xx-xx.eu-west-1.compute.internal,d8e1a0a630121994c1f6cf7486c2573781827c0c7c779860ce88b9eb0777218e-1707316590,d8e1a0a630121994c1f6cf7486c2573781827c0c7c779860ce88b9eb0777218e,4.0,"OpenSSL 0.9.8i on the Gaisler Research LEON3 SoC on the Xilinx Virtex-II Pro FPGA uses a Fixed Width Exponentiation (FWE) algorithm for certain signature calculations, and does not verify the signature before providing it to a caller, which makes it easier for physically proximate attackers to determine the private key via a modified supply voltage for the microprocessor, related to a ""fault-based attack.""",,CVE-2010-0928,https://www.cve.org/CVERecord?id=CVE-2010-0928,medium,4.0,base,secpipe-core-prd-ip-10-xxx-xx-xx.eu-west-1.compute.internal,,False

[manuel-sommer]

@shodanwashere you can review the PR

[mtesauro]

@shodanwashere Do you have any docs/instructions/something on how someone else who has Threatmapper can reproduce the CSVs you've provided for the tool?

I'm wondering if having a parser that takes in CSVs that aren't easy to replicate for others makes sense or is just going to generate a bunch of Github issues we can't answer since we don't have experience with that tool.

[manuel-sommer]

@shodanwashere, could you please answer the question of @mtesauro? Then, I can finish the PR

[shodanwashere]

@shodanwashere, could you please answer the question of @mtesauro? Then, I can finish the PR

yes of course sorry, i've had some parallel issues on my side and didnt have the time to check this issue

@shodanwashere Do you have any docs/instructions/something on how someone else who has Threatmapper can reproduce the CSVs you've provided for the tool?

I'm wondering if having a parser that takes in CSVs that aren't easy to replicate for others makes sense or is just going to generate a bunch of Github issues we can't answer since we don't have experience with that tool.

Threatmapper currently only returns reports in either PDF (which is not easily consumable) and XLSX (Excel spreadsheet). The team that is administrating threatmapper at my organization wrote this simple python script to convert those spreadsheets to CSV:

import pandas as pd

# Path to your Excel file
excel_file = 'threatmapperreport.xlsx'

# Read the Excel file into a pandas DataFrame
df = pd.read_excel(excel_file)

# Path to save the CSV file
csv_file = 'threatmapperreport.csv'

# Save the DataFrame to a CSV file
df.to_csv(csv_file, index=False)

It's a crude method, but since Threatmapper doesn't export to more easily consumable formats like JSON, XML or even CSV, we had to come up with this method.

[manuel-sommer]

Hi @shodanwashere , could you then rather support .xlsx files as we could parse them here as well?

[manuel-sommer]

Reminder @shodanwashere

[manuel-sommer]

Reminder @shodanwashere. Could you please provide .xlsx files?

[shodanwashere]

ComplianceReport.xlsx MalwareReport.xlsx VulnerabilityReport.xlsx SecretReport.xlsx

@manuel-sommer here you go sorry for the late response

[ibreakthecloud]

@shodanwashere

Threatmapper currently only returns reports in either PDF (which is not easily consumable) and XLSX (Excel spreadsheet).

In ThreatMapper, if you use HTTP Endpoint integration, it sends in JSON.

[manuel-sommer]

Let's first get xlsx merged in the PR and then, I can make followup PR to also provide JSON

[manuel-sommer]

However, if you want, you can provide sample files in JSON @ibreakthecloud

[ibreakthecloud]

However, if you want, you can provide sample files in JSON @ibreakthecloud

@manuel-sommer attaching example json for vulnerability. LMK if you need files for other types of scans too.

context: ThreatMapper has an HTTP Endpoint where we can configure it with a webhook url, and resource(vulnerability, secret,malware,etc) and threatmapper will POST the result JSON on the url. If DefectDojo exposes an webhook url that supports POSTing JSON, it will work out of the box.

vulnerability-http-endpoint.json

p.s: I am one of the maintainers of Deepfence/ThreatMapper. Feel free to reach out, in case if you need any help.

[shodanwashere]

@manuel-sommer any updates?

[manuel-sommer]

Waiting for the pending Review

[shodanwashere]

Hey all! Any updates on the review needed for the pull request? It's been about a month since my last update request, but I got no response... if any of you could give me any feedback, I'd appreciate it a lot.

[mtesauro]

See #9688

[manuel-sommer]

@shodanwashere can you close this issue?