/api/vulnerabilities has different behavior from /vulnerabilities from UI

[MihaiOnofrei]

Issues and Contributing

When entering /vulnerability/listing.jsp on my rapid7 instance i get 830 vulnerabilities that affect assets.

When requesting the vulnerabilities with the api i get 150k listings. There is no method to fetch only the vulnerabilities that affect assets.

It the same behavior from the UI achievable with this api?

[tschmidtke-r7]

Hi @MihaiOnofrei. The API does not directly provide this capability without retrieving the vulnerabilities for each asset individually (/api/3/assets/{id}/vulnerabilities). However, a SQL query report could be generated, downloaded, and parsed to obtain this data using significantly fewer API requests.

We have an example of the report creation, generation, and download process here: https://github.com/rapid7/vm-console-client-python/blob/master/samples/generate_sql_report.py

[MihaiOnofrei]

Unfortunately, both /api/3/assets/{id}/vulnerabilities and generated reports do not provide enough information about the vulnerabilities as the '/api/3/vulnerabilities' does.

I need to gather at least the fields: description, severity, categories, modified and some href for each vulnerability that affects an asset. Do you have any suggestion on what is the fastest way to achieve this via the API?

Current version I have is creating multiple async requests to get all the vulnerabilities( 500 per request) until I find the ones I need. The API does not support any filtering to access only the elements I need.

[tschmidtke-r7]

I think that a query similar to the one below may get you the data that you require (albeit with a bit of additional processing of list columns). It could be modified to filter for specific assets/vulnerabilities.

WITH urls AS (
  SELECT
    v.vulnerability_id,
    vr.reference
  FROM
    dim_vulnerability v
  JOIN dim_vulnerability_reference vr USING (vulnerability_id)
  WHERE vr.source = 'URL'
), cves AS (
    SELECT
    v.vulnerability_id,
    vr.reference
  FROM
    dim_vulnerability v
  JOIN dim_vulnerability_reference vr USING (vulnerability_id)
  WHERE vr.source = 'CVE'
)
SELECT
  da.asset_id,
  da.host_name,
  da.ip_address,
  dos.description AS operating_system,
  dv.title,
  htmltotext(dv.description) AS description,
  dv.severity,
  round(dv.cvss_score::numeric, 2) AS cvss_score,
  string_agg(DISTINCT dvc.category_name, ',') AS categories,
  string_agg(DISTINCT urls.reference, ',') AS urls,
  string_agg(DISTINCT cves.reference, ',') AS cves
FROM dim_asset da
JOIN dim_operating_system dos USING (operating_system_id)
JOIN fact_asset_vulnerability_finding favf USING (asset_id)
JOIN dim_vulnerability dv USING (vulnerability_id)
JOIN dim_vulnerability_category dvc USING (vulnerability_id)
JOIN urls USING (vulnerability_id)
JOIN cves USING (vulnerability_id)
GROUP BY da.asset_id, da.host_name, da.ip_address, dos.description,
  dv.title, dv.description, dv.severity, dv.cvss_score;

The results from that query are as follows:

asset_id	host_name	ip_address	operating_system	title	description	severity	cvss_score	categories	urls	cves
74436	host1	192.168.1.1	Microsoft Windows Server 2008 R2, Standard Edition SP1	IP Source Routing Enabled	The host is configured to honor IP source routing options. Source routing is a feature of the IP protocol which allows the sender of a packet to specify which route the packet should take on the way to its destination (and on the way back). Source routing was originally designed to be used when a host did not have proper default routes in its routing table. However, source routing is rarely used for legitimate purposes nowadays. Attackers can abuse source routing to bypass firewalls or to map your network.	Critical	7.50	Network	https://dl.packetstormsecurity.net/advisories/nai/nai.99-09-20.windows_ip_source_routing	CVE-1999-0510,CVE-1999-0909
74444	Host1	192.168.1.1	Microsoft Windows Server 2008 R2, Standard Edition SP1	Rapid7 Insight Agent: CVE-2017-5252: Insight Agent on Windows is vulnerable to loading malicious libraries placed in its dependency search path	Insight Agent on Windows systems searches for local dependencies in several locations, including in directories in the system PATH variable. As this can include arbitrary directories, and the Agent doesn't specify the directories to search, an attacker with local admin access could place a (potentially malicious) DLL in a directory in that path, causing the Agent to load that library.	Severe	6.80	Rapid7,Rapid7 Insight Agent	https://blog.rapid7.com/2017/10/06/vulnerabilities-affecting-four-rapid7-products-fixed	CVE-2017-5252
74445	ubuntu	192.168.1.2	Ubuntu Linux 14.04	Apache Struts: CVE-2014-0114: Arbitrary code execution	Apache Commons BeanUtils, as distributed in lib/commons-beanutils-1.8.0.jar in Apache Struts 1.x through 1.3.10 and in other products requiring commons-beanutils through 1.9.2, does not suppress the class property, which allows remote attackers to "manipulate" the ClassLoader and execute arbitrary code via the class parameter, as demonstrated by the passing of this parameter to the getClass method of the ActionForm object in Struts 1.	Critical	7.50	Apache,Apache Struts,J2EE,Remote Execution,Web	http://advisories.mageia.org/MGASA-2014-0219.html,http://commons.apache.org/proper/commons-beanutils/javadocs/v1.9.2/RELEASE-NOTES.txt,http://lists.fedoraproject.org/pipermail/package-announce/2014-August/136958.html,http://marc.info/?l=bugtraq&m=140119284401582&w=2,http://marc.info/?l=bugtraq&m=140801096002766&w=2,http://marc.info/?l=bugtraq&m=141451023707502&w=2,http://openwall.com/lists/oss-security/2014/06/15/10,http://openwall.com/lists/oss-security/2014/07/08/1,http://seclists.org/fulldisclosure/2014/Dec/23,http://secunia.com/advisories/57477,http://secunia.com/advisories/58710,http://secunia.com/advisories/58947,http://secunia.com/advisories/59118,http://secunia.com/advisories/59228,http://secunia.com/advisories/59245,http://secunia.com/advisories/59246,http://secunia.com/advisories/59430,http://secunia.com/advisories/59464,http://secunia.com/advisories/59479,http://secunia.com/advisories/59480,http://secunia.com/advisories/59718,http://www-01.ibm.com/support/docview.wss?uid=swg21674128,http://www-01.ibm.com/support/docview.wss?uid=swg21674812,http://www-01.ibm.com/support/docview.wss?uid=swg21675266,http://www-01.ibm.com/support/docview.wss?uid=swg21675387,http://www-01.ibm.com/support/docview.wss?uid=swg21675689,http://www-01.ibm.com/support/docview.wss?uid=swg21675898,http://www-01.ibm.com/support/docview.wss?uid=swg21675972,http://www-01.ibm.com/support/docview.wss?uid=swg21676091,http://www-01.ibm.com/support/docview.wss?uid=swg21676110,http://www-01.ibm.com/support/docview.wss?uid=swg21676303,http://www-01.ibm.com/support/docview.wss?uid=swg21676375,http://www-01.ibm.com/support/docview.wss?uid=swg21676931,http://www-01.ibm.com/support/docview.wss?uid=swg21677110,http://www-01.ibm.com/support/docview.wss?uid=swg27042296,http://www.debian.org/security/2014/dsa-2940,http://www.ibm.com/support/docview.wss?uid=swg21675496,http://www.mandriva.com/security/advisories?name=MDVSA-2014:095,http://www.oracle.com/technetwork/security-advisory/cpuoct2016-2881722.html,http://www.oracle.com/technetwork/topics/security/cpujan2015-1972971.html,http://www.oracle.com/technetwork/topics/security/cpujul2014-1972956.html,http://www.oracle.com/technetwork/topics/security/cpuoct2014-1972960.html,http://www.securityfocus.com/archive/1/archive/1/534161/100/0/threaded,http://www.securityfocus.com/bid/67121,http://www.vmware.com/security/advisories/VMSA-2014-0008.html,http://www.vmware.com/security/advisories/VMSA-2014-0012.html,https://access.redhat.com/solutions/869353,https://bugzilla.redhat.com/show_bug.cgi?id=1091938,https://bugzilla.redhat.com/show_bug.cgi?id=1116665,https://h20566.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay?docId=emr_na-c05324755,https://issues.apache.org/jira/browse/BEANUTILS-463,https://security.gentoo.org/glsa/201607-09	CVE-2014-0114
74445	ubuntu	192.168.1.2	Ubuntu Linux 14.04	Apache Struts: CVE-2015-0899: Remote attackers can bypass intended access restrictions via a modified page parameter	The MultiPageValidator implementation in Apache Struts 1 1.1 through 1.3.10 allows remote attackers to bypass intended access restrictions via a modified page parameter.	Severe	7.50	Apache,Apache Struts,J2EE,Web	http://jvn.jp/en/jp/JVN86448949/index.html,http://jvndb.jvn.jp/jvndb/JVNDB-2015-000042,http://www.securityfocus.com/bid/74423,https://en.osdn.jp/projects/terasoluna/wiki/StrutsPatch2-EN	CVE-2015-0899
74445	ubuntu	192.168.1.2	Ubuntu Linux 14.04	Apache Struts: CVE-2016-1181: Arbitrary code execution, denial of service	ActionServlet.java in Apache Struts 1 1.x through 1.3.10 mishandles multithreaded access to an ActionForm instance, which allows remote attackers to execute arbitrary code or cause a denial of service (unexpected memory access) via a multipart request, a related issue to CVE-2015-0899.	Severe	8.10	Apache,Apache Struts,Denial of Service,J2EE,Remote Execution,Web	http://jvn.jp/en/jp/JVN03188560/index.html,http://jvndb.jvn.jp/jvndb/JVNDB-2016-000096,http://www.oracle.com/technetwork/security-advisory/cpujul2016-2881720.html,http://www.oracle.com/technetwork/security-advisory/cpuoct2016-2881722.html,http://www.securityfocus.com/bid/91068,http://www.securityfocus.com/bid/91787,https://bugzilla.redhat.com/show_bug.cgi?id=1343538,https://github.com/kawasima/struts1-forever/commit/eda3a79907ed8fcb0387a0496d0cb14332f250e8,https://security-tracker.debian.org/tracker/CVE-2016-1181	CVE-2016-1181

[zyoutz-r7]

As Tyler mentioned, using a query is likely your best option when trying to retrieve very specific information - especially in such large quantity. A sample script of creating and generating a report can be found here: https://github.com/rapid7/vm-console-client-python/blob/master/samples/generate_sql_report.py. Hope this helps!