DependencyTrack / dependency-track

Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain.
https://dependencytrack.org/
Apache License 2.0
2.43k stars 529 forks source link

Improve performance of findings retrieval #3869

Closed nscuro closed 1 week ago

nscuro commented 1 week ago

Description

The /v1/finding/{projectUuid} endpoint has historically been slow to respond (#3811). While the "main" query behind it is somewhat optimized SQL already, it still suffered from various performance killers:

Performance was improved via the following changes:

  1. Filtering of suppressed findings is moved to the main SQL query, voiding the need to fetch individual Analysis records later. This also reduces the overall result set that needs to be transferred and mapped.
  2. Mapping of Clob fields is done within the Finding constructor, voiding the need to re-fetch Vulnerability records in order to retrieve String values for them.
  3. Aliases are loaded in bulk, and in a way that avoids redundant queries if the same Vulnerability appears multiple times within a list of Findings.
  4. Latest component versions are loaded in bulk, and in a way that avoids redundant queries if the same Component appears multiple times within a list of Findings.

Because the modified functionality is re-used across the code base, multiple features benefit from this enhancement:

[!NOTE] Tested with H2, MSSQL, and PostgreSQL.

Addressed Issue

Relates to #3811

Additional Details

Performance comparison with local Docker Compose setup:

cd dev
docker compose -f docker-compose.yml -f docker-compose.postgres.yml up -d

Details about the setup:

API server images compared:

Load tested the /api/v1/finding/${projectUuid} endpoint using hey:

hey -H "X-Api-Key: $APIKEY" \
    http://localhost:8080/api/v1/finding/project/d7b6156e-c58b-4b85-b3ef-db61e9667cd4
Results for snapshot ``` Summary: Total: 41.4942 secs Slowest: 11.3247 secs Fastest: 9.3886 secs Average: 10.3208 secs Requests/sec: 4.8199 Response time histogram: 9.389 [1] |■ 9.582 [3] |■■■ 9.776 [11] |■■■■■■■■■■ 9.969 [20] |■■■■■■■■■■■■■■■■■■ 10.163 [35] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 10.357 [44] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 10.550 [34] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 10.744 [18] |■■■■■■■■■■■■■■■■ 10.937 [17] |■■■■■■■■■■■■■■■ 11.131 [11] |■■■■■■■■■■ 11.325 [6] |■■■■■ Latency distribution: 10% in 9.8559 secs 25% in 10.0557 secs 50% in 10.2913 secs 75% in 10.5717 secs 90% in 10.9142 secs 95% in 11.0154 secs 99% in 11.2493 secs Details (average, fastest, slowest): DNS+dialup: 0.0010 secs, 9.3886 secs, 11.3247 secs DNS-lookup: 0.0004 secs, 0.0000 secs, 0.0025 secs req write: 0.0000 secs, 0.0000 secs, 0.0008 secs resp wait: 10.2449 secs, 9.3476 secs, 11.2671 secs resp read: 0.0749 secs, 0.0295 secs, 0.1779 secs Status code distribution: [200] 200 responses ```
Results for local ``` Summary: Total: 3.6591 secs Slowest: 1.5565 secs Fastest: 0.1958 secs Average: 0.8520 secs Requests/sec: 54.6580 Response time histogram: 0.196 [1] |■ 0.332 [4] |■■■■ 0.468 [10] |■■■■■■■■■■ 0.604 [16] |■■■■■■■■■■■■■■■ 0.740 [39] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 0.876 [31] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 1.012 [42] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 1.148 [35] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 1.284 [14] |■■■■■■■■■■■■■ 1.420 [7] |■■■■■■■ 1.557 [1] |■ Latency distribution: 10% in 0.5335 secs 25% in 0.6879 secs 50% in 0.8691 secs 75% in 1.0409 secs 90% in 1.1557 secs 95% in 1.2686 secs 99% in 1.4125 secs Details (average, fastest, slowest): DNS+dialup: 0.0010 secs, 0.1958 secs, 1.5565 secs DNS-lookup: 0.0005 secs, 0.0000 secs, 0.0025 secs req write: 0.0000 secs, 0.0000 secs, 0.0008 secs resp wait: 0.6837 secs, 0.1470 secs, 1.4515 secs resp read: 0.1673 secs, 0.0281 secs, 0.5001 secs Status code distribution: [200] 200 responses ```

Summary

Coming out to a fairly consistent 10x improvement.

Checklist

codacy-production[bot] commented 1 week ago

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
:white_check_mark: -0.04% (target: -1.00%) :white_check_mark: 71.56% (target: 70.00%)
Coverage variation details | | Coverable lines | Covered lines | Coverage | | ------------- | ------------- | ------------- | ------------- | | Common ancestor commit (bc2de4296c6f0d9f9e27db615230c177f77c064d) | 21698 | 16498 | 76.03% | | | Head commit (ba17eb211f421f804d97b9312a358ca0403e6c24) | 21787 (+89) | 16556 (+58) | 75.99% (**-0.04%**) | **Coverage variation** is the difference between the coverage for the head and common ancestor commits of the pull request branch: ` - `
Diff coverage details | | Coverable lines | Covered lines | Diff coverage | | ------------- | ------------- | ------------- | ------------- | | Pull request (#3869) | 109 | 78 | **71.56%** | **Diff coverage** is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: `/ * 100%`

See your quality gate settings    Change summary preferences


:rocket: Don’t miss a bit, follow what’s new on Codacy.

Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more