Sage-Bionetworks / sage-monorepo

Where OpenChallenges, Schematic, and other Sage open source apps are built
https://sage-bionetworks.github.io/sage-monorepo/
Apache License 2.0
21 stars 12 forks source link

[Story] Profile the storage usage of Trivy #1743

Closed tschaffter closed 1 year ago

tschaffter commented 1 year ago

What projects is this story for?

No response

As a user, I want

NA

Description

We are bumping into an issue where the CI workflow does not have enough storage to build, scan and publish a large number of images (>5?). The goal of this ticket is to evaluate:

Acceptance criteria

No response

Tasks

No response

Anything else?

No response

Have you linked this story to a GitHub Project?

tschaffter commented 1 year ago

On the GH runner, after building, testing most projects, before applying any steps relative to images:

Run df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        84G   70G   14G  84% /
tmpfs           3.4G  172K  3.4G   1% /dev/shm
tmpfs           1.4G  1.2M  1.4G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb15      105M  6.1M   99M   6% /boot/efi
/dev/sda1        14G  4.1G  9.0G  31% /mnt
tmpfs           694M   12K  694M   1% /run/user/1001

Docker artifacts, likely relative to the dev container:

Note Actually these are probably images from the Docker cache (from a previous CI workflow run).


Run docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          22        1         8.554GB   8.032GB (93%)
Containers      1         1         1.532GB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     22        0         77.32kB   77.32kB
tschaffter commented 1 year ago
vscode@ad5701e36f83:/workspaces/sage-monorepo$ trivy image ghcr.io/sage-bionetworks/openchallenges-zipkin:edge
2023-07-18T02:30:55.651Z        INFO    Need to update DB
2023-07-18T02:30:55.651Z        INFO    DB Repository: ghcr.io/aquasecurity/trivy-db
2023-07-18T02:30:55.651Z        INFO    Downloading DB...
38.44 MiB / 38.44 MiB [-----------------------------------------------------------] 100.00% 30.42 MiB p/s 1.5s
2023-07-18T02:30:59.396Z        INFO    Vulnerability scanning is enabled
2023-07-18T02:30:59.396Z        INFO    Secret scanning is enabled
2023-07-18T02:30:59.396Z        INFO    If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-07-18T02:30:59.396Z        INFO    Please see also https://aquasecurity.github.io/trivy/v0.43/docs/scanner/secret/#recommendation for faster secret detection
2023-07-18T02:31:01.495Z        INFO    JAR files found
2023-07-18T02:31:01.495Z        INFO    Java DB Repository: ghcr.io/aquasecurity/trivy-java-db:1
2023-07-18T02:31:01.495Z        INFO    Downloading the Java DB...
443.28 MiB / 443.28 MiB [----------------------------------------------------------] 100.00% 43.25 MiB p/s 10s
2023-07-18T02:31:12.940Z        INFO    The Java DB is cached for 3 days. If you want to update the database more frequently, the '--reset' flag clears the DB cache.
2023-07-18T02:31:12.953Z        INFO    Analyzing JAR files takes a while...
2023-07-18T02:31:13.011Z        INFO    Detected OS: alpine
2023-07-18T02:31:13.011Z        INFO    Detecting Alpine vulnerabilities...
2023-07-18T02:31:13.014Z        INFO    Number of language-specific files: 1
2023-07-18T02:31:13.014Z        INFO    Detecting jar vulnerabilities...

ghcr.io/sage-bionetworks/openchallenges-zipkin:edge (alpine 3.18.0)

Total: 4 (UNKNOWN: 0, LOW: 2, MEDIUM: 0, HIGH: 2, CRITICAL: 0)

┌────────────┬───────────────┬──────────┬───────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│  Library   │ Vulnerability │ Severity │ Installed Version │ Fixed Version │                            Title                            │
├────────────┼───────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ libcrypto3 │ CVE-2023-2650 │ HIGH     │ 3.1.0-r4          │ 3.1.1-r0      │ Possible DoS translating ASN.1 object identifiers           │
│            │               │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-2650                   │
│            ├───────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│            │ CVE-2023-2975 │ LOW      │                   │ 3.1.1-r2      │ Issue summary: The AES-SIV cipher implementation contains a │
│            │               │          │                   │               │ bug that c ......                                           │
│            │               │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-2975                   │
├────────────┼───────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│ libssl3    │ CVE-2023-2650 │ HIGH     │                   │ 3.1.1-r0      │ Possible DoS translating ASN.1 object identifiers           │
│            │               │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-2650                   │
│            ├───────────────┼──────────┤                   ├───────────────┼─────────────────────────────────────────────────────────────┤
│            │ CVE-2023-2975 │ LOW      │                   │ 3.1.1-r2      │ Issue summary: The AES-SIV cipher implementation contains a │
│            │               │          │                   │               │ bug that c ......                                           │
│            │               │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-2975                   │
└────────────┴───────────────┴──────────┴───────────────────┴───────────────┴─────────────────────────────────────────────────────────────┘
2023-07-18T02:31:13.022Z        INFO    Table result includes only package filenames. Use '--format json' option to get the full path to the package file.

Java (jar)

Total: 5 (UNKNOWN: 0, LOW: 0, MEDIUM: 3, HIGH: 2, CRITICAL: 0)

┌─────────────────────────────────────────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────────┐
│                         Library                         │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                           Title                           │
├─────────────────────────────────────────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────────┤
│ io.netty:netty-codec-haproxy                            │ CVE-2022-41881 │ HIGH     │ 4.1.78.Final      │ 4.1.86.Final  │ HAProxyMessageDecoder Stack Exhaustion DoS                │
│ (netty-codec-haproxy-4.1.78.Final.jar)                  │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-41881                │
├─────────────────────────────────────────────────────────┼────────────────┼──────────┤                   ├───────────────┼───────────────────────────────────────────────────────────┤
│ io.netty:netty-handler (netty-handler-4.1.78.Final.jar) │ CVE-2023-34462 │ MEDIUM   │                   │ 4.1.94.Final  │ Netty is an asynchronous event-driven network application │
│                                                         │                │          │                   │               │ framework fo ...                                          │
│                                                         │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-34462                │
├─────────────────────────────────────────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────────┤
│ org.xerial.snappy:snappy-java (snappy-java-1.1.8.4.jar) │ CVE-2023-34455 │ HIGH     │ 1.1.8.4           │ 1.1.10.1      │ snappy-java's unchecked chunk length leads to DoS         │
│                                                         │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-34455                │
│                                                         ├────────────────┼──────────┤                   │               ├───────────────────────────────────────────────────────────┤
│                                                         │ CVE-2023-34453 │ MEDIUM   │                   │               │ snappy-java's Integer Overflow vulnerability in shuffle   │
│                                                         │                │          │                   │               │ leads to DoS                                              │
│                                                         │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-34453                │
│                                                         ├────────────────┤          │                   │               ├───────────────────────────────────────────────────────────┤
│                                                         │ CVE-2023-34454 │          │                   │               │ snappy-java's Integer Overflow vulnerability in compress  │
│                                                         │                │          │                   │               │ leads to DoS                                              │
│                                                         │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-34454                │
└─────────────────────────────────────────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────────┘

Cache location:

$ du -h ~/.cache/trivy/
88K     /home/vscode/.cache/trivy/fanal
378M    /home/vscode/.cache/trivy/db
732M    /home/vscode/.cache/trivy/java-db
1.1G    /home/vscode/.cache/trivy/

Conclusion

Trivy DB takes about 1.1 GB.