search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.27k stars 541 forks source link

Address security vulnarabilities in the image/binary/repo #3538

Closed humblec closed 2 months ago

humblec commented 1 year ago

Describe the bug

We are getting many reports against Ceph CSI image and the vulnerabilities it hold. it is required/better to address as much as we can. as part of this effort I have started enabling trvivy scanner in the repo via https://github.com/ceph/ceph-csi/pull/3537 and initial report says 

quay.io/cephcsi/cephcsi:test (redhat 8.6)
=========================================
Total: 14 (UNKNOWN: 0, LOW: 1, MEDIUM: 12, HIGH: 1, CRITICAL: 0)

┌───────────────────────┬────────────────┬──────────┬──────────────────────────────────────┬────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│        Library        │ Vulnerability  │ Severity │          Installed Version           │             Fixed Version              │                            Title                             │
├───────────────────────┼────────────────┼──────────┼──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ gnutls                │ CVE-2022-2509  │ MEDIUM   │ 3.6.16-5.el8                         │ 3.6.16-5.el8_6                         │ gnutls: Double free during gnutls_pkcs7_verify               │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-2509                    │
├───────────────────────┼────────────────┼──────────┼──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ libksba               │ CVE-2022-3515  │ HIGH     │ 1.3.5-7.el8                          │ 1.3.5-8.el8_6                          │ libksba: integer overflow may lead to remote code execution  │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-3515                    │
├───────────────────────┼────────────────┼──────────┼──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ platform-python       │ CVE-2015-20107 │ MEDIUM   │ 3.6.8-47.el8                         │ 3.6.8-47.el8_6                         │ python: mailcap: findmatch() function does not sanitize the  │
│                       │                │          │                                      │                                        │ second argument                                              │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2015-20107                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2022-0391  │          │                                      │                                        │ python: urllib.parse does not sanitize URLs containing ASCII │
│                       │                │          │                                      │                                        │ newline and tabs                                             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-0391                    │
├───────────────────────┼────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│ platform-python-devel │ CVE-2015-20107 │          │                                      │                                        │ python: mailcap: findmatch() function does not sanitize the  │
│                       │                │          │                                      │                                        │ second argument                                              │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2015-20107                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2022-0391  │          │                                      │                                        │ python: urllib.parse does not sanitize URLs containing ASCII │
│                       │                │          │                                      │                                        │ newline and tabs                                             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-0391                    │
├───────────────────────┼────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│ python3-libs          │ CVE-2015-20107 │          │                                      │                                        │ python: mailcap: findmatch() function does not sanitize the  │
│                       │                │          │                                      │                                        │ second argument                                              │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2015-20107                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2022-0391  │          │                                      │                                        │ python: urllib.parse does not sanitize URLs containing ASCII │
│                       │                │          │                                      │                                        │ newline and tabs                                             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-0391                    │
├───────────────────────┼────────────────┤          ├──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ python3-scipy         │ CVE-2021-20270 │          │ 1.0.0-21.module_el8.5.0+771+e5d9a225 │ 1.0.0-21.module+el8.5.0+10916+41bd434d │ python-pygments: Infinite loop in SML lexer may lead to DoS  │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-20270                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2021-27291 │          │                                      │                                        │ python-pygments: ReDoS in multiple lexers                    │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-27291                   │
├───────────────────────┼────────────────┤          ├──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ python36              │ CVE-2021-20270 │          │ 3.6.8-38.module_el8.5.0+895+a459eca8 │ 3.6.8-38.module+el8.5.0+12207+5c5719bc │ python-pygments: Infinite loop in SML lexer may lead to DoS  │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-20270                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2021-27291 │          │                                      │                                        │ python-pygments: ReDoS in multiple lexers                    │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-27291                   │
├───────────────────────┼────────────────┤          ├──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ sqlite-libs           │ CVE-2020-35527 │          │ 3.26.0-16.el8                        │ 3.26.0-16.el8_6                        │ sqlite: Out of bounds access during table rename             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2020-35527                   │
│                       ├────────────────┼──────────┤                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2020-35525 │ LOW      │                                      │                                        │ sqlite: Null pointer derreference in src/select.c            │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2020-35525                   │
└───────────────────────┴────────────────┴──────────┴──────────────────────────────────────┴────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Python (python-pkg)
===================
Total: 3 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 2)

┌───────────────────────────────────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│                      Library                      │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                            Title                             │
├───────────────────────────────────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ joblib (PKG-INFO)                                 │ CVE-2022-21797 │ CRITICAL │ 0.16.0            │ 1.2.0         │ The package joblib from 0 and before 1.2.0 are vulnerable to │
│                                                   │                │          │                   │               │ Arbitrary...                                                 │
│                                                   │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-21797                   │
├───────────────────────────────────────────────────┼────────────────┤          ├───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ scikit-learn (scikit_learn-0.19.1-py3.6.egg-info) │ CVE-2020-13092 │          │ 0.19.1            │ 0.23.1        │ ** DISPUTED ** scikit-learn (aka sklearn) through 0.23.0 can │
│                                                   │                │          │                   │               │ unseriali ...                                                │
│                                                   │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2020-13092                   │
│                                                   ├────────────────┼──────────┤                   ├───────────────┼──────────────────────────────────────────────────────────────┤
│                                                   │ CVE-2020-28975 │ HIGH     │                   │ 0.24.dev0     │ ** DISPUTED ** svm_predict_values in svm.cpp in Libsvm v324, │
│                                                   │                │          │                   │               │ as used in...                                                │
│                                                   │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2020-28975                   │
└───────────────────────────────────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘

usr/local/bin/cephcsi (gobinary)
================================
Total: 6 (UNKNOWN: 0, LOW: 0, MEDIUM: 4, HIGH: 1, CRITICAL: 1)

┌────────────────────────────┬────────────────┬──────────┬───────────────────┬────────────────────────────┬──────────────────────────────────────────────────────────────┐
│          Library           │ Vulnerability  │ Severity │ Installed Version │       Fixed Version        │                            Title                             │
├────────────────────────────┼────────────────┼──────────┼───────────────────┼────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/hashicorp/vault │ CVE-2020-16250 │ CRITICAL │ v1.4.2            │ 1.5.1, 1.5.1, 1.2.5, 1.3.8 │ Authentication Bypass by Spoofing                            │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2020-16250                   │
│                            ├────────────────┼──────────┤                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-32923 │ HIGH     │                   │ 1.7.2, 1.7.2, 1.7.2, 1.5.9 │ vault: Token leases incorrectly treated as non-expiring      │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-32923                   │
│                            ├────────────────┼──────────┤                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-38553 │ MEDIUM   │                   │ 1.8.0                      │ vault: Underlying database file with excessively broad       │
│                            │                │          │                   │                            │ filesystem permissions                                       │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-38553                   │
│                            ├────────────────┤          │                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-38554 │          │                   │ 1.6.6, 1.7.4               │ vault: UI erroneously cached and exposed user-viewed secrets │
│                            │                │          │                   │                            │ between sessions in a...                                     │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-38554                   │
│                            ├────────────────┤          │                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-41802 │          │                   │ 1.7.5, 1.8.4               │ vault: Incorrect Permission Assignment for Critical Resource │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-41802                   │
│                            ├────────────────┤          │                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-43998 │          │                   │ 1.7.6, 1.8.5               │ Incorrect Permission Assignment for Critical Resource        │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-43998                   │
└────────────────────────────┴────────────────┴──────────┴───────────────────┴────────────────────────────┴──────────────────────────────────────────────────────────────┘
0s
nixpanic commented 1 year ago

Except for the CVEs in usr/local/bin/cephcsi, you'll need to report them to the Ceph team that builds the base container-image.

The Vault ones look related to the server only, not to the API/client?

VladimirMarkelov commented 1 year ago

Besides mentioned CVEs for Hashicorp, there is a newer one: CVE-2022-36129 (9.1 Critical): https://cve.report/CVE-2022-36129

Fixed in 1.11.1, 1.10.5, and 1.9.8 - https://discuss.hashicorp.com/t/vault-1-11-1-1-10-5-and-1-9-8-released/42389

nixpanic commented 1 year ago

On Tue, Nov 22, 2022 at 11:35:32AM -0800, Vladimir Markelov wrote:

Besides mentioned CVEs for Hashicorp, there is a newer one: CVE-2022-36129 (9.1 Critical): https://cve.report/CVE-2022-36129

Fixed in 1.11.1, 1.10.5, and 1.9.8 - https://discuss.hashicorp.com/t/vault-1-11-1-1-10-5-and-1-9-8-released/42389

This CVE seems only applicable for the Vault server. Ceph-CSI uses the Vault client API only, so it is not affected.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

mohag commented 1 year ago

Except for the CVEs in usr/local/bin/cephcsi, you'll need to report them to the Ceph team that builds the base container-image.

It is possible update the packages from the Dockerfile... Updating it in the base image is better and smaller, but that works....

humblec commented 1 year ago

Except for the CVEs in usr/local/bin/cephcsi, you'll need to report them to the Ceph team that builds the base container-image.

It is possible update the packages from the Dockerfile... Updating it in the base image is better and smaller, but that works....

@mohag we do that here already ? https://github.com/ceph/ceph-csi/blob/devel/deploy/cephcsi/image/Dockerfile#L31 or you mean something we are missing ?

mohag commented 1 year ago

It is possible update the packages from the Dockerfile... Updating it in the base image is better and smaller, but that works....

@mohag we do that here already ? https://github.com/ceph/ceph-csi/blob/devel/deploy/cephcsi/image/Dockerfile#L31 or you mean something we are missing ?

Like that yes, but in the release image (somewhere under line 59) as well (that one is in the build image)

humblec commented 1 year ago

@mohag we can do that as well. do you want to submit a PR ? or I can do that. please let me know.

mohag commented 1 year ago

@mohag we can do that as well. do you want to submit a PR ? or I can do that. please let me know.

I'll attempt a PR.

mohag commented 1 year ago

A big part of the issue with the OS packages here is that the quay.io/centos/centos:8stream image seems to not be routinely updated. (the quay.io/ceph/ceph image uses that as a base) (It should be rebuilt every time an update to a package in that image is available. I could not track down the repo where those Dockerfiles are kept to try and nag them though)

jeroenlandheer commented 1 year ago

This is probably the one you're looking for: https://github.com/tgagor/docker-centos/blob/master/stream8/Dockerfile

Update: Nevermind, this wasn't the original image but an image that has a built-in update.

mohag commented 1 year ago

Let's see if we can get the underlying base images upgraded....

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

mohag commented 1 year ago

The base images have been upgraded

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

Starttoaster commented 6 months ago

Can we get this re-opened? Joblib in particular in the ceph-csi image raises a cve score of 9.8 https://avd.aquasec.com/nvd/2022/cve-2022-21797/

Seems potentially worth looking at since it's vulnerable to arbitrary code execution. I'm running v3.11.0 which seems to be the newest version, and is still vulnerable to this.

humblec commented 6 months ago

Can we get this re-opened? Joblib in particular in the ceph-csi image raises a cve score of 9.8 https://avd.aquasec.com/nvd/2022/cve-2022-21797/

Seems potentially worth looking at since it's vulnerable to arbitrary code execution. I'm running v3.11.0 which seems to be the newest version, and is still vulnerable to this.

Considering most of the vulnerabilities are in base image, thats the place we have to look into

Starttoaster commented 6 months ago

If it's not a necessary dependency, the option of uninstalling it from the image in the Dockerfile here is an option that is available as well. If it is a necessary dependency, there's not much to really do (maybe look for alternatives) since there's apparently no patched version.

I understand that the maintainers of ceph-csi might find that to be less than a "clean" solution. But ceph-csi is a product that is only expected to be ran in a container. So hardening the production image everyone uses seems like it shouldn't be an incredibly tall order for this project to take on, imho, even if it's a stopgap to getting the fix in the upstream base image. Certainly not such a tall order that a 9.8 score CVE stays in the production image for a year and a half (the approximate age of this issue.)

I'd even be happy to try my hand at helping contribute this fix if the maintainers here are open to the fix being implemented here. I mean... it's an arbitrary code execution vulnerability in an image running as root in my clusters with host mode networking and touches my storage clusters. I feel like that sounds like a pretty important thing to tighten up. If I'm being melodramatic let me know, but it seems like something worth acting on last year. That all being said, I'm extremely grateful for the tool, both this cluster client, and ceph in general are amazing. Absolutely wanted to underscore I'm not undermining the awesomeness of it, I just want it to be awesome and actually reasonably secure to run.

Starttoaster commented 6 months ago

If ceph-csi's maintainers are dead set on the vulnerability resolution being implemented in the upstream container, can we get a link to the Issue tracking it upstream over here? I'd very much appreciate it!

humblec commented 6 months ago

@Starttoaster Issues like : https://github.com/ceph/ceph-container/issues/2077 try to cover this request.

github-actions[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

Starttoaster commented 5 months ago

Still not stale. Relevant here but still tracking in ceph-container as well

Starttoaster commented 2 months ago

This might be somewhat solved now. There's still a critical vulnerability but now it's just from slightly out of date Go dependencies, since this switched to a base CentOS Stream 9 image. Thanks @Madhu-1 !!

Madhu-1 commented 2 months ago

Closing this one as we have updated to use centos 9 image. @Starttoaster Thanks for checking, feel free to open issue for go dependencies :)