search

csaf-poc / csaf_distribution

Tools to download or provide CSAF (Common Security Advisory Framework) documents.
https://csaf.io
38 stars 22 forks source link

Make it easier to use csaf_distribution as go library to access CSAF file contents #367

Open bernhardreiter opened 1 year ago

bernhardreiter commented 1 year ago

To facilitate more usage of CSAF, how would a go library have to be constructed to be able to help implementors to access the contents of the advisories?

potential use cases

bernhardreiter commented 1 year ago

Ideas and considerations

bernhardreiter commented 1 year ago

publish the module on https://pkg.go.dev upstream steps

See following reports that affect us:

bernhardreiter commented 1 year ago

With the new https://github.com/csaf-poc/csaf_distribution/releases/tag/v2.2.0 release and #371 done, we are now listed on https://pkg.go.dev/github.com/csaf-poc/csaf_distribution/v2

@knqyf263 @mpermar @beltran-rubo with the above improvements, you shall be able to technically experiment using the csaf_distribution v2 code as go modules. Note that this is not an official API. Still experimenting with it, may help you to see how close we are already to a usable library.

bernhardreiter commented 1 year ago

@tschmidtb51 this issue itself is a rough concept of how to access go as a library. My suggestion is: We remove the service+dev label, as the part of doing a rough concept is done.

And then we rename the issue to something like "Make it easier to use csaf_distribution as go library to access CSAF file contents". Because the links are useful to keep in an open issue.

mpermar commented 1 year ago

@bernhardreiter I've been looking at the code, and to be honest I never have done much golang so it is pretty likely that I'm missing the obvious here, but any hints on how could I use the library?

The typical use case for supporting Trivy VEX filtering flow is to read a CSAF VEX file, then loop over the product tree finding purl packages and use that information to match vulnerabilities and detect which vulnerabilities can be filtered out through the VEX statuses. And finally the VEX code returns the filtered list of vulnerabilities. This can be seen here for CycloneDX and OpenVex.

When looking at the csaf package, I'm not totally sure how I can import a struct that I could use for unmarshalling the CSAF file and run the logic above.

{
    "document": {
      "category": "csaf_vex",
      "csaf_version": "2.0",
      "notes": [
        {
          "category": "summary",
          "text": "Example Company VEX document. Unofficial content for demonstration purposes only.",
          "title": "Author comment"
        }
      ],
      "publisher": {
        "category": "vendor",
        "name": "Example Company ProductCERT",
        "namespace": "https://psirt.example.com"
      },
      "title": "AquaSecurity example VEX document",
      "tracking": {
        "current_release_date": "2022-03-03T11:00:00.000Z",
        "generator": {
          "date": "2022-03-03T11:00:00.000Z",
          "engine": {
            "name": "Secvisogram",
            "version": "1.11.0"
          }
        },
        "id": "2022-EVD-UC-01-A-001",
        "initial_release_date": "2022-03-03T11:00:00.000Z",
        "revision_history": [
          {
            "date": "2022-03-03T11:00:00.000Z",
            "number": "1",
            "summary": "Initial version."
          }
        ],
        "status": "final",
        "version": "1"
      }
    },
    "product_tree": {
      "branches": [
        {
          "branches": [
            {
              "branches": [
                {
                  "category": "product_version",
                  "name": "2.6.0",
                  "product": {
                    "name": "Spring Boot 2.6.0",
                    "product_id": "SPB-00260",
                    "product_identification_helper": {
                      "purl": "pkg:maven/org.springframework.boot/spring-boot@2.6.0"
                    }
                  }
                }
              ],
              "category": "product_name",
              "name": "Spring Boot"
            }
          ],
          "category": "vendor",
          "name": "Spring"
        }
      ]
    },
    "vulnerabilities": [
      {
        "cve": "CVE-2021-44228",
        "notes": [
          {
            "category": "description",
            "text": "Apache Log4j2 2.0-beta9 through 2.15.0 (excluding security releases 2.12.2, 2.12.3, and 2.3.1) JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. From version 2.16.0 (along with 2.12.2, 2.12.3, and 2.3.1), this functionality has been completely removed. Note that this vulnerability is specific to log4j-core and does not affect log4net, log4cxx, or other Apache Logging Services projects.",
            "title": "CVE description"
          }
        ],
        "product_status": {
          "known_not_affected": [
            "SPB-00260"
          ]
        },
        "threats": [
          {
            "category": "impact",
            "details": "Class with vulnerable code was removed before shipping.",
            "product_ids": [
              "SPB-00260"
            ]
          }
        ]    
      }
    ]
  }
mpermar commented 1 year ago

To add on the above. Here is a possible implementation on CSAF filtering based on very simple and incomplete structs. I've just gone and make the basic filtering unit test pass just to make the point and better express the use case. As you will notice, the code essentially loops over CVEs from a Trivy report and then for each of the CVEs it tries to find if exists a VEX statement for that CVE id marked as fixed or known_not_affected and if so then the CVE is discarded from the Trivy report.

That test is passing but using hardcoded structs on Trivy itself. The model plus the utility methods to work through the model is what I would expect from a CSAF golang library. One of those utilities could be perhaps for example find the identifications (purls, cpes, sbom_urls) for a given vulnerability id. Just like the code tries naively to do here.

bernhardreiter commented 1 year ago

@mpermar though we believe to have a pretty complete model to be useful, there maybe model parts missing depending on the use case. So thanks a lot for your question and the details in the request!

(The last weeks were summer holiday time here in Germany. So we can only start to look at this in the next days.)

One goal of this issue is to find out what is missing (in terms of model parts, helping functions, documentation or whatever) to make this a useful library. Please keep in mind that csaf_distribution does not promise an API currently, only experimental access to the internal possibilities. ;)

mpermar commented 1 year ago

Hey @bernhardreiter , likewise on PTO here for the next few days, but just wanted to chime in to mention that if I recall it right pretty much most of the VEX profile objects were missing so the example above could not be implemented. Have a look at the code. General CSAF concepts like Publisher or Provider do exist but I cannot find anything related to a Vulnerability or product statuses for example.

bernhardreiter commented 1 year ago

@mpermar

pretty much most of the VEX profile objects were missing

thanks for adding more detail. I think our devs are also thinking along those lines and we see if someone can come up with a simple example. Enjoy your PTO!

cintek commented 1 year ago

@mpermar I'm working on a fork for a PR to add the code that is necessary to work with csaf_distribution as a library. The most interesting part is the new file advisory.go.

Then I have an example which shows how to find PURLs in an advisory by providing product IDs. That is one functionality of those in your linked comparison.

With the additional structs and functions you could also edit an existing advisory and save it as a new file.

mpermar commented 12 months ago

Hey @cintek , @bernhardreiter , I got some time to redo the experimental filter on my fork with the work you did there and it worked perfectly!

Here is an example of how this works. With the following SBOM running a Trivy scan will show:

bash-3.2$ trivy sbom trivy.sbom.cdx
2023-09-15T18:25:55.095+0200    INFO    Vulnerability scanning is enabled
2023-09-15T18:25:55.100+0200    INFO    Detected SBOM format: cyclonedx-json
2023-09-15T18:25:55.110+0200    WARN    Ignore the OS package as no OS information is found.
2023-09-15T18:25:55.156+0200    INFO    Number of language-specific files: 1
2023-09-15T18:25:55.156+0200    INFO    Detecting gomod vulnerabilities...

go.mod (gomod)

Total: 3 (UNKNOWN: 0, LOW: 0, MEDIUM: 3, HIGH: 0, CRITICAL: 0)

┌───────────────────────────────────────┬─────────────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│                Library                │    Vulnerability    │ Severity │ Installed Version │ Fixed Version │                            Title                             │
├───────────────────────────────────────┼─────────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/cloudflare/circl           │ CVE-2023-1732       │ MEDIUM   │ 1.1.0             │ 1.3.3         │ Improper random reading in CIRCL                             │
│                                       │                     │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-1732                    │
├───────────────────────────────────────┼─────────────────────┤          ├───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/cyphar/filepath-securejoin │ GHSA-6xv5-86q9-7xr8 │          │ 0.2.3             │ 0.2.4         │ SecureJoin: on windows, paths outside of the rootfs could be │
│                                       │                     │          │                   │               │ inadvertently produced...                                    │
│                                       │                     │          │                   │               │ https://github.com/advisories/GHSA-6xv5-86q9-7xr8            │
├───────────────────────────────────────┼─────────────────────┤          ├───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/sigstore/rekor             │ CVE-2023-33199      │          │ 1.1.1             │ 1.2.0         │ malformed proposed intoto entries can cause a panic          │
│                                       │                     │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-33199                   │
└───────────────────────────────────────┴─────────────────────┴──────────┴───────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘

Now, if we create a CSAF VEX assessment for CVE-2023-1732 for example, and we run the command again:

bash-3.2$ ./trivy sbom trivy.sbom.cdx --vex trivy.vex.csaf
2023-09-15T18:28:36.082+0200    INFO    Vulnerability scanning is enabled
2023-09-15T18:28:36.084+0200    INFO    Detected SBOM format: cyclonedx-json
2023-09-15T18:28:36.093+0200    WARN    Ignore the OS package as no OS information is found.
2023-09-15T18:28:36.135+0200    INFO    Number of language-specific files: 1
2023-09-15T18:28:36.135+0200    INFO    Detecting gomod vulnerabilities...
2023-09-15T18:28:36.178+0200    INFO    Detected VEX format: CSAF
Found URLs for SPB-00260:
1. pkg:golang/github.com/cloudflare/circl@1.1.0
2023-09-15T18:28:36.178+0200    INFO    Found the following VEX statements: %!w([]vex.Statement=[{CVE-2023-1732 [pkg:golang/github.com/cloudflare/circl@1.1.0] not_affected TODO}])
2023-09-15T18:28:36.178+0200    INFO    Filtering with filter CSAF
2023-09-15T18:28:36.178+0200    INFO    Filtering with filter CSAF
2023-09-15T18:28:36.178+0200    INFO    Filtered out the detected vulnerability {"VEX format": "CSAF", "vulnerability-id": "CVE-2023-1732", "status": "not_affected", "justification": "TODO"}
2023-09-15T18:28:36.178+0200    INFO    Filtering with filter CSAF

go.mod (gomod)

Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 2, HIGH: 0, CRITICAL: 0)

┌───────────────────────────────────────┬─────────────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│                Library                │    Vulnerability    │ Severity │ Installed Version │ Fixed Version │                            Title                             │
├───────────────────────────────────────┼─────────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/cyphar/filepath-securejoin │ GHSA-6xv5-86q9-7xr8 │ MEDIUM   │ 0.2.3             │ 0.2.4         │ SecureJoin: on windows, paths outside of the rootfs could be │
│                                       │                     │          │                   │               │ inadvertently produced...                                    │
│                                       │                     │          │                   │               │ https://github.com/advisories/GHSA-6xv5-86q9-7xr8            │
├───────────────────────────────────────┼─────────────────────┤          ├───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/sigstore/rekor             │ CVE-2023-33199      │          │ 1.1.1             │ 1.2.0         │ malformed proposed intoto entries can cause a panic          │
│                                       │                     │          │                   │               │ https://avd.aquasec.com/nvd/cve-2023-33199                   │
└───────────────────────────────────────┴─────────────────────┴──────────┴───────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘

And the CVE is gone from the scanner results 👏

mpermar commented 12 months ago

@bernhardreiter , @cintek , I think this is great and should work for sending some PRs for the scanners to support this type of native filtering for CSAF (like Trivy does today with OpenVEX/CycloneDX-VEX). Do you think it is realistic to have a golang package available in the short term? We can take care of sending the PRs to the scanners.

Also a couple of observations @cintek, Trivy requires golang 1.20 but this CSAF distribution library requires golang 1.21. My golang knowledge is very basic so not sure if that would work.

Other one, I wonder if it could be possible to add product finding as a library util in this module. I've implemented it on the client side for the sake of this test, but due to the number of different product identification helpers it kind of feels that maybe it should be on the library side.

And last one by the way, I initially was trying with SBOM links from CycloneDX, but those don't seem to be supported by CSAF. Or I presume that for SBOM Links one should use sbom_urls?

Great weekend!

mpermar commented 11 months ago

Any news here. Is there any other way we can help to accelerate adoption? We are happy to help you guys to put this in prod if there is anything else we can help with here.

tschmidtb51 commented 11 months ago

Unfortunately not. We are working in the background on some bureaucracy to push the topic forward. Keeping you posted.

mpermar commented 10 months ago

Hey, I saw the advisory model is now merged and looks very promising. From your perspective, anything left to do in that regard?

Also, I'm kind of curious about the csaf-poc naming. Our team is interested in contributing now CSAF VEX profile integrations to different scanners just like the PoC above and what exists in Trivy today, and what others contributed to Grype recently to support OpenVEX. But I could understand some eyebrows on the poc part.

Any thoughts?

tschmidtb51 commented 10 months ago

Hey, I saw the advisory model is now merged and looks very promising. From your perspective, anything left to do in that regard?

I'll leave that to @csaf-poc/dev to answer.

tschmidtb51 commented 10 months ago

Also, I'm kind of curious about the csaf-poc naming. Our team is interested in contributing now CSAF VEX profile integrations to different scanners just like the PoC above and what exists in Trivy today, and what others contributed to Grype recently to support OpenVEX. But I could understand some eyebrows on the poc part.

Any thoughts?

The poc part has his historic reasons: Originally, the project was only intended to be a PoC but then it was so successful that it became the reference implementation. Now there are some concerns about changing the location as it is already always advertised under this name...

Nevertheless, it is supported, production-ready and used in production...

bernhardreiter commented 10 months ago

Hey, I saw the advisory model is now merged and looks very promising. From your perspective, anything left to do in that regard?

At least I do not know about specific todos on our list. Maybe @s-l-teichmann or @cintek know more.

Please note that some of the disclaimers from above still hold, this is not yet a fully supported API, however it can be used to build software that may even be ready for some real use cases, though you have to take responsibility for it yourself. (The go modules versioning should prevent at least some accidents.)

A while ago you've also asked:

Trivy requires golang 1.20 but this CSAF distribution library requires golang 1.21. My golang knowledge is very basic so not sure if that would work.

Go 1 comes with a promisse of compatibility, which for 1.21 means: expect almost all Go programs to continue to compile and run as before.

In practice you should be able to build Trivy with Go 1.21 without problems and then you can use csaf_distribution as a dependency.

I wonder if it could be possible to add product finding as a library util in this module.

Please open an issue for this wish. (It sound like a good idea in principle.)

I initially was trying with SBOM links from CycloneDX, but those don't seem to be supported by CSAF. Or I presume that for SBOM Links one should use sbom_urls?

That is a question for @tschmidtb51 ;)

tschmidtb51 commented 10 months ago

I initially was trying with SBOM links from CycloneDX, but those don't seem to be supported by CSAF. Or I presume that for SBOM Links one should use sbom_urls?

That is a question for @tschmidtb51 ;)

Sorry - I missed that. sbom_urls are just the download URLs for SBOMs. (However, as each version has a different SBOM, that gives you a nice product identifier.) For the BOM-Link format to link to a specific component, you should use the x_generic_uris. See also examples 16 and 17 in the specification and the documentation in https://github.com/oasis-tcs/csaf/issues/484.

Does that answer your question?

mpermar commented 10 months ago

Sure it does. Thanks @tschmidtb51 !

Also thanks @bernhardreiter ! Opened a request. We're going to start working on contributions to some of the other OSS scanners. We will check the Golang 1.21. It should work, a few weeks ago they were struggling with some Golang 1.21 dependency but it seems they got around those issues. Will ping back soon.

oxisto commented 4 months ago

publish the module on https://pkg.go.dev upstream steps

See following reports that affect us:

Hi! We are currently trying to evaluate integrating CSAF support into our cloud compliance framework Clouditor (https://github.com/clouditor/clouditor). However, the warning page on https://pkg.go.dev/github.com/csaf-poc/csaf_distribution/v3/csaf was a little bit concerning. The contents in csaf are MIT-licensed, correct? Is there a specific reasons for the LICENSES folder that also contains the Go license? Are you directly using parts of the Go code in this somehow?

As long as its clear the the csaf and util folders are MIT-licensed (as it looks to be from the headers), it should be ok for us, but we might need to silence some automated license checkers for that, we will see.

tschmidtb51 commented 4 months ago

Hi! We are currently trying to evaluate integrating CSAF support into our cloud compliance framework Clouditor (https://github.com/clouditor/clouditor). However, the warning page on https://pkg.go.dev/github.com/csaf-poc/csaf_distribution/v3/csaf was a little bit concerning. The contents in csaf are MIT-licensed, correct? Is there a specific reasons for the LICENSES folder that also contains the Go license? Are you directly using parts of the Go code in this somehow?

I did some digging: We added the license file with https://github.com/csaf-poc/csaf_distribution/commit/9cba4eec30bc914a6cd1cbb7086f8e4047cfd211 to resolve #223. As Go 1.19 was not released at that point, we imported the file from the Go Code. It was removed with https://github.com/csaf-poc/csaf_distribution/commit/6b9ecead89c5b40e86928c6e7f416903e0a495e1 which IMHO forgot to remove the License file.

@s-l-teichmann / @bernhardreiter : Is that correct? Can we remove the file safely or did I miss something?

bernhardreiter commented 4 months ago

@tschmidtb51 yes, we can remove that license file safely. #544 does so.

@oxisto the warning at https://pkg.go.dev/github.com/csaf-poc/csaf_distribution/v3/csaf is highly missleading, as explained on the linked issues we've opened with the Go infrastructure. Anyhow, the LicenseRef-Go119-BSD-Patentgrant.txt wouldn't be any problem for you. And it is quite usual to have files with several licenses within a repository. We do have one with BSD-3-Clause inside as well, see #542.