Add validation tests

[tschmidtb51]

We have some parts which can't be enforced be the schema itself but should be tested programmatically. Therefore, I suggest to add those tests to the standard to ease the pain. Here are some ideas:

• Test whether definition of product_id exists

  • Elements of type "#/definitions/product_id_t" (implicit: "#/definitions/products_t") which are not inside a Full Product Name (full_product_name_t) and therefore reference an element within the product_tree => test whether the Full Product Name with the product_id exists
  • relevant paths: click to expand

    ``` /product_tree/product_groups[]/product_ids[] /product_tree/relationships[]/product_reference /product_tree/relationships[]/relates_to_product_reference /vulnerabilities[]/product_status/first_affected[] /vulnerabilities[]/product_status/first_fixed[] /vulnerabilities[]/product_status/fixed[] /vulnerabilities[]/product_status/known_affected[] /vulnerabilities[]/product_status/known_not_affected[] /vulnerabilities[]/product_status/last_affected[] /vulnerabilities[]/product_status/recommended[] /vulnerabilities[]/product_status/under_investigation[] /vulnerabilities[]/remediations[]/product_ids[] /vulnerabilities[]/scores[]/products[] /vulnerabilities[]/threats[]/product_ids[] ```

• Test whether the definition of product_id ends up in a circle Example:

  "product_tree": {
      "full_product_names": [
        {
      "product_id": "CSAFPID-0001",
      "name": "Product A"
        }
      ],
      "relationships": [
        {
      "full_product_name": {
        "name": "Product B",
        "product_id": "CSAFPID-0002"
      },
      "product_reference": "CSAFPID-0001",
      "relates_to_product_reference": "CSAFPID-0002",
      "relationship_type": "installed_on"
        }
      ]
    }

• Test whether definition of group_id exists

  • Elements of type "#/definitions/product_group_id_t" (implicit: "#/definitions/product_groups_t") which are not inside a Product Group (product_groups[]) and therefore reference an element within the product_tree => test whether the Product Group with the group_id exists
  • relevant paths: click to expand

    ``` /vulnerabilities[]/remediations[]/group_ids /vulnerabilities[]/threats[]/group_ids ```

• Test whether product_id was already defined

  • For each product_id (type: "#/definitions/product_id_t") in Full Product Name elements (type: "#/definitions/full_product_name_t"): was the product_id already defined (within the document)?
  • relevant paths: click to expand

    ``` /product_tree/branches[](/...)/product/product_id /product_tree/full_product_names[]/product_id /product_tree/relationships[]/full_product_name/product_id ```

• Test whether group_id was already defined

  • For each group_id (type: "#/definitions/product_group_id_t") in Product Group elements (type: "product_tee/product_groups[]/"): was the group_id already defined (within the document)?
  • relevant path: click to expand

    ``` /product_tree/product_groups[]/group_id ```

• Test whether product_status has contradicting elements

  • The same product_id should not be in contradicting arrays, e.g. fixed and known_affected
  • contradicting groups: click to expand

    - Affected ``` /vulnerabilities[]/product_status/first_affected[] /vulnerabilities[]/product_status/known_affected[] /vulnerabilities[]/product_status/last_affected[] ``` - Not affected ``` /vulnerabilities[]/product_status/known_not_affected[] ``` - Fixed ``` /vulnerabilities[]/product_status/first_fixed[] /vulnerabilities[]/product_status/fixed[] ``` - Under investigation ``` /vulnerabilities[]/product_status/under_investigation[] ``` Note: An issuer might `recommend`a version from any group - also from the affected group i.e. if it is harder to exploit.

• Test whether products has elements in other items of scores within one vulnerability

  • The same product_id should not be listed with more then one CVSS-Vectors with the same version. Example:

    "scores": [
    {
      "products": [
        "CSAFPID-0001",
        "CSAFPID-0007",
        "CSAFPID-0008",
        "CSAFPID-0017"
      ],
      "cvss_v3": {
        "version": "3.1",
        "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H",
        "baseScore": 10,
        "baseSeverity": "CRITICAL",
        "attackVector": "NETWORK",
        "attackComplexity": "LOW",
        "privilegesRequired": "NONE",
        "userInteraction": "NONE",
        "scope": "CHANGED",
        "confidentialityImpact": "HIGH",
        "integrityImpact": "HIGH",
        "availabilityImpact": "HIGH"
      }
    },
    {
      "products": [
        "CSAFPID-0017",
        "CSAFPID-0018"
      ],
      "cvss_v3": {
        "version": "3.1",
        "vectorString": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H",
        "baseScore": 9.8,
        "baseSeverity": "CRITICAL",
        "attackVector": "NETWORK",
        "attackComplexity": "LOW",
        "privilegesRequired": "NONE",
        "userInteraction": "NONE",
        "scope": "UNCHANGED",
        "confidentialityImpact": "HIGH",
        "integrityImpact": "HIGH",
        "availabilityImpact": "HIGH"
      }
    }
    ],

    "CSAFPID-0017" would break that test

• Test whether CVSS is valid (according to JSON schema)

• Test whether CVSS is computed correctly

• Test whether CWE exists and is valid

• Test whether language_t is valid and exists

• Test whether date in elements of revision_history are ascending when sorting the elements by number (according to version_t schema)

• Test whether definition of product_id is used in the advisory

[tschmidtb51]

We should also take that into account in #193 and #191.

[mprpic]

+1 for implementing validation checks. I assume this could be baked into the existing parser library (https://github.com/oasis-open/csaf-parser), though I guess it should be updated to support CSAF 2.0 first. Or do we think the community would prefer a standalone library for parsing and validation for CSAF 2.0 (I don't really have an opinion on this)?

[tschmidtb51]

I'm not sure what the current status of the csaf-parser is and whether it is written in a way that it can be upgraded easily.

Maybe @sthagen can elaborate on that...

As a user, I would probably prefer a library which I can include in my own implementation...

[sthagen]

I am open to support csaf-parser but it may have a lot of baggage from the past.

I will start to implement the suggested schematron like tests of this ticket in csaf-lint.

But, not tomorrow, as tomorrow I have a day off ... anything 😄

[tschmidtb51]

This is also related to #214. The following tests are mentioned there:

• Version 0 and 0.y.z must have document status draft
• Each Revision item which has a number of 0 or 0.y.z MUST be removed from the document if the document status is final.
• Versions of the document which are pre-release SHALL NOT have its own revision item.
• Pre-release MUST NOT be included if /document/tracking/status is final.
• Revision items should not include build metadata.
• CSAF content management systems must check the correctness of versioning

[tschmidtb51]

This is also related to #202. The following test is mentioned:

• source_lang must be present and set if the value translator is used for /document/publisher/category

[sthagen]

One question placed here may well be moved ... what is the expected value of the source_lang field when translating more than once (not iterating but from language to language ...)?

Is the true source language the value of the lang field from the very first advisory or just the "previous" one?

(source_lang, lang): (en, en) -> (en, fr) -> (en, de) ... (en, es)

or (I assume it can be this latter):

(source_lang, lang): (en, en) -> (en, fr) -> (fr, de) ... (pl, es)

[tschmidtb51]

Here is a list of potential "informational" tests:

• usage of CVSSv2.0 => recommend using version 3.1 (only if no cvss_v3 is given, otherwise skip)
• usage of CVSSv3.0 => recommend using version 3.1
• missing cwe => recommend adding one
• missing cve => recommend adding one
• missing distribution => recommend adding TLP statement

[tschmidtb51]

This is also related to #239:

• Error:
  • one algorithm must no be used with one filename multiple times
• Warning:
  • md5 should not be present alone => add second "strong" hash
  • sha1 should not be present alone => add second "strong" hash
  • Information:
  • hash value length < 64 for all items => possible hash collision => add second "strong" hash

[mprpic]

All the tests listed here make sense to me!

Couple more suggestions for informational tests:

• Check if all URLs resolve with a status code of <400
• Spell check certain text fields against the lang of the document