minvws / nl-kat-coordination

Repo nl-kat-coordination for minvws
European Union Public License 1.2
126 stars 57 forks source link

Invalid schema in rocky logs for various plugins #2740

Closed stephanie0x00 closed 6 months ago

stephanie0x00 commented 6 months ago

Describe the bug The rocky logs show that some plugins/normalizers have an invalid schema and/or Unknown OOI type

$ docker compose logs rocky |grep 'Invalid schema'
rocky-1  | Invalid schema found for plugin website-software, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin webpage-analysis, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin dns-sec, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin dns-zone, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin webpage-capture, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin webpage-analysis, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin pdio-subfinder, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin ssl-version, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin webpage-analysis, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin ssl-certificates, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin snyk, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin security_txt_downloader, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin rdns, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin nuclei-cve, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin nuclei-exposed-panels, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin nuclei-takeover, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin many-ports-open, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin green-hosting, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin fierce, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin fierce, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin dicom, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin CVE_2023_35078, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin CVE-2023-34039, None is not of type 'object', 'boolean'
rocky-1  | Invalid schema found for plugin adr-validator, None is not of type 'object', 'boolean'

The following `Unknown OOI types were identified. These may or may not be related to the above messages:

$ docker compose logs rocky |grep 'Unknown OOI'
rocky-1  | Unknown OOI type OOIType for normalizer produces kat_manual_csv
rocky-1  | Unknown OOI type OOIType for normalizer produces kat_manual_ooi
rocky-1  | Unknown OOI type Findings for normalizer produces kat_nuclei_cve_normalize
rocky-1  | Unknown OOI type Findings for normalizer produces kat_nuclei_exposed_panels_normalize
rocky-1  | Unknown OOI type Findings for normalizer produces kat_nuclei_takeover_normalize
rocky-1  | Unknown OOI type SNYKFindingType for normalizer produces kat_snyk_finding_types_normalize

To Reproduce Steps to reproduce the behavior:

  1. Add some hosts.
  2. Enable all plugins.
  3. Look through the rocky logs to see which have the 'Invalid schema' messages.

Expected behavior All schemas are properly verified.

OpenKAT version We thought this may have been caused by #2662 (commit 8fef333), however this is still seen with commit a629fe4.

Example of an Invalid schema found for plugin dns-sec:

rocky-1  | [29/Mar/2024 08:53:36] "GET /en/ee/kat-alogus/plugins/nmap-ports/cover.jpg HTTP/1.1" 200 37620
rocky-1  | [29/Mar/2024 08:53:36] "GET /en/ee/kat-alogus/plugins/nmap/cover.jpg HTTP/1.1" 200 40769
rocky-1  | [29/Mar/2024 08:53:36] "GET /en/ee/kat-alogus/plugins/nmap-udp/cover.jpg HTTP/1.1" 200 85036
rocky-1  | Invalid schema found for plugin dns-sec, None is not of type 'object', 'boolean'
rocky-1  | 
rocky-1  | Failed validating 'type' in metaschema['allOf'][0]:
rocky-1  |     {'$defs': {'anchorString': {'pattern': '^[A-Za-z_][-A-Za-z0-9._]*$',
rocky-1  |                                 'type': 'string'},
rocky-1  |                'uriReferenceString': {'format': 'uri-reference',
rocky-1  |                                       'type': 'string'},
rocky-1  |                'uriString': {'format': 'uri', 'type': 'string'}},
rocky-1  |      '$dynamicAnchor': 'meta',
rocky-1  |      '$id': 'https://json-schema.org/draft/2020-12/meta/core',
rocky-1  |      '$schema': 'https://json-schema.org/draft/2020-12/schema',
rocky-1  |      '$vocabulary': {'https://json-schema.org/draft/2020-12/vocab/core': True},
rocky-1  |      'properties': {'$anchor': {'$ref': '#/$defs/anchorString'},
rocky-1  |                     '$comment': {'type': 'string'},
rocky-1  |                     '$defs': {'additionalProperties': {'$dynamicRef': '#meta'},
rocky-1  |                               'type': 'object'},
rocky-1  |                     '$dynamicAnchor': {'$ref': '#/$defs/anchorString'},
rocky-1  |                     '$dynamicRef': {'$ref': '#/$defs/uriReferenceString'},
rocky-1  |                     '$id': {'$comment': 'Non-empty fragments not allowed.',
rocky-1  |                             '$ref': '#/$defs/uriReferenceString',
rocky-1  |                             'pattern': '^[^#]*#?$'},
rocky-1  |                     '$ref': {'$ref': '#/$defs/uriReferenceString'},
rocky-1  |                     '$schema': {'$ref': '#/$defs/uriString'},
rocky-1  |                     '$vocabulary': {'additionalProperties': {'type': 'boolean'},
rocky-1  |                                     'propertyNames': {'$ref': '#/$defs/uriString'},
rocky-1  |                                     'type': 'object'}},
rocky-1  |      'title': 'Core vocabulary meta-schema',
rocky-1  |      'type': ['object', 'boolean']}
rocky-1  | 
rocky-1  | On schema:
rocky-1  |     None
rocky-1  | [29/Mar/2024 08:53:37] "POST /en/ee/kat-alogus/plugins/boefje/dns-sec/False/ HTTP/1.1" 302 0
rocky-1  | [29/Mar/2024 08:53:37] "POST /en/ee/kat-alogus/plugins/boefje/dns-records/False/ HTTP/1.1" 302 0
rocky-1  | Unknown OOI type OOIType for normalizer produces kat_manual_csv
rocky-1  | Unknown OOI type OOIType for normalizer produces kat_manual_ooi
rocky-1  | Unknown OOI type Findings for normalizer produces kat_nuclei_cve_normalize
rocky-1  | Unknown OOI type Findings for normalizer produces kat_nuclei_exposed_panels_normalize
rocky-1  | Unknown OOI type Findings for normalizer produces kat_nuclei_takeover_normalize
rocky-1  | Unknown OOI type SNYKFindingType for normalizer produces kat_snyk_finding_types_normalize
rocky-1  | [29/Mar/2024 08:53:37] "GET /en/ee/kat-alogus/ HTTP/1.1" 200 157199
rocky-1  | [29/Mar/2024 08:53:37] "GET /static/dist/app.css HTTP/1.1" 304 0
underdarknl commented 6 months ago

I think the schema errors are present because we expect each plugin to have a schema, which they do not.