Nikto Parser: Support new JSON report format

[moxli]

Scanner Name Nikto 2.5.0 has introduced breaking changes to the JSON output format.

https://github.com/sullo/nikto/wiki/2.5.0-Potentially-Breaking-Changes

Sample File

[
  {
    "host": "example.com",
    "ip": "203.0.113.37",
    "port": "443",
    "banner": "",
    "vulnerabilities": [
      {
        "id": "999986",
        "method": "GET",
        "url": "/",
        "msg": "Retrieved via header: 1.1 google."
      },
      {
        "id": "999978",
        "references": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options",
        "method": "GET",
        "url": "/",
        "msg": "/:X-Frame-Options header is deprecated and has been replaced with the Content-Security-Policy HTTP header with the frame-ancestors directive instead."
      },
      {
        "id": "999103",
        "references": "https://www.netsparker.com/web-vulnerability-scanner/vulnerabilities/missing-content-type-header/",
        "method": "GET",
        "url": "/robots.txt",
        "msg": "The X-Content-Type-Options header is not set. This could allow the user agent to render the content of the site in a different fashion to the MIME type."
      },
      {
        "id": "999996",
        "references": "https://developer.mozilla.org/en-US/docs/Glossary/Robots.txt",
        "method": "GET",
        "url": "/robots.txt",
        "msg": "contains 1 entry which should be manually viewed."
      },
      {
        "id": "999962",
        "method": "GET",
        "url": "/images/8f7c0a6.ico",
        "msg": ": Server banner changed from 'nginx' to 'ghs'."
      },
      {
        "id": "999966",
        "references": "http://breachattack.com/",
        "method": "GET",
        "url": "/",
        "msg": "The Content-Encoding header is set to \\\"deflate\\\" which may mean that the server is vulnerable to the BREACH attack."
      },
      {
        "id": "999100",
        "method": "GET",
        "url": "/",
        "msg": "Uncommon header 'x-google-gfe-load-report' found, with contents: utilization_percent: 51.177505 queries_per_second: 20382 errors_per_second: 0."
      },
      {
        "id": "999100",
        "method": "GET",
        "url": "/",
        "msg": "Uncommon header 'x-google-gfe-backend-request-cost' found, with contents: 51.177506531784765."
      }
    ]
  }
]

[manuel-sommer]

@moxli I made a PR for this issue. See linked PR.

[moxli]

@manuel-sommer awesome thank you!

One thing I noticed is that this change might cause some issues?

The "osvdb" field in all databases has been renamed to "references"

Apparently the medium severity is assigned automatically if an entry in ovsdb exists: https://github.com/DefectDojo/django-DefectDojo/blob/ccd711a8ffc838e820ce4b7302840080adceff87/dojo/tools/nikto/parser.py#L73

This might not be feasible anymore since the references field can be anything.

[manuel-sommer]

The references fields in your example do not reflect the severity. But you are right, I can add the references field in finding.reference. I will push a fix today.

[manuel-sommer]

@manuel-sommer awesome thank you!

One thing I noticed is that this change might cause some issues?

The "osvdb" field in all databases has been renamed to "references"

Apparently the medium severity is assigned automatically if an entry in ovsdb exists:

https://github.com/DefectDojo/django-DefectDojo/blob/ccd711a8ffc838e820ce4b7302840080adceff87/dojo/tools/nikto/parser.py#L73

This might not be feasible anymore since the references field can be anything.

done, could you take a look?

[moxli]

@manuel-sommer hm I think this would break backwards compatibility since the old json format does not have the references field.

https://github.com/DefectDojo/django-DefectDojo/pull/9275/files#diff-65a24f137064f6e8a8f290785ba98515073314a119512be9bd96d451475370eaR73

I think we should check if the field exists before accessing it.

In my opinion backwards compatibility shall be maintained because Nikto 2.1.5 is still widely available. Example: https://packages.ubuntu.com/search?keywords=nikto

[manuel-sommer]

I know that backwards compatibility is crucial. I guess this does not break backwards compatibility and also the unittests did not fail.

[moxli]

@manuel-sommer Ah nice perfect :)

I think I miss interpreted the message I got when I tried to import the new format:

[04/Jan/2024 10:03:46] ERROR [dojo.api_v2.exception_handler:36] 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create
    self.perform_create(serializer)
  File "/app/dojo/api_v2/views.py", line 3238, in perform_create
    serializer.save(push_to_jira=push_to_jira)
  File "/app/dojo/api_v2/serializers.py", line 2502, in save
    ) = reimporter.reimport_scan(
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/importers/reimporter/reimporter.py", line 609, in reimport_scan
    parsed_findings = parser.get_findings(scan, test)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/nikto/parser.py", line 40, in get_findings
    return self.process_json(filename, test)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/dojo/tools/nikto/parser.py", line 48, in process_json
    host = data.get("host")
           ^^^^^^^^

This was due to the fact that the entire thing is an array now and not because it could not find the "host" field?

I thought, that the parser fails if a field like references can not be found.

[manuel-sommer]

This can be closed @mtesauro