Http 500 while import semgrep results

[drJabber]

Bug description There is an error while import semgrep result using API or UI import function Stripped semgrep output attached semgrep2.json.txt

Steps to reproduce Steps to reproduce the behavior:

1. Take attached json and import in some engagement using Semgrep importer
2. See error "Well... ....this was unexpected"

Expected behavior Semgrep output should be uploaded correctly

Deployment method (select with an X)

• [X ] Docker Compose

Environment information

• Operating System: Debian 11
• DefectDojo version (see footer) or commit message: 2.22.2

Logs

django-defectdojo-uwsgi-1  | [27/Jul/2023 16:44:16] INFO [django.request:241] OK: /ddojo/engagement/60/import_scan_results
django-defectdojo-uwsgi-1  | Traceback (most recent call last):
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 56, in inner
django-defectdojo-uwsgi-1  |     response = get_response(request)
django-defectdojo-uwsgi-1  |                ^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 138, in __call__
django-defectdojo-uwsgi-1  |     response = self.process_response(request, response)
django-defectdojo-uwsgi-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/watson/middleware.py", line 37, in process_response
django-defectdojo-uwsgi-1  |     self._close_search_context(request)
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/watson/middleware.py", line 33, in _close_search_context
django-defectdojo-uwsgi-1  |     search_context_manager.end()
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 261, in end
django-defectdojo-uwsgi-1  |     list(chain.from_iterable(engine._update_obj_index_iter(obj)
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 505, in _update_obj_index_iter
django-defectdojo-uwsgi-1  |     "content": adapter.get_content(obj),
django-defectdojo-uwsgi-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 147, in get_content
django-defectdojo-uwsgi-1  |     return self.prepare_content(" ".join(
django-defectdojo-uwsgi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 101, in prepare_content
django-defectdojo-uwsgi-1  |     content = strip_tags(content)
django-defectdojo-uwsgi-1  |               ^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 246, in wrapper
django-defectdojo-uwsgi-1  |     return func(*args, **kwargs)
django-defectdojo-uwsgi-1  |            ^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/html.py", line 175, in strip_tags
django-defectdojo-uwsgi-1  |     new_value = _strip_once(value)
django-defectdojo-uwsgi-1  |                 ^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/html.py", line 163, in _strip_once
django-defectdojo-uwsgi-1  |     s.feed(value)
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/html/parser.py", line 110, in feed
django-defectdojo-uwsgi-1  |     self.goahead(0)
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/html/parser.py", line 178, in goahead
django-defectdojo-uwsgi-1  |     k = self.parse_html_declaration(i)
django-defectdojo-uwsgi-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/html/parser.py", line 263, in parse_html_declaration
django-defectdojo-uwsgi-1  |     return self.parse_marked_section(i)
django-defectdojo-uwsgi-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django-defectdojo-uwsgi-1  |   File "/usr/local/lib/python3.11/_markupbase.py", line 154, in parse_marked_section
django-defectdojo-uwsgi-1  |     raise AssertionError(
django-defectdojo-uwsgi-1  | AssertionError: unknown status keyword 'A-Z' in marked section

Sample scan files attached

Screenshots nope

Additional context (optional) I think this error is from bad html-tag-like content in field "lines" of json file This content comes from code snippet, which semgrep extracted from defective code

[quine]

We're seeing the same when importing SARIF results (since it flows down into the same culprit code) on DefectDojo v2.25.2:

[01/Sep/2023 05:00:08] INFO [django.request:241] OK: /api/v2/import-scan/
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/deprecation.py", line 138, in __call__
    response = self.process_response(request, response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/watson/middleware.py", line 37, in process_response
    self._close_search_context(request)
  File "/usr/local/lib/python3.11/site-packages/watson/middleware.py", line 33, in _close_search_context
    search_context_manager.end()
  File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 261, in end
    list(chain.from_iterable(engine._update_obj_index_iter(obj)
  File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 505, in _update_obj_index_iter
    "content": adapter.get_content(obj),
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 147, in get_content
    return self.prepare_content(" ".join(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/watson/search.py", line 101, in prepare_content
    content = strip_tags(content)
              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/functional.py", line 246, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/html.py", line 175, in strip_tags
    new_value = _strip_once(value)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/html.py", line 163, in _strip_once
    s.feed(value)
  File "/usr/local/lib/python3.11/html/parser.py", line 110, in feed
    self.goahead(0)
  File "/usr/local/lib/python3.11/html/parser.py", line 178, in goahead
    k = self.parse_html_declaration(i)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/html/parser.py", line 263, in parse_html_declaration
    return self.parse_marked_section(i)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/_markupbase.py", line 154, in parse_marked_section
    raise AssertionError(
AssertionError: unknown status keyword 'A-Z' in marked section

[quine]

In our case, the offending SARIF file had content similar to the following:

              "contextRegion": {
                "startLine": 1,
                "endLine": 1,
                "snippet": {
                  "text": "<![>"

If the <![> value was removed from results.locations[].contextRegion.text, the SARIF file was processed correctly.

[manuel-sommer]

This issue is still present in v. 2.31.0-dev

[manuel-sommer]

https://bugs.python.org/issue32876

[manuel-sommer]

This issue can be closed @mtesauro