Async finding import bug

[kir-b]

Bug description Hi! Some reports are imported with an error if the parser parses the dates from the report. This issue occurs only if async finding import is enabled (DD_ASYNC_FINDING_IMPORT=true) and the report parser supports date parsing. I have tested with nuclei and detect_secret reports.

Error message in UI:

An exception error occurred during the report import:['“2021-05-19T10:40:18Z” value has an invalid date format. It must be in YYYY-MM-DD format.']: (dojo.finding:pk=2308) field_value was '2021-05-19T10:40:18Z'

Exceptions in logs:

defectdojo-uwsgi-1         | [31/Jul/2023 14:38:30] ERROR [dojo.engagement.views:703] ['“2021-05-19T10:40:18Z” value has an invalid date format. It must be in YYYY-MM-DD format.']: (dojo.finding:pk=2304) field_value was '2021-05-19T10:40:18Z'
defectdojo-uwsgi-1         | Traceback (most recent call last):
defectdojo-uwsgi-1         |   File "/usr/local/lib/python3.11/site-packages/django/core/serializers/python.py", line 167, in Deserializer
defectdojo-uwsgi-1         |     data[field.name] = field.to_python(field_value)
defectdojo-uwsgi-1         |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
defectdojo-uwsgi-1         |   File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/__init__.py", line 1399, in to_python
defectdojo-uwsgi-1         |     raise exceptions.ValidationError(
defectdojo-uwsgi-1         | django.core.exceptions.ValidationError: ['“2021-05-19T10:40:18Z” value has an invalid date format. It must be in YYYY-MM-DD format.']
defectdojo-uwsgi-1         |
defectdojo-uwsgi-1         | During handling of the above exception, another exception occurred:
defectdojo-uwsgi-1         |
defectdojo-uwsgi-1         | Traceback (most recent call last):
defectdojo-uwsgi-1         |   File "/app/dojo/engagement/views.py", line 687, in import_scan_results
defectdojo-uwsgi-1         |     test, finding_count, closed_finding_count, _ = importer.import_scan(scan, scan_type, engagement, user, environment, active=active, verified=verified, tags=tags,
defectdojo-uwsgi-1         |                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
defectdojo-uwsgi-1         |   File "/app/dojo/importers/importer/importer.py", line 333, in import_scan
defectdojo-uwsgi-1         |     new_findings += [next(serializers.deserialize("json", finding)).object for finding in serial_new_findings]
defectdojo-uwsgi-1         |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
defectdojo-uwsgi-1         |   File "/app/dojo/importers/importer/importer.py", line 333, in <listcomp>
defectdojo-uwsgi-1         |     new_findings += [next(serializers.deserialize("json", finding)).object for finding in serial_new_findings]
defectdojo-uwsgi-1         |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
defectdojo-uwsgi-1         |   File "/usr/local/lib/python3.11/site-packages/tagulous/serializers/base.py", line 51, in wrapper
defectdojo-uwsgi-1         |     for obj in obj_generator:
defectdojo-uwsgi-1         |   File "/usr/local/lib/python3.11/site-packages/django/core/serializers/json.py", line 70, in Deserializer
defectdojo-uwsgi-1         |     yield from PythonDeserializer(objects, **options)
defectdojo-uwsgi-1         |   File "/usr/local/lib/python3.11/site-packages/django/core/serializers/python.py", line 169, in Deserializer
defectdojo-uwsgi-1         |     raise base.DeserializationError.WithData(
defectdojo-uwsgi-1         | django.core.serializers.base.DeserializationError: ['“2021-05-19T10:40:18Z” value has an invalid date format. It must be in YYYY-MM-DD format.']: (dojo.finding:pk=2304) field_value was '2021-05-19T10:40:18Z'

However, in the end the report is imported. If DD_ASYNC_FINDING_IMPORT is 'false' no error occurs.

Steps to reproduce Steps to reproduce the behavior:

1. Set DD_ASYNC_FINDING_IMPORT to 'true'
2. Start up dojo
3. Try to import detect_secrets or nuclei report, for example https://github.com/DefectDojo/django-DefectDojo/blob/master/unittests/scans/detect_secrets/many_findings.json
4. See error

Expected behavior Successful import without errors.

Deployment method (select with an X)

• [X] Docker Compose
• [ ] Kubernetes
• [ ] GoDojo

Environment information

• Operating System: Ubuntu 20.04
• DefectDojo version v. 2.24.3

[mtesauro]

I could be wrong here but I'm pretty sure the async finding import was an experimental feature and should really only be used if you're having performance issues that cannot be addressed by any other means - consider turning it on as a hail mary.

If you're running on insufficient resources turning on that feature won't really make a difference.

Are you getting the same or similar errors when async is off?

[kir-b]

Yes, I know that asynchronous imports are experimental. But I've been using this feature for over a year and had no problems before. However, this feature helps us solve problems with importing reports with a large number of findings. I don't know how we can import such reports in another way. As I wrote this problem occurs only if asynchronous import is used. I found this exception source here, but I'm not familiar enough with django and dojo to offer a fix.