Reimport error 500 with ASYNC_IMPORT enabled

[C0mebreathe]

Bug description When ASYNC_IMPORT is enabled some of reimports constantly fail with 500 error. Seems like a bug for me, cause if I switch to ASYNC_IMPORT=False everything is ok. scan_type = Generic Findings Import.

Expected behavior Reimports success with 201 status code with ASYNC_IMPORT enabled.

Deployment method (select with an X)

• [x ] Docker Compose

Environment information

• Operating System: debian 12
• DefectDojo version 2.25.4

Dockerfile.django-debian ...

• ENV \

  DD_CELERY_WORKER_POOL_TYPE="solo" \

  DD_CELERY_WORKER_POOL_TYPE="prefork" \ DD_CELERY_WORKER_AUTOSCALE_MIN="24" \ DD_CELERY_WORKER_AUTOSCALE_MAX="48" \ DD_CELERY_WORKER_CONCURRENCY="24" \ DD_CELERY_WORKER_PREFETCH_MULTIPLIER="256" \ ... DD_UWSGI_NUM_OF_PROCESSES="4" \ DD_UWSGI_NUM_OF_THREADS="4" ...

Logs [12/Sep/2023 10:59:03] ERROR [dojo.api_v2.exception_handler:36] list index out of range Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch response = handler(request, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create self.perform_create(serializer) File "/app/dojo/api_v2/views.py", line 3471, in perform_create serializer.save(push_to_jira=push_to_jira) File "/app/dojo/api_v2/serializers.py", line 2502, in save ) = reimporter.reimport_scan( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/dojo/importers/reimporter/reimporter.py", line 654, in reimport_scan ) = results.get() ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 247, in get return self.backend.wait_for_pending( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/celery/backends/base.py", line 763, in wait_for_pending return result.maybe_throw(propagate=propagate, callback=callback) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 361, in maybe_throw self.throw(value, self._to_remote_traceback(tb)) File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 354, in throw self.on_ready.throw(args, **kwargs) File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 234, in throw reraise(type(exc), exc, tb) File "/usr/local/lib/python3.11/site-packages/vine/utils.py", line 30, in reraise raise value IndexError: list index out of range [12/Sep/2023 10:59:03] ERROR [django.request:241] Internal Server Error: /api/v2/reimport-scan/

[1azunna]

Experienced the same issue as well

[08/Jun/2023 15:59:17] ERROR [dojo.api_v2.exception_handler:32] list index out of range
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create
    self.perform_create(serializer)
  File "/app/dojo/api_v2/views.py", line 2447, in perform_create
    serializer.save(push_to_jira=push_to_jira)
  File "/app/dojo/api_v2/serializers.py", line 1846, in save
    reimporter.reimport_scan(scan, scan_type, test, active=active, verified=verified,
  File "/app/dojo/importers/reimporter/reimporter.py", line 654, in reimport_scan
    ) = results.get()
        ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 224, in get
    return self.backend.wait_for_pending(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/backends/base.py", line 764, in wait_for_pending
    return result.maybe_throw(propagate=propagate, callback=callback)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 336, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 329, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 234, in throw
    reraise(type(exc), exc, tb)
  File "/usr/local/lib/python3.11/site-packages/vine/utils.py", line 30, in reraise
    raise value
IndexError: list index out of range
[08/Jun/2023 15:59:17] ERROR [django.request:241] Internal Server Error: /api/v2/reimport-scan/

Here’s my request Payload:

{'scan_type': 'Generic Findings Import', 'product_name': 'Kubernetes Cluster', 'test_title': 'Dynamic Analysis', 'engagement': 3, 'engagement_name': 'DAST Engagement', 'push_to_jira': False, 'active': True, 'verified': True, 'close_old_findings': True, 'minimum_severity': 'Info', 'tags': ['dojo-ci'], 'scan_date': '2023-06-08', 'branch_tag': ''}

From the payload, i'm using the Generic Findings Import Parser to upload custom scan results to defectdojo.

[C0mebreathe]

Another side effect of async reimports with generic parser - emerging of endpoints that should be migrated

celeryworker_1 | [15/Sep/2023 06:38:13] ERROR [celery.app.trace:270] Task dojo.importers.utils.add_endpoints_to_unsaved_finding[152c578e-b815-41f8-b0a6-33532a5bb244] raised unexpected: Exception('Endpoints in your database are broken. Please access /endpoint/migrate and migrate them to new format or remove them.')

[mtesauro]

ASYNC_IMPORT should be considered an experimental feature. It works in some cases and doesn't in others. I believe it was written with 'traditional' imports in mind, not re-imports though it's been long enough that I don't trust my memory on the specifics.

Someone should probably add something to settings.py that says "Use at your own risk" TBH.

[C0mebreathe]

ASYNC_IMPORT should be considered an experimental feature. It works in some cases and doesn't in others. I believe it was written with 'traditional' imports in mind, not re-imports though it's been long enough that I don't trust my memory on the specifics.

Someone should probably add something to settings.py that says "Use at your own risk" TBH.

Thank you for the explanation, Matt. I think it should be mentioned here as well: https://defectdojo.github.io/django-DefectDojo/getting_started/running-in-production/

[ragdked]

We have the same issue by uploading Nessus Scan

Endpoints in your database are broken. Please access /endpoint/migrate and migrate them to new format or remove them. uwsgi_1         | Traceback (most recent call last): uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch uwsgi_1         |     response = handler(request, *args, *kwargs) uwsgi_1         |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create uwsgi_1         |     self.perform_create(serializer) uwsgi_1         |   File "/app/dojo/api_v2/views.py", line 3266, in perform_create uwsgi_1         |     serializer.save(push_to_jira=push_to_jira) uwsgi_1         |   File "/app/dojo/api_v2/serializers.py", line 2219, in save uwsgi_1         |     ) = importer.import_scan( uwsgi_1         |         ^^^^^^^^^^^^^^^^^^^^^ uwsgi_1         |   File "/app/dojo/importers/importer/importer.py", line 338, in import_scan uwsgi_1         |     serial_new_findings = results.get() uwsgi_1         |                           ^^^^^^^^^^^^^ uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 250, in get uwsgi_1         |     return self.backend.wait_for_pending( uwsgi_1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/celery/backends/base.py", line 763, in wait_for_pending uwsgi_1         |     return result.maybe_throw(propagate=propagate, callback=callback) uwsgi_1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 364, in maybe_throw uwsgi_1         |     self.throw(value, self._to_remote_traceback(tb)) uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 357, in throw uwsgi_1         |     self.on_ready.throw(args, **kwargs) uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 234, in throw uwsgi_1         |     reraise(type(exc), exc, tb) uwsgi_1         |   File "/usr/local/lib/python3.11/site-packages/vine/utils.py", line 30, in reraise uwsgi_1         |     raise value uwsgi_1         | Exception: Endpoints in your database are broken. Please access /endpoint/migrate and migrate them to new format or remove them.

But changing settings.dist.py to DD_ASYNC_FINDING_IMPORT=(bool, False) not solved this issue.

Any ideas how it can be fixed?

[C0mebreathe]

We have the same issue by uploading Nessus Scan

Endpoints in your database are broken. Please access /endpoint/migrate and migrate them to new format or remove them. uwsgi_1 | Traceback (most recent call last): uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch uwsgi_1 | response = handler(request, *args, *kwargs) uwsgi_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create uwsgi_1 | self.perform_create(serializer) uwsgi_1 | File "/app/dojo/api_v2/views.py", line 3266, in perform_create uwsgi_1 | serializer.save(push_to_jira=push_to_jira) uwsgi_1 | File "/app/dojo/api_v2/serializers.py", line 2219, in save uwsgi_1 | ) = importer.import_scan( uwsgi_1 | ^^^^^^^^^^^^^^^^^^^^^ uwsgi_1 | File "/app/dojo/importers/importer/importer.py", line 338, in import_scan uwsgi_1 | serial_new_findings = results.get() uwsgi_1 | ^^^^^^^^^^^^^ uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 250, in get uwsgi_1 | return self.backend.wait_for_pending( uwsgi_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/celery/backends/base.py", line 763, in wait_for_pending uwsgi_1 | return result.maybe_throw(propagate=propagate, callback=callback) uwsgi_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 364, in maybe_throw uwsgi_1 | self.throw(value, self._to_remote_traceback(tb)) uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/celery/result.py", line 357, in throw uwsgi_1 | self.on_ready.throw(args, **kwargs) uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/vine/promises.py", line 234, in throw uwsgi_1 | reraise(type(exc), exc, tb) uwsgi_1 | File "/usr/local/lib/python3.11/site-packages/vine/utils.py", line 30, in reraise uwsgi_1 | raise value uwsgi_1 | Exception: Endpoints in your database are broken. Please access /endpoint/migrate and migrate them to new format or remove them.

But changing settings.dist.py to DD_ASYNC_FINDING_IMPORT=(bool, False) not solved this issue.

Any ideas how it can be fixed?

Have you already tried to migrate endpoints? It's available from dojo GUI

[ragdked]

Oh my god! Really, now it's working fine! Thank you very much.

[manuel-sommer]

I guess this can be closed @mtesauro