kiblik
commented
9 months ago Can you please double-check your DefectDojo version? Because it hasn't been released in version 1.6.91.
Closed John-Aow closed 9 months ago
kiblik
commented
9 months ago Can you please double-check your DefectDojo version? Because it hasn't been released in version 1.6.91.
bitnesswise
commented
9 months ago I'm using version 2.29.1 and I'm getting this error when using the /api/v2/import-scan/ endpoint.
[11/Dec/2023 07:52:56] ERROR [dojo.api_v2.exception_handler:36] column dojo_notifications.scan_added_empty does not exist
LINE 1: ...."test_added", "dojo_notifications"."scan_added", "dojo_noti...
^
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedColumn: column dojo_notifications.scan_added_empty does not exist
LINE 1: ...."test_added", "dojo_notifications"."scan_added", "dojo_noti...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create
self.perform_create(serializer)
File "/app/dojo/api_v2/views.py", line 3485, in perform_create
serializer.save(push_to_jira=push_to_jira)
File "/app/dojo/api_v2/serializers.py", line 2502, in save
) = reimporter.reimport_scan(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/reimporter/reimporter.py", line 750, in reimport_scan
notifications_helper.notify_scan_added(
File "/app/dojo/notifications/helper.py", line 399, in notify_scan_added
create_notification(event=event, title=title, findings_new=new_findings, findings_mitigated=findings_mitigated, findings_reactivated=findings_reactivated,
File "/app/dojo/notifications/helper.py", line 103, in create_notification
for user in users:
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 394, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1869, in _fetch_all
self._prefetch_related_objects()
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1258, in _prefetch_related_objects
prefetch_related_objects(self._result_cache, *self._prefetch_related_lookups)
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 2298, in prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 2440, in prefetch_one_level
) = prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 716, in get_prefetch_queryset
for rel_obj in queryset:
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 394, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1867, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 87, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.ProgrammingError: column dojo_notifications.scan_added_empty does not exist
LINE 1: ...."test_added", "dojo_notifications"."scan_added", "dojo_noti...
^
[11/Dec/2023 07:52:56] ERROR [django.request:241] Internal Server Error: /api/v2/reimport-scan/This seems to happen on both the import-scan and reimport-scan endpoints
kiblik
commented
9 months ago Looks like the migration of your database has not been performed during the latest upgrade.
I see, you use k8s and helm. Can you check, when was the last time, your initializer was running? It should be triggered during each upgrade.
If it failed, why (logs)?
If it wasn't trigged, can you check, how you set run?. If it is true, you might need to set also trackConfig to enabled to trigger initializer for each upgrade.
lamej02
commented
9 months ago How to check and run the initializer by hand if the trackConfig is disabled?
dsever
commented
9 months ago trackConfig ensures for every change in configuration your pods will be restarted. Bit I don't think this is the case here, logs are indicating you have new version of container but old version of DB (as @kiblik explained init process was skipped for some reason). And one reason could be you have explicitly define to skip it.
https://github.com/DefectDojo/django-DefectDojo/blob/master/helm/defectdojo/values.yaml#L302
I would say you are missing migrations, so can you go into one of the django containers and list migrations:
https://docs.djangoproject.com/en/5.0/ref/django-admin/#showmigrations
If you will identify some of the missing migrations, then trigger migration
lamej02
commented
9 months ago initializer: run: true
lamej02
commented
9 months ago showmigrations in the pod is missing a value for DJANGO_SETTINGS_MODULE ?
The log:
django-admin showmigrations --list
Traceback (most recent call last):
File "/usr/local/bin/django-admin", line 8, in
kiblik
commented
9 months ago Can you try to run /app/manage.py instead of django-admin?
lamej02
commented
9 months ago The log using /app/manage.py:
/app/manage.py showmigrations --list
[13/Dec/2023 09:49:09] INFO [dojo.models:4295] enabling audit logging
/usr/local/lib/python3.11/site-packages/coreapi/codecs/download.py:5: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
import cgi
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/template/backends/django.py", line 128, in get_package_libraries
module = import_module(entry[1])
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.11/site-packages/rest_framework/templatetags/rest_framework.py", line 12, in <module>
from rest_framework.renderers import HTMLFormRenderer
File "/usr/local/lib/python3.11/site-packages/rest_framework/renderers.py", line 21, in <module>
from rest_framework import VERSION, exceptions, serializers, status
File "/usr/local/lib/python3.11/site-packages/rest_framework/serializers.py", line 29, in <module>
from rest_framework.fields import get_error_detail, set_value
File "/usr/local/lib/python3.11/site-packages/rest_framework/fields.py", line 34, in <module>
from rest_framework.settings import api_settings
File "/usr/local/lib/python3.11/site-packages/rest_framework/settings.py", line 22, in <module>
from django.test.signals import setting_changed
File "/usr/local/lib/python3.11/site-packages/django/test/__init__.py", line 3, in <module>
from django.test.client import AsyncClient, AsyncRequestFactory, Client, RequestFactory
File "/usr/local/lib/python3.11/site-packages/django/test/client.py", line 23, in <module>
from django.test.utils import ContextList
File "/usr/local/lib/python3.11/site-packages/django/test/utils.py", line 14, in <module>
from unittest import TestCase, skipIf, skipUnless
File "/app/dojo/settings/unittest.py", line 5, in <module>
from .settings import *
ImportError: attempted relative import with no known parent package
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 443, in execute
self.check()
File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 475, in check
all_issues = checks.run_checks(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/checks/registry.py", line 88, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/checks/templates.py", line 60, in check_for_template_tags_with_the_same_name
for module_name, module_path in get_template_tag_modules():
File "/usr/local/lib/python3.11/site-packages/django/template/backends/django.py", line 105, in get_template_tag_modules
for name in get_package_libraries(pkg):
File "/usr/local/lib/python3.11/site-packages/django/template/backends/django.py", line 130, in get_package_libraries
raise InvalidTemplateLibrary(
django.template.library.InvalidTemplateLibrary: Invalid template library specified. ImportError raised when trying to load 'rest_framework.templatetags.rest_framework': attempted relative import with no known parent package
Weird. And cd /app; ./manage.py showmigrations --list?
lamej02
commented
9 months ago ./manage.py showmigrations --list [13/Dec/2023 10:34:17] INFO [dojo.models:4295] enabling audit logging /usr/local/lib/python3.11/site-packages/coreapi/codecs/download.py:5: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13 import cgi admin [X] 0001_initial [X] 0002_logentry_remove_auto_add [X] 0003_logentry_add_action_flag_choices auditlog [X] 0001_initial [X] 0002_auto_support_long_primary_keys [X] 0003_logentry_remote_addr [X] 0004_logentry_detailed_object_repr [X] 0005_logentry_additional_data_verbose_name [X] 0006_object_pk_index [X] 0007_object_pk_type [X] 0008_action_index [X] 0009_alter_logentry_additional_data [X] 0010_alter_logentry_timestamp [X] 0011_logentry_serialized_data [X] 0012_add_logentry_action_access auth [X] 0001_initial [X] 0002_alter_permission_name_max_length [X] 0003_alter_user_email_max_length [X] 0004_alter_user_username_opts [X] 0005_alter_user_last_login_null [X] 0006_require_contenttypes_0002 [X] 0007_alter_validators_add_error_messages [X] 0008_alter_user_username_max_length [X] 0009_alter_user_last_name_max_length [X] 0010_alter_group_name_max_length [X] 0011_update_proxy_permissions [X] 0012_alter_user_first_name_max_length authtoken [X] 0001_initial [X] 0002_auto_20160226_1747 [X] 0003_tokenproxy contenttypes [X] 0001_initial [X] 0002_remove_content_type_name django_celery_results [X] 0001_initial [X] 0002_add_task_name_args_kwargs [X] 0003_auto_20181106_1101 [X] 0004_auto_20190516_0412 [X] 0005_taskresult_worker [X] 0006_taskresult_date_created [X] 0007_remove_taskresult_hidden [X] 0008_chordcounter [X] 0009_groupresult [X] 0010_remove_duplicate_indices [X] 0011_taskresult_periodic_task_name dojo [X] 0001_squashed_0090_index_duplicate_finding (90 squashed migrations) [X] 0091_npm_audit_path_censoring [X] 0092_is_mitigated [X] 0093_django_tagging_removal [X] 0094_remove_system_settings_s_finding_severity_naming [X] 0095_remove_old_product_contact_fields [X] 0096_grype_name_change [X] 0097_engagement_type [X] 0098_anchore_vuln_id [X] 0099_delete_report [X] 0100_dojo_user_for_authv2 [X] 0101_enable_features [X] 0102_dojo_group [X] 0103_report_notification [X] 0104_endpoint_userinfo_creation [X] 0105_endpoint_host_migration [X] 0106_role_model [X] 0107_global_role [X] 0108_blank_fields [X] 0109_group_user_role [X] 0110_auth_v2_migrate_user_roles [X] 0111_group_user_rename [X] 0112_group_user_rename_2 [X] 0113_endpoint_protocol [X] 0114_cyclonedx_vuln_uniqu [X] 0115_language_types [X] 0116_test_type_active [X] 0117_usercontactinfo_force_password_reset [X] 0118_remove_finding_images [X] 0119_default_group_is_staff [X] 0120_sonarqube_test_and_clean [X] 0121_user_restrict [X] 0122_cobaltio_product [X] 0123_scan_type [X] 0124_sonarqube_api_type_length_change [X] 0125_sonarqube_clean [X] 0126_finding_publish_date [X] 0127_remove_hashes [X] 0128_pytz_update [X] 0129_finding_deprecated_fields [X] 0130_product_api_scan_configuration [X] 0131_migrate_sonarcube_cobalt [X] 0132_remove_configs_from_test [X] 0133_finding_service [X] 0134_sonarque_cobaltio_removal [X] 0135_email_from [X] 0136_default_group_help_text [X] 0137_system_settings_enable_endpoint_metadata_import [X] 0138_remove_authorized_users [X] 0139_google_sheets_rules_framework_enable [X] 0140_auth_group [X] 0141_enable_user_profile_editable [X] 0142_environment_delete [X] 0143_objects [X] 0144_import_action_untouched [X] 0145_system_settings_default_group_email_pattern [X] 0146_lead_optional [X] 0147_rename_sslyze_parser [X] 0148_default_notifications [X] 0149_harmonize_user_format [X] 0150_dedupe_endpoint_status [X] 0151_index_endpoint_status [X] 0152_notifications_template [X] 0153_migrate_endpoint_mitigated [X] 0154_remove_endpoint_mitigated [X] 0155_enable_finding_groups [X] 0156_migrate_finding_groups_setting [X] 0157_vulnerability_reference [X] 0158_vulnerability_id [X] 0159_remove_broken_endpoint_statuses [X] 0160_set_notnull_endpoint_statuses [X] 0161_alter_dojo_group_social_provider [X] 0162_created_and_updated [X] 0163_system_settings_enable_calendar [X] 0164_remove_system_settings_staff_user_email_pattern [X] 0165_custom_sla [X] 0166_copy_sla_from_system_settings [X] 0167_system_settings_add_vulnerability_id_to_jira_label [X] 0168_alter_system_settings_time_zone [X] 0169_planned_remediation_date [X] 0170_jira_project_custom_fields [X] 0171_jira_labels_per_product_and_engagement [X] 0172_optimize_usage_of_endpoint_status [X] 0173_alter_risk_acceptance_name [X] 0174_jira_project_default_assignee [X] 0175_system_settings_enable_notify_sla [X] 0176_custom_password_requirements [X] 0177_alter_system_settings_time_zone [X] 0178_alter_answer_polymorphic_ctype_and_more [X] 0179_alter_finding_verified [X] 0180_announcement_userannouncement [X] 0181_jira_instance_finding_jira_sync [X] 0182_alter_jira_instance_default_issue_type [X] 0183_system_settings_enable_notify_sla_exponential_backoff_and_more [X] 0184_remove_child_rule_parent_rule_delete_fieldrule_and_more [X] 0185_product_disable_sla_breach_notifications_and_more [X] 0186_system_settings_non_common_password_required [X] 0187_nessus_to_tenable [X] 0188_product_enable_product_tag_inheritance_and_more [X] 0189_finding_effort_and_remediation_for_fixing [X] 0190_system_settings_experimental_fp_history [ ] 0191_alter_notifications_risk_acceptance_expiration [ ] 0192_notifications_scan_added_empty sessions [X] 0001_initial sites [X] 0001_initial [X] 0002_alter_domain_unique social_django [X] 0001_initial (2 squashed migrations) [X] 0002_add_related_name (2 squashed migrations) [X] 0003_alter_email_max_length (2 squashed migrations) [X] 0004_auto_20160423_0400 (2 squashed migrations) [X] 0005_auto_20160727_2333 (1 squashed migrations) [X] 0006_partial [X] 0007_code_timestamp [X] 0008_partial_timestamp [X] 0009_auto_20191118_0520 [X] 0010_uid_db_index [X] 0011_alter_id_fields [ ] 0012_usersocialauth_extra_data_new [ ] 0013_migrate_extra_data [ ] 0014_remove_usersocialauth_extra_data [ ] 0015_rename_extra_data_new_usersocialauth_extra_data tagging [X] 0001_initial [X] 0002_on_delete [X] 0003_adapt_max_tag_length watson [X] 0001_initial [X] 0002_alter_searchentry_object_id
kiblik
commented
9 months ago 6 migrations are missing. You can run ./manage.py migrate to perform them.
It will fix the situation for now. But I highly recommend investigating, why initializer wasn't started again during upgrade.
lamej02
commented
9 months ago Thanks for the support!
So initializer should run when "run: true" is configured in the value yaml file. And should also TrackConfig be enabled?
Running migrations: Applying dojo.0191_alter_notifications_risk_acceptance_expiration... OK Applying dojo.0192_notifications_scan_added_empty... OK Applying social_django.0012_usersocialauth_extra_data_new... OK Applying social_django.0013_migrate_extra_data... OK Applying social_django.0014_remove_usersocialauth_extra_data... OK Applying social_django.0015_rename_extra_data_new_usersocialauth_extra_data... OK
kiblik
commented
9 months ago I would enable it but as @dsever mentioned, this should not be the reason. I'm not able to provide you with better support.
lamej02
commented
9 months ago Thanks for all the support! The environment is working again.
Bug description Open product detail page then got 500 internal server page . another page seem normal
Steps to reproduce Steps to reproduce the behavior:
Expected behavior see product detail.
Deployment method (select with an
X)Environment information
Logs File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 56, in inner response = get_response(request) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, *callback_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/dojo/authorization/authorization_decorators.py", line 35, in _wrapped return func(request, args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/dojo/product/views.py", line 156, in view_product instance=Notifications.objects.filter(user=request.user).filter(product=prod).first()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1047, in first for obj in (self if self.ordered else self.order_by("pk"))[:1]: File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 394, in iter self._fetch_all() File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1867, in _fetch_all self._result_cache = list(self._iterable_class(self)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 87, in iter results = compiler.execute_sql( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1398, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute with self.db.wrap_database_errors: File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in exit raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ django.db.utils.ProgrammingError: column dojo_notifications.scan_added_empty does not exist LINE 1: ...."test_added", "dojo_notifications"."scan_added", "dojo_noti...