search

DefectDojo / django-DefectDojo

DevSecOps, ASPM, Vulnerability Management. All on one platform.
https://defectdojo.com
BSD 3-Clause "New" or "Revised" License
3.72k stars 1.56k forks source link

500 Error on Displaying System Settings #1934

Closed kslo-resideo closed 4 years ago

kslo-resideo commented 4 years ago

Bug description I get a weird 500 error when I try to view the system settings. When I take a look at the output from django I get the error later on in this ticket under the console logs heading. It looks like it's requesting a sqlite db, but I'm using a mySQL database on another system. Not sure why it would be trying to open a local sqlite database.

Either way, I've tried to troubleshoot the issue, but I'm afraid I don't know the inner workings of the application well enough. The application is working fine otherwise. This is also a new install so there is a good chance I messed up the install and didn't start something properly.

Steps to reproduce Steps to reproduce the behavior:

  1. Click on gear icon
  2. Click on system settings

Expected behavior I would expect the page to load and not throw an internal server error.

Deployment method (select with an X)

Environment information

Sample scan files (optional) If applicable, add sample scan files to help reproduce your problem.

Screenshots (optional) If applicable, add screenshots to help explain your problem.

Console logs (optional) Internal Server Error: /system_settings Traceback (most recent call last): File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect return fn() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect return _ConnectionFairy._checkout(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 238, in _do_get return self._create_connection() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in init self.connect(first_connect_check=True) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in connect connection = pool._invoke_creator(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, *cparams) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 489, in connect return self.dbapi.connect(cargs, **cparams) sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/dojo/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors yield File "/opt/dojo/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured return fun(*args, kwargs) File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 194, in _publish [maybe_declare(entity) for entity in declare] File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 194, in [maybe_declare(entity) for entity in declare] File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 102, in maybe_declare return maybe_declare(entity, self.channel, retry, retry_policy) File "/opt/dojo/lib/python3.6/site-packages/kombu/common.py", line 121, in maybe_declare return _maybe_declare(entity, channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/common.py", line 161, in _maybe_declare entity.declare(channel=channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/entity.py", line 609, in declare self._create_queue(nowait=nowait, channel=channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/entity.py", line 618, in _create_queue self.queue_declare(nowait=nowait, passive=False, channel=channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/entity.py", line 653, in queue_declare nowait=nowait, File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 531, in queue_declare self._new_queue(queue, kwargs) File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 82, in _new_queue self._get_or_create(queue) File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 70, in _get_or_create obj = self.session.query(self.queuecls) \ File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 65, in session , Session = self._open() File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 58, in _open metadata.create_all(engine) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4316, in create_all ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2048, in _run_visitor with self._optional_conn_ctx_manager(connection) as conn: File "/usr/lib/python3.6/contextlib.py", line 81, in enter return next(self.gen) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2040, in _optional_conn_ctx_manager with self._contextual_connect() as conn: File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2242, in _contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2280, in _wrap_pool_connect e, dialect, self File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1547, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise raise value.with_traceback(tb) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect return fn() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect return _ConnectionFairy._checkout(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 238, in _do_get return self._create_connection() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in init self.connect(first_connect_check=True) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in connect connection = pool._invoke_creator(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, *cparams) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 489, in connect return self.dbapi.connect(cargs, cparams) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file (Background on this error at: http://sqlalche.me/e/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/dojo/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/opt/dojo/lib/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response response = self.process_exception_by_middleware(e, request) File "/opt/dojo/lib/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/opt/dojo/lib/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view return view_func(request, args, kwargs) File "/opt/dojo/django-DefectDojo/dojo/system_settings/views.py", line 27, in system_settings celery_bool = get_celery_worker_status() File "/opt/dojo/django-DefectDojo/dojo/utils.py", line 1849, in get_celery_worker_status res = celery_status.apply_async() File "/opt/dojo/lib/python3.6/site-packages/celery/app/task.py", line 568, in apply_async options File "/opt/dojo/lib/python3.6/site-packages/celery/app/base.py", line 771, in send_task amqp.send_task_message(P, name, message, options) File "/opt/dojo/lib/python3.6/site-packages/celery/app/amqp.py", line 559, in send_task_message properties File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish exchange_name, declare, File "/opt/dojo/lib/python3.6/site-packages/kombu/connection.py", line 551, in _ensured errback and errback(exc, 0) File "/usr/lib/python3.6/contextlib.py", line 99, in exit self.gen.throw(type, value, traceback) File "/opt/dojo/lib/python3.6/site-packages/kombu/connection.py", line 444, in _reraise_as_library_errors sys.exc_info()[2]) File "/opt/dojo/lib/python3.6/site-packages/vine/five.py", line 194, in reraise raise value.with_traceback(tb) File "/opt/dojo/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors yield File "/opt/dojo/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured return fun(args, kwargs) File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 194, in _publish [maybe_declare(entity) for entity in declare] File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 194, in [maybe_declare(entity) for entity in declare] File "/opt/dojo/lib/python3.6/site-packages/kombu/messaging.py", line 102, in maybe_declare return maybe_declare(entity, self.channel, retry, retry_policy) File "/opt/dojo/lib/python3.6/site-packages/kombu/common.py", line 121, in maybe_declare return _maybe_declare(entity, channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/common.py", line 161, in _maybe_declare entity.declare(channel=channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/entity.py", line 609, in declare self._create_queue(nowait=nowait, channel=channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/entity.py", line 618, in _create_queue self.queue_declare(nowait=nowait, passive=False, channel=channel) File "/opt/dojo/lib/python3.6/site-packages/kombu/entity.py", line 653, in queue_declare nowait=nowait, File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 531, in queue_declare self._new_queue(queue, kwargs) File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 82, in _new_queue self._get_or_create(queue) File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 70, in _get_or_create obj = self.session.query(self.queuecls) \ File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 65, in session , Session = self._open() File "/opt/dojo/lib/python3.6/site-packages/kombu/transport/sqlalchemy/init.py", line 58, in _open metadata.create_all(engine) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4316, in create_all ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2048, in _run_visitor with self._optional_conn_ctx_manager(connection) as conn: File "/usr/lib/python3.6/contextlib.py", line 81, in enter return next(self.gen) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2040, in _optional_conn_ctx_manager with self._contextual_connect() as conn: File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2242, in _contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2280, in _wrap_pool_connect e, dialect, self File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1547, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise raise value.with_traceback(tb) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect return fn() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect return _ConnectionFairy._checkout(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout rec = pool._do_get() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 238, in _do_get return self._create_connection() File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection return _ConnectionRecord(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in init self.connect(first_connect_check=True) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in connect connection = pool._invoke_creator(self) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, *cparams) File "/opt/dojo/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 489, in connect return self.dbapi.connect(cargs, **cparams) kombu.exceptions.OperationalError: (sqlite3.OperationalError) unable to open database file (Background on this error at: http://sqlalche.me/e/e3q8) [27/Feb/2020 14:28:51] "GET /system_settings HTTP/1.0" 500 27

Additional context (optional) Add any other context about the problem here.

valentijnscholten commented 4 years ago

Did you double check the settings.py file and .env.prod file?

kslo-resideo commented 4 years ago

@valentijnscholten I just looked in my settings.py file and it looks like maybe celery db is trying to use sqlite:

DD_CELERY_BROKER_URL=(str, ''),
DD_CELERY_BROKER_SCHEME=(str, 'sqla+sqlite'),
DD_CELERY_BROKER_USER=(str, ''),
DD_CELERY_BROKER_PASSWORD=(str, ''),
DD_CELERY_BROKER_HOST=(str, ''),
DD_CELERY_BROKER_PORT=(int, -1),
DD_CELERY_BROKER_PATH=(str, '/dojo.celerydb.sqlite'),
DD_CELERY_TASK_IGNORE_RESULT=(bool, True),
DD_CELERY_RESULT_BACKEND=(str, 'django-db'),
DD_CELERY_RESULT_EXPIRES=(int, 86400),
DD_CELERY_BEAT_SCHEDULE_FILENAME=(str, root('dojo.celery.beat.db')),
DD_CELERY_TASK_SERIALIZER=(str, 'pickle'),

I don't know much about celery to really understand what this is trying to do. I did a find on the root directory for a file named dojo.celerydb.sqlite and I didn't find anything (thinking that the DD_CELERY_BROKER_PATH might have weird permissions).

What should I be looking for in the .env.prod file?

valentijnscholten commented 4 years ago

The page is checking the celery service status. It does that by connecting to the broker, in this case the sqlite db. If it's absent, it will create it. Could be a permission problem? Corrupted sqllite file? Try to remove it.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

rodolphohub commented 2 years ago

Have any solution with error sqlite? @kslo-resideo

guilhermepaulozup commented 2 years ago

Is there a solution to this problem? I have almost the same problem in our deploy, but, in our case, the error is:

file "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 551, in create_connect_args
    raise exc.ArgumentError(
sqlalchemy.exc.ArgumentError: Invalid SQLite URL: sqlite://defectdojo:***@/dojo.celerydb.sqlite
Valid SQLite URL forms are:
 sqlite:///:memory: (or, sqlite://)
 sqlite:///relative/path/to/file.db
 sqlite:////absolute/path/to/file.db

Any help?

SeanPesce commented 11 months ago

I was able to work around this on DefectDojo v2.28.3 by setting the DD_CELERY_BROKER_PATH environment variable to /dojo/settings/dojo.celerydb.sqlite; I don't think it's a complete/proper fix but I can at least view/edit the settings page and submit findings now.