bmarsh9 / gapps

Security compliance platform - SOC2, CMMC, ASVS, ISO27001, HIPAA, NIST CSF, NIST 800-53, CSC CIS 18, PCI DSS, SSF tracking. https://gapps.darkbanner.com
Other
437 stars 99 forks source link

Internal Server Error: Service not usable with version 3.4.0 upgrade #62

Closed manuel-sommer closed 1 year ago

manuel-sommer commented 1 year ago

@bmarsh9 , the following commit hash was the last working commit: ee65f9c89b8878e7f0ab946a1aa52a2b79d6d5fd

The upgrade to 3.4.0 introduced an Internal Server Error in this commit: https://github.com/bmarsh9/gapps/commit/e642ad4a8c9311e7d8a52ecc9654ab533594a6db

Gapps is not usable anymore in the recommended way to use the Service:

  1. docker-compose up
  2. try to login to the app
  3. Internal Server Error
bmarsh9 commented 1 year ago

@manuel-sommer It’s likely a database migration issue. Could you run: “docker logs gapps” and paste the output of the logs (last 100 lines is fine)

manuel-sommer commented 1 year ago

[INFO] Checking if we can connect to the database server: postgresql://db1:db1@postgres/db1 [INFO] Successfully connected to the database server [INFO] Checking if we should create the models [INFO] Successfully queried the database models [INFO] Starting the server with 2 workers [2023-05-31 22:12:50 +0000] [23] [INFO] Starting gunicorn 20.1.0 [2023-05-31 22:12:50 +0000] [23] [INFO] Listening at: http://0.0.0.0:5000 (23) [2023-05-31 22:12:50 +0000] [23] [INFO] Using worker: sync [2023-05-31 22:12:50 +0000] [25] [INFO] Booting worker with pid: 25 [2023-05-31 22:12:50 +0000] [26] [INFO] Booting worker with pid: 26 [2023-05-31 22:12:59,843] ERROR in app: Exception on /controls [GET] Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UndefinedColumn: column users.can_user_create_tenant does not exist LINE 1: ...e AS users_last_name, users.super AS userssuper, users.can... ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1519, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) File "/app/app/utils/decorators.py", line 208, in decorator if not current_user.is_authenticated: File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 278, in get obj = instance._get_current_object() File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 407, in _get_current_object return self.local() # type: ignore File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 26, in current_user = LocalProxy(lambda: _get_user()) File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 384, in _get_user current_app.login_manager._load_user() File "/usr/local/lib/python3.8/dist-packages/flask_login/login_manager.py", line 355, in _load_user user = self._user_callback(user_id) File "/app/app/models.py", line 1660, in load_user return User.query.get(int(user_id)) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 1018, in get return self._get_impl(ident, loading.load_on_pk_identity) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 1135, in _get_impl return db_load_fn(self, primary_key_identity) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity return q.one() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3490, in one ret = self.one_or_none() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none ret = list(self) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3535, in iter__ return self._execute_and_instances(context) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context self._handle_dbapi_exception( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapiexception util.raise( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column users.can_user_create_tenant does not exist LINE 1: ...e AS users_last_name, users.super AS userssuper, users.can... ^

[SQL: SELECT users.id AS users_id, users.is_active AS users_is_active, users.email AS users_email, users.username AS users_username, users.email_confirmed_at AS users_email_confirmed_at, users.password AS users_password, users.last_password_change AS users_last_password_change, users.first_name AS users_first_name, users.last_name AS users_last_name, users.super AS users_super, users.can_user_create_tenant AS users_can_user_create_tenant, users.date_added AS users_date_added, users.date_updated AS users_date_updated FROM users WHERE users.id = %(param_1)s] [parameters: {'param_1': 1}] (Background on this error at: http://sqlalche.me/e/13/f405) [2023-05-31 22:12:59 +0000] [25] [ERROR] Error handling request /controls Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UndefinedColumn: column users.can_user_create_tenant does not exist LINE 1: ...e AS users_last_name, users.super AS userssuper, users.can... ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1519, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request rv = self.dispatch_request() 172.18.0.1 - - [31/May/2023:22:12:59 +0000] "GET /controls HTTP/1.1" 500 0 "-" "-" File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) File "/app/app/utils/decorators.py", line 208, in decorator if not current_user.is_authenticated: File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 278, in get obj = instance._get_current_object() File "/usr/local/lib/python3.8/dist-packages/werkzeug/local.py", line 407, in _get_current_object return self.local() # type: ignore File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 26, in current_user = LocalProxy(lambda: _get_user()) File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 384, in _get_user current_app.login_manager._load_user() File "/usr/local/lib/python3.8/dist-packages/flask_login/login_manager.py", line 355, in _load_user user = self._user_callback(user_id) File "/app/app/models.py", line 1660, in load_user return User.query.get(int(user_id)) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 1018, in get return self._get_impl(ident, loading.load_on_pk_identity) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 1135, in _get_impl return db_load_fn(self, primary_key_identity) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity return q.one() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3490, in one ret = self.one_or_none() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none ret = list(self) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3535, in iter__ return self._execute_and_instances(context) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context self._handle_dbapi_exception( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapiexception util.raise( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column users.can_user_create_tenant does not exist LINE 1: ...e AS users_last_name, users.super AS userssuper, users.can... ^

[SQL: SELECT users.id AS users_id, users.is_active AS users_is_active, users.email AS users_email, users.username AS users_username, users.email_confirmed_at AS users_email_confirmed_at, users.password AS users_password, users.last_password_change AS users_last_password_change, users.first_name AS users_first_name, users.last_name AS users_last_name, users.super AS users_super, users.can_user_create_tenant AS users_can_user_create_tenant, users.date_added AS users_date_added, users.date_updated AS users_date_updated FROM users WHERE users.id = %(param_1)s] [parameters: {'param_1': 1}] (Background on this error at: http://sqlalche.me/e/13/f405)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/sync.py", line 136, in handle self.handle_request(listener, req, client, addr) File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/sync.py", line 179, in handle_request respiter = self.wsgi(environ, resp.start_response) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2091, in call return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2076, in wsgi_app response = self.handle_exception(e) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1432, in handle_exception server_error = self.ensure_sync(handler)(server_error) File "/app/app/init.py", line 57, in internal_error return render_template("layouts/errors/default.html", title="Internal error"), e.code File "/usr/local/lib/python3.8/dist-packages/flask/templating.py", line 153, in render_template ctx.app.update_template_context(context) File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 750, in update_template_context context.update(func()) File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 417, in _user_context_processor return dict(current_user=_get_user()) File "/usr/local/lib/python3.8/dist-packages/flask_login/utils.py", line 384, in _get_user current_app.login_manager._load_user() File "/usr/local/lib/python3.8/dist-packages/flask_login/login_manager.py", line 355, in _load_user user = self._user_callback(user_id) File "/app/app/models.py", line 1660, in load_user return User.query.get(int(user_id)) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 1018, in get return self._get_impl(ident, loading.load_on_pk_identity) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 1135, in _get_impl return db_load_fn(self, primary_key_identity) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity return q.one() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3490, in one ret = self.one_or_none() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none ret = list(self) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3535, in iter return self._execute_and_instances(context) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context self._handle_dbapi_exception( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapiexception util.raise( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.InternalError: (psycopg2.errors.InFailedSqlTransaction) current transaction is aborted, commands ignored until end of transaction block

[SQL: SELECT users.id AS users_id, users.is_active AS users_is_active, users.email AS users_email, users.username AS users_username, users.email_confirmed_at AS users_email_confirmed_at, users.password AS users_password, users.last_password_change AS users_last_password_change, users.first_name AS users_first_name, users.last_name AS users_last_name, users.super AS users_super, users.can_user_create_tenant AS users_can_user_create_tenant, users.date_added AS users_date_added, users.date_updated AS users_date_updated FROM users WHERE users.id = %(param_1)s] [parameters: {'param_1': 1}] (Background on this error at: http://sqlalche.me/e/13/2j85) [2023-05-31 22:13:30 +0000] [23] [CRITICAL] WORKER TIMEOUT (pid:26) [2023-05-31 22:13:30 +0000] [26] [INFO] Worker exiting (pid: 26) [2023-05-31 22:13:31 +0000] [53] [INFO] Booting worker with pid: 53 [2023-05-31 22:23:57 +0000] [23] [CRITICAL] WORKER TIMEOUT (pid:25) [2023-05-31 22:23:57 +0000] [23] [CRITICAL] WORKER TIMEOUT (pid:53) [2023-05-31 22:23:57 +0000] [25] [INFO] Worker exiting (pid: 25) [2023-05-31 22:23:57 +0000] [53] [INFO] Worker exiting (pid: 53) [2023-05-31 22:23:58 +0000] [67] [INFO] Booting worker with pid: 67 [2023-05-31 22:23:58 +0000] [23] [WARNING] Worker with pid 25 was terminated due to signal 9 [2023-05-31 22:23:58 +0000] [68] [INFO] Booting worker with pid: 68

bmarsh9 commented 1 year ago

@manuel-sommer Here is the documentation to perform the migration - that should fix it

https://github.com/bmarsh9/gapps#perform-database-migration