MIT-AI-Accelerator / c3po-model-server

Other
1 stars 1 forks source link

investigate 'is_trending' error on p1 (during train) #195

Open emiliecowen opened 2 months ago

emiliecowen commented 1 month ago

@dygwon I think this is what is causing the is_trending issue - the DB schema is stale because reinitialization is not working on P1:

running staging script INFO:main:Using initialization environment: staging INFO:main:Using migration toggle: False INFO:main:Clearing database Traceback (most recent call last): File "/home/python/python-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context self.dialect.do_execute( File "/home/python/python-packages/sqlalchemy/engine/default.py", line 921, in do_execute cursor.execute(statement, parameters) psycopg2.errors.DependentObjectsStillExist: cannot drop type originationenum because other objects depend on it DETAIL: column originated_from of table gpt4allpretrainedmodel depends on type originationenum HINT: Use DROP ... CASCADE to drop the dependent objects too. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/python/app/initial_data.py", line 559, in main() File "/home/python/app/initial_data.py", line 542, in main db = get_db(environment, migration_toggle) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/python/app/initial_data.py", line 62, in get_db wipe_db() File "/home/python/app/db/init_db.py", line 19, in wipe_db Base.metadata.drop_all(bind=engine) # noqa: F401 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/sql/schema.py", line 5850, in drop_all bind._run_ddl_visitor( File "/home/python/python-packages/sqlalchemy/engine/base.py", line 3239, in _run_ddl_visitor conn._run_ddl_visitor(visitorcallable, element, kwargs) File "/home/python/python-packages/sqlalchemy/engine/base.py", line 2443, in _run_ddl_visitor visitorcallable(self.dialect, self, kwargs).traverse_single(element) File "/home/python/python-packages/sqlalchemy/sql/visitors.py", line 670, in traverse_single return meth(obj, kw) ^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/sql/ddl.py", line 1071, in visit_metadata with self.with_ddl_events( File "/usr/local/lib/python3.11/contextlib.py", line 144, in exit next(self.gen) File "/home/python/python-packages/sqlalchemy/sql/ddl.py", line 845, in with_ddl_events target.dispatch.after_drop( File "/home/python/python-packages/sqlalchemy/event/attr.py", line 487, in call fn(*args, *kw) File "/home/python/python-packages/sqlalchemy/util/langhelpers.py", line 848, in call return getattr(self.target, self.name)(arg, kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/sql/sqltypes.py", line 1179, in _on_metadata_drop t._on_metadata_drop(target, bind, kw) File "/home/python/python-packages/sqlalchemy/dialects/postgresql/named_types.py", line 114, in _on_metadata_drop self.drop(bind=bind, checkfirst=checkfirst) File "/home/python/python-packages/sqlalchemy/dialects/postgresql/named_types.py", line 359, in drop super().drop(bind, checkfirst=checkfirst) File "/home/python/python-packages/sqlalchemy/dialects/postgresql/named_types.py", line 64, in drop bind._run_ddl_visitor(self.DDLDropper, self, checkfirst=checkfirst) File "/home/python/python-packages/sqlalchemy/engine/base.py", line 2443, in _run_ddl_visitor visitorcallable(self.dialect, self, kwargs).traverse_single(element) File "/home/python/python-packages/sqlalchemy/sql/visitors.py", line 670, in traverse_single return meth(obj, kw) ^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/dialects/postgresql/named_types.py", line 162, in visit_enum self.connection.execute(DropEnumType(enum)) File "/home/python/python-packages/sqlalchemy/engine/base.py", line 1412, in execute return meth( ^^^^^ File "/home/python/python-packages/sqlalchemy/sql/ddl.py", line 181, in _execute_on_connection return connection._execute_ddl( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/engine/base.py", line 1524, in _execute_ddl ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context return self._exec_single_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/python/python-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context self._handle_dbapi_exception( File "/home/python/python-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception raise sqlalchemy_exception.with_traceback(exc_info[2]) from e File "/home/python/python-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context self.dialect.do_execute( File "/home/python/python-packages/sqlalchemy/engine/default.py", line 921, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.InternalError: (psycopg2.errors.DependentObjectsStillExist) cannot drop type originationenum because other objects depend on it DETAIL: column originated_from of table gpt4allpretrainedmodel depends on type originationenum HINT: Use DROP ... CASCADE to drop the dependent objects too. [SQL: DROP TYPE originationenum] (Background on this error at: https://sqlalche.me/e/20/2j85) init script finished **

emiliecowen commented 3 weeks ago

https://github.com/MIT-AI-Accelerator/c3po-model-server/pull/220