danswer background container fail to start after upgrade

[moshe]

I've upgraded danswer from v0.3.25 to v0.3.42 and started getting:

DETAIL:  Failing row contains (44424, 2024-03-07 07:51:04.71755+00, 2024-03-07 07:51:04.71755+00, NOT_STARTED, null, 2, 1, 0, null, 0, 2, null, null, null).

[SQL: INSERT INTO index_attempt (connector_id, credential_id, status, new_docs_indexed, total_docs_indexed, error_msg, embedding_model_id, time_started) VALUES (%(connector_id)s, %(credential_id)s, %(status)s, %(new_docs_indexed)s, %(total_docs_indexed)s, %(error_msg)s, %(embedding_model_id)s, %(time_started)s) RETURNING index_attempt.id, index_attempt.time_created, index_attempt.time_updated]
[parameters: {'connector_id': 2, 'credential_id': 1, 'status': 'NOT_STARTED', 'new_docs_indexed': 0, 'total_docs_indexed': 0, 'error_msg': None, 'embedding_model_id': 2, 'time_started': None}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "from_beginning" of relation "index_attempt" violates not-null constraint
DETAIL:  Failing row contains (44424, 2024-03-07 07:51:04.71755+00, 2024-03-07 07:51:04.71755+00, NOT_STARTED, null, 2, 1, 0, null, 0, 2, null, null, null).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/danswer/background/update.py", line 432, in update_loop
    create_indexing_jobs(existing_jobs=existing_jobs)
  File "/app/danswer/background/update.py", line 184, in create_indexing_jobs
    create_index_attempt(
  File "/app/danswer/db/index_attempt.py", line 46, in create_index_attempt
    db_session.commit()
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1906, in commit
    trans.commit(_to_root=True)
  File "<string>", line 2, in commit
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1221, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1196, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4154, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4290, in _flush
    with util.safe_reraise():
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4251, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 467, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1223, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1413, in execute
    return meth(
           ^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1987, in _exec_single_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "from_beginning" of relation "index_attempt" violates not-null constraint
DETAIL:  Failing row contains (44424, 2024-03-07 07:51:04.71755+00, 2024-03-07 07:51:04.71755+00, NOT_STARTED, null, 2, 1, 0, null, 0, 2, null, null, null).

For now, I've fixed it by running alter table public.index_attempt alter column from_beginning set default false;, but I guess something in the code needs to be changed

[brianjking]

Did you figure out the issue @moshe ?

[moshe]

No, I added default value to the column and waiting for an official fix

[moshe]

Found the issue, forgot to upgrade background container