lanl / BEE

Other
13 stars 3 forks source link

Provide Clear Error for DB Schema Changes #758

Open rstyd opened 6 months ago

rstyd commented 6 months ago

When the DB schema changes, our DB drivers error out with a very unclear error message. We should handle any case where that happens and either provide a clearer message that it needs to be reset or ask the user if they'd like to delete the old one and generate a new one, or create a backup of the old one and generate a new one.

rstyd commented 6 months ago

This is the kind of error we'll normally see when encountering this bug:

    workflow_list = db.workflows.get_workflows()
  File "/BEE/beeflow/common/db/wfm_db.py", line 69, in get_workflows
    workflows = [self.Workflow(*workflow) for workflow in result]
  File "/BEE/beeflow/common/db/wfm_db.py", line 69, in <listcomp>
    workflows = [self.Workflow(*workflow) for workflow in result]
TypeError: Workflow.__new__() missing 2 required positional arguments: 'https_port' and 'gdb_pid'

The issue is because the named tuple Workflow within the *_db.py files requires the exact number of arguments. So if the number of arguments doesn't match it's either an error from the caller or the schema has changed. We could potentially verify that by querying the SQLite db and finding how many elements it has although that could still miss something if there's a change that keeps the same number of elements.

jtronge commented 6 months ago

I wonder if there's a way to make updates from one version of a database schema to the next. I know they do something like this for production SQL databases, perhaps using some sort of versioning scheme and then a list of transformations from each version to the next. This might be more complicated, but I wonder if we might want to keep old database info in newer versions of BEE.

rstyd commented 6 months ago

Hmm. SQLite does support alter table so in theory we could figure something out, but I think this might be a good motivation from pivoting to SQlite to a more robust implementation like PostgreSQL.