settings sync database button - create db if not exist

[filefolder]

minor things in ui/component/settings.py

        if st.button("Sync Database", help="Synchronizes the archive database with the available local seismic data based on the above parameters."):
            self.reset_is_new_cred_added()
            save_filter(self.settings)
            populate_database_from_sds(
                sds_path=self.settings.sds_path,
                db_path=self.settings.db_path,
                search_patterns=search_patterns,
                newer_than=newer_than,
                num_processes=self.settings.proccess.num_processes,
                gap_tolerance=self.settings.proccess.gap_tolerance
            )

I think someone is possibly working on this now, but good to add

1) a check if the file doesn't exist, if it doesn't, import setup_database from db.py and run to create one 2) run join_continuous_segments afterwards

[bmotevalli]

@filefolder I was not able to reproduce the db not existing issue. I added following handle to the constructor of DatabaseManager class a while ago. That should handle non-existing scenario. I did the following tests and worked fine:

• Deleted the db, then I did standard workflows and it recreated the db.
• Deleted the db, then run Sync, it run fine
• Defined a new path for db in settings, then run Sync, and it run fine.

It would be helpful to know in what scenarios it does not get created.

[filefolder]

hmm not sure why i am having trouble

1) delete database 2) go to settings / "sync database"

OperationalError: unable to open database file
Traceback:

File "/home/seis/installed/seed-vault/.venv/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
             ^^^^^^
File "/home/seis/installed/seed-vault/.venv/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
    exec(code, module.__dict__)
File "/home/seis/installed/seed-vault/seed_vault/ui/pages/10_settings.py", line 28, in <module>
    settings_page.render()
File "/home/seis/installed/seed-vault/seed_vault/ui/components/settings.py", line 167, in render
    self.render_db()
File "/home/seis/installed/seed-vault/seed_vault/ui/components/settings.py", line 118, in render_db
    populate_database_from_sds(
File "/home/seis/installed/seed-vault/seed_vault/service/seismoloader.py", line 131, in populate_database_from_sds
    db_manager = DatabaseManager(db_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/seis/installed/seed-vault/seed_vault/service/db.py", line 25, in __init__
    self.setup_database()
File "/home/seis/installed/seed-vault/seed_vault/service/db.py", line 54, in setup_database
    with self.connection() as conn:
File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
File "/home/seis/installed/seed-vault/seed_vault/service/db.py", line 33, in connection
    conn = sqlite3.connect(self.db_path, timeout=20)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

here is my current code which should be the same:

class DatabaseManager:
    def __init__(self, db_path):
        self.db_path = db_path
        parent_dir = Path(db_path).parent
        parent_dir.mkdir(parents=True, exist_ok=True)
        self.setup_database()
    @contextlib.contextmanager
    def connection(self, max_retries=3, initial_delay=1):
        """Context manager for safe database connections with retry mechanism."""
        retry_count = 0
        delay = initial_delay
        while retry_count < max_retries:
            try:
                conn = sqlite3.connect(self.db_path, timeout=20)
                yield conn
                conn.commit()
                return
            except sqlite3.OperationalError as e:
                if "database is locked" in str(e):
                    retry_count += 1
                    if retry_count >= max_retries:
                        print(f"Failed to connect to database after {max_retries} retries.")
                        raise
                    print(f"Database is locked. Retrying in {delay} seconds...")
                    time.sleep(delay)
                    delay *= 2  # Exponential backoff
                    delay += random.uniform(0, 1)  # Add jitter
                else:
                    raise
            finally:
                if 'conn' in locals():
                    conn.close()

it seems like my system isn't running __init__ again for some reason ? is there a nuclear option to restart everything from scratch?

[bmotevalli]

@filefolder from what I see in the error it looks like it is running__init__ method and actually, the error occurs in the __init__ when it calls self.setup_database().

However, I noticed something strange. The error is complaining on line 25. However, in my code, self.setup_database() is line 27! So I am not sure why there is a discrepancy. Please note, if you had the app running and while running, you have added the two below lines or have synced your code with the repository. Then, streamlit would not include the new changes, even if you refresh the page. You need to cancel and re-run the app again. I got suspecious that this could be one possible reason, as there is a two line discrepancies in what the Error reports and the latest code. Otherwise, it is hard to tell what is happening without seeing it. Maybe we can do a separate catch up on this or we can discuss it in the next meeting.

parent_dir = Path(db_path).parent
parent_dir.mkdir(parents=True, exist_ok=True)

[filefolder]

i did a full wipe and re-cloned, and now it seems to work. i must have contaminated something somewhere...

[bmotevalli]

great, then we can close this issue.