search

simonsobs / socs

Simons Observatory specific OCS agents.
BSD 2-Clause "Simplified" License
12 stars 12 forks source link

Error in smurf-sync and timestream-sync dockers #504

Open jlashner opened 1 year ago

jlashner commented 1 year ago

In both the smurf-sync and timestream-sync dockers on smrfso3 I'm seeing errors that look like this: 

2023-08-14T04:12:08+0000 Exception during reset or similar
2023-08-14T04:12:08+0000 Traceback (most recent call last):
2023-08-14T04:12:08+0000   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 739, in _finalize_fairy
2023-08-14T04:12:08+0000     fairy._reset(pool)
2023-08-14T04:12:08+0000   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 988, in _reset
2023-08-14T04:12:08+0000     pool._dialect.do_rollback(self)
2023-08-14T04:12:08+0000   File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 682, in do_rollback
2023-08-14T04:12:08+0000     dbapi_connection.rollback()
2023-08-14T04:12:08+0000 sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140261117826816 and this is thread id 140262513887040.

It seems like afterwards it continues to copy new files, but this could be related to why finalization files are not appearing.

guanyilun commented 1 year ago

FYI I've seen similar error in imprinter when there is some kind of branching that creates a new thread. I fixed it by adding some connect_args, see https://github.com/simonsobs/sotodlib/blob/master/sotodlib/site_pipeline/make_book.py#L20