IDSIA / sacred

Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.
MIT License
4.21k stars 380 forks source link

First Run Always Fails with SQLite Observer #275

Closed kwrobert closed 6 years ago

kwrobert commented 6 years ago

First off, sweet project! Been looking for a tool like this to manage my computation electromagnetics experiments for some time now.

It seems like the first run always fails when using an SQLObserver with examples/modular.py. Running this exact sequence in a clean virtual environment

git clone https://github.com/IDSIA/sacred.git
cd sacred
pip install ./
pip install sqlalchemy
cd examples
./modular.py --sql='sqlite:///test.db'

results in the following error

WARNING - modular_example - An error ocurred in the '<sacred.observers.sql.SqlObserver object at 0x7f3c6df3c828>' observer: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 139897952802560 and this is thread id 139897628923648 (Background on this error at: http://sqlalche.me/e/f405)
ERROR - modular_example - Traceback (most recent call last):
  File "/home/kyle/.virtualenvs/sacred/lib/python3.5/site-packages/sacred/run.py", line 396, in _final_call
    getattr(observer, method)(**kwargs)
  File "/home/kyle/.virtualenvs/sacred/lib/python3.5/site-packages/sacred/observers/sql.py", line 88, in completed_event
    self.save()
  File "/home/kyle/.virtualenvs/sacred/lib/python3.5/site-packages/sacred/observers/sql.py", line 113, in save
    self.session.commit()
  File "/home/kyle/.virtualenvs/sacred/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/home/kyle/.virtualenvs/sacred/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 465, in commit
    self._assert_active(prepared_ok=True)
  File "/home/kyle/.virtualenvs/sacred/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 276, in _assert_active
    % self._rollback_exception
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 139897952802560 and this is thread id 139897628923648 [SQL: 'UPDATE run SET heartbeat=?, captured_out=?, info=? WHERE run.id = ?'] [parameters: [{'info': '{}', 'captured_out': 'INFO - modular_example - Running command \'main\'\nINFO - modular_example - Started run with ID "0"\nINFO - modular_example - Completed after 0:00:00\n', 'heartbeat': datetime.datetime(2018, 4, 27, 15, 20, 58, 365700), 'run_id_1': 1}]] (Background on this error at: http://sqlalche.me/e/f405)

WARNING - modular_example - The observer '<sacred.observers.sql.SqlObserver object at 0x7f3c6df3c828>' failed at some point during the run.

The test.db file is created despite this error (bottom file):

ls -l 

total 64
-rwxrwxr-x 1 kyle kyle  1073 Apr 27 11:23 01_hello_world.py
-rwxrwxr-x 1 kyle kyle  1267 Apr 27 11:23 02_hello_config_dict.py
-rwxrwxr-x 1 kyle kyle  1739 Apr 27 11:23 03_hello_config_scope.py
-rwxrwxr-x 1 kyle kyle  1329 Apr 27 11:23 04_captured_functions.py
-rwxrwxr-x 1 kyle kyle  1997 Apr 27 11:23 05_my_commands.py
-rwxrwxr-x 1 kyle kyle  2352 Apr 27 11:23 06_randomness.py
-rwxrwxr-x 1 kyle kyle  1426 Apr 27 11:23 captured_out_filter.py
-rwxrwxr-x 1 kyle kyle  1294 Apr 27 11:23 ingredient.py
-rw-rw-r-- 1 kyle kyle   103 Apr 27 11:23 __init__.py
-rwxrwxr-x 1 kyle kyle  1064 Apr 27 11:23 log_example.py
-rwxrwxr-x 1 kyle kyle  1502 Apr 27 11:23 modular.py
drwxrwxr-x 1 kyle kyle    26 Apr 27 11:23 my_runs
-rwxrwxr-x 1 kyle kyle   485 Apr 27 11:23 named_config.py
-rw-r--r-- 1 kyle kyle 15360 Apr 27 11:25 test.db

And if I run the same command (./modular.py --sql='sqlite:///test.db') again, everything works just fine. I'm running Kubuntu 16.04, CPython version 3.5.2. I'd be happy to provide any other details you need.

Qwlouse commented 6 years ago

Thanks for catching that one. I just pushed a fix to master.