EmilyAlsentzer
commented
3 years ago Hi @yuanluo, It looks like your issue is similar to this one. Looks like it's an issue of collisions with accessing the sql database.
We haven't run into this problem before. Any chance you are pointing to NFS? Apparently Optuna doesn't work well when the database is on NFS.
One option is to switch the database from sqllite to postgres, which is supposed to work better for distributed optimization. I think this should primarily involve changing this line here to the postgres equivalent.
luoyuanlab
When running the command python train_config.py -config_path config_files/hpo_metab/metab_config.json
Got the following error, is there any configuration that needs to be set on the sqlite3 part?
Thanks!
Running 50 Trials of optuna Logging to subgnn_data/tensorboard/S_metab_optuna Traceback (most recent call last): File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context cursor, statement, parameters, context File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "train_config.py", line 283, in
main()
File "train_config.py", line 271, in main
load_if_exists=True)
File "/opt/anaconda3/lib/python3.7/site-packages/optuna/study.py", line 695, in create_study
storage = storages.get_storage(storage)
File "/opt/anaconda3/lib/python3.7/site-packages/optuna/storages/init.py", line 18, in get_storage
return _CachedStorage(RDBStorage(storage))
File "/opt/anaconda3/lib/python3.7/site-packages/optuna/storages/_rdb/storage.py", line 114, in init
models.BaseModel.metadata.create_all(self.engine)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4287, in create_all
ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2033, in _run_visitor
conn._run_visitor(visitorcallable, element, kwargs)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1607, in _run_visitor
visitorcallable(self.dialect, self, kwargs).traverse_single(element)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 131, in traverse_single
return meth(obj, kw)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 781, in visit_metadata
_is_metadata_operation=True,
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 131, in traverse_single
return meth(obj, kw)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 826, in visit_table
include_foreign_key_constraints,
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1050, in _execute_ddl
compiled,
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
raise value.with_traceback(tb)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 550, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL:
CREATE TABLE studies (
study_id INTEGER NOT NULL,
study_name VARCHAR(512) NOT NULL,
direction VARCHAR(8) NOT NULL,
PRIMARY KEY (study_id),
CONSTRAINT studydirection CHECK (direction IN ('NOT_SET', 'MINIMIZE', 'MAXIMIZE'))
)
] (Background on this error at: http://sqlalche.me/e/e3q8)