Chronic: microSALT attempts to write to a read only database

[barrystokman]

Cron hiseq.clinical@hasta $CRONIC ${PRODUCTION_HOME}/servers/resources/hasta.scilifelab.se/crontabs/microsalt-start.sh >> ${LOG_BASE}/microsalt.analysis.log 2> >(tee -a ${LOG_BASE}/microsalt.analysis.log >&2)

Cronic detected failure for the command: /home/proj/production/servers/resources/hasta.scilifelab.se/crontabs/microsalt-start.sh

touch: cannot touch ‘/home/proj/production/microbial/meta/microsalt.db’: Permission denied ERROR - Reference update function failed prematurely. Review immediately INFO - pubMLST reference for Staphylococcus aureus updated to 2020-07-13 from 2020-07-10 INFO - Re-indexed contents of /home/proj/production/microbial/references/ST_loci/staphylococcus_aureus Traceback (most recent call last): File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context cursor, statement, parameters, context File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: attempt to write a readonly database

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/proj/bin/conda/envs/P_microSALT/bin/microSALT", line 8, in sys.exit(root()) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(args, *kwargs) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), args, **kwargs) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/microSALT/cli.py", line 396, in autobatch ext_refs.update_refs() File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/microSALT/utils/referencer.py", line 82, in update_refs self.fetch_pubmlst(self.force) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/microSALT/utils/referencer.py", line 484, in fetch_pubmlst {"version": external_ver}, File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/microSALT/store/db_manipulator.py", line 160, in upd_rec eval(megastring + ".update(upd_dict)") File "", line 1, in File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 4009, in update updateop.exec() File "/home/proj/bin/conda/envs/PmicroSALT/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1697, in exec self._do_exec() File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1893, in _do_exec self._execute_stmt(update_stmt) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1702, in _execute_stmt self.result = self.query._execute_crud(stmt, self.mapper) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3514, in _execute_crud return conn.execute(stmt, self._params) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute return meth(self, multiparams, params) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement distilled_params, File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context e, statement, parameters, cursor, context File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception sqlalchemy_exception, with_traceback=excinfo[2], from=e File "/home/proj/bin/conda/envs/PmicroSALT/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise raise exception File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context cursor, statement, parameters, context File "/home/proj/bin/conda/envs/P_microSALT/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [SQL: UPDATE versions SET version=? WHERE versions.name = ?] [parameters: ('2020-07-13', 'profile_staphylococcus_aureus')] (Background on this error at: http://sqlalche.me/e/e3q8)

RESULT CODE: 1

[barrystokman]

@sylvinite Chronic detected a failure for starting microsalt analyses, I noticed the mention of microsalt.db in the message.

[sylvinite]

We've been seeing issues due to a readonly database multiple times now. Nothing has changed in the code itself, so either the hardware or the usage is to blame.

[barrystokman]

I'm not sure how to solve this, maybe @eriksjolund can help out?

[eriksjolund]

Regarding

touch: cannot touch ‘/home/proj/production/microbial/meta/microsalt.db’: Permission denied

I don't have access to that part of the filesystem. (I have a pending support request for getting extra permissions)

-bash-4.2$ ls /home/proj/production/microbial
ls: cannot open directory /home/proj/production/microbial: Permission denied
-bash-4.2$ 

It would for instance be interesting to verify that the directory /home/proj/production/microbial/meta/ exists.

Does the error message

touch: cannot touch ‘/home/proj/production/microbial/meta/microsalt.db’: Permission denied

originate from https://github.com/Clinical-Genomics/microSALT/blob/77884b19c375df5b8a13cc3fc19141c0c2a6c109/microSALT/__init__.py#L121?

If touch fails, it looks like the program just continues.

If it is a sanity check, I guess it would be good to add

if proc.returncode != 0
     somehow act up on the error

after the line

https://github.com/Clinical-Genomics/microSALT/blob/77884b19c375df5b8a13cc3fc19141c0c2a6c109/microSALT/__init__.py#L125

or maybe use https://docs.python.org/3/library/pathlib.html#pathlib.Path.touch

[sylvinite]

Added @eriksjolund solution to next branch to be released, which will clarify error.

[sylvinite]

Lock manually released. Added code to check for possible locks in the future.