Chimp: Server error on file upload

[jluethi]

We currently get an error when uploading files to Chimp. Some files are uploaded, but pretty soon, we run into this error: something related to sqlalchemy and some id he can't find @HackerMD Any idea what causes that? And how we can solve it?

2017-04-24 08:33:55 | ERROR | wsgi | Exception on /api/experiments/dG1hcHMyMg==/acquisitions/dG1hcHMx/microscope-file [POST] Traceback (most recent call last): File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functionsrule.endpoint File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask_jwt/init.py", line 176, in decorator _jwt_required(realm or current_app.config['JWT_DEFAULT_REALM']) File "/home/tissuemaps/.local/lib/python2.7/site-packages/flask_jwt/init.py", line 162, in _jwt_required _request_ctx_stack.top.current_identity = identity = _jwt.identity_callback(payload) File "/home/tissuemaps/tmserver/tmserver/extensions/auth.py", line 48, in load_user user = current_session.query(tm.User).get(payload['uid']) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 864, in get return self._get_impl(ident, loading.load_on_ident) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 897, in _get_impl return fallback_fn(self, key) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 223, in load_on_ident return q.one() File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2814, in one ret = self.one_or_none() File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none ret = list(self) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in iter return self._execute_and_instances(context) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1414, in _handle_dbapi_exception self.engine.pool._invalidate(dbapi_conn_wrapper, e) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 351, in _invalidate connection.invalidate(exception) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 917, in invalidate self._checkin() File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 811, in _checkin self._pool, None, self._echo, fairy=self) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 703, in _finalize_fairy connection_record.checkin() File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 545, in checkin pool.dispatch.checkin(connection, self) File "/home/tissuemaps/.local/lib/python2.7/site-packages/sqlalchemy/event/attr.py", line 218, in call fn(*args, **kw) File "/home/tissuemaps/tmlibrary/tmlib/models/utils.py", line 209, in _on_pool_checkin dbapi_con.get_backend_pid() AttributeError: 'NoneType' object has no attribute 'get_backend_pid'

[jluethi]

Problem comes up again after server restart => restarting the server and thus resetting the backend pool doesn't solve it.

Could it have something to do with multiple users currently using Chimp? Because Reinoud, Scott and I are currently using it in parallel...

[jluethi]

@riccardomurri This is the database error I referred to in today's meeting. The problem seems to be that some data base connections get closed and it still tries to use them. Markus logged in to (I think) the data base workers or master process and looked at the logs there to trace it down.

[hackermd]

It seems the database is failing with some queries and somehow doesn't close the connections properly. You should be able to find the error in the log on one of the database servers. Restarting all database servers solved the problem last time.

[scottberry]

See #72 for more info

[scottberry]

Resolving this should be top priority in establishing a functional multi-user TissueMAPS deployment

[hackermd]

Resolving this should be top priority in establishing a functional multi-user TissueMAPS deployment

I don't think this is a general multi-tenancy problem. From the messages that I found in the database logs, it seems that specific queries are failing and the affected connections are not reset/renewed in the pool, but get reused by subsequent queries.

We should figure out:

1. Why queries fail (this seems to be experiment-specific and only happens in some jterator pipelines).
2. Why the corresponding connections are not properly reset. We may need to specify additional parameters upon pool creation, see sqlalchemy.pool.Pool

[hackermd]

SQLAlchemy provides a mechanism to test a connection before checking it out from the pool. See dealing with disconnects section of the docs. Using pool_pre_ping might help in this case.