Open ezio-melotti opened 4 years ago
ezio-melotti
commented
4 years ago The next day a new user registered, tried to start a new game with similar results (celery-worker_1 returning earlier and celery-worker_2 giving errors), played a bit with the graphs in the config, and left.
ezio-melotti
commented
4 years ago I managed to accidentally reproduce this locally. Yesterday I ran a 10-humans sim on the websocket branch to completion, then left everything there for about ~30 hours. Today I picked up the browser again (tab still open), used the back button to go back to the config wizard, and started a 1-human sim and got the error below. Further attempts results in more celery.exceptions.TimeoutError: The operation timed out. errors from the flask-app_1 container. Even if I log out and log in again I get the same errors when I try to start a new sim.
This is the error from celery-worker_1:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
conn = self._revalidate_connection()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "/simoc/celery_worker/tasks.py", line 64, in new_game
user = get_user(username)
File "/simoc/celery_worker/tasks.py", line 34, in get_user
user = User.query.filter_by(username=username).all()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3211, in all
return list(self)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
e, util.text_type(statement), parameters, None, None
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
conn = self._revalidate_connection()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
[SQL: SELECT user.id AS user_id, user.username AS user_username, user.password_hash AS user_password_hash, user.date_created AS user_date_created, user.da
FROM user
WHERE user.username = %(username_1)s]
[parameters: [immutabledict({})]]And this is the error from flask-app_1 (after a 60s timeout):
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 54, in drain_events_until
raise socket.timeout()
socket.timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/dist-packages/flask_login/utils.py", line 261, in decorated_view
return func(*args, **kwargs)
File "/simoc/simoc_server/views.py", line 196, in new_game
result = tasks.new_game.delay(get_standard_user_obj().username, game_config).get(timeout=60)
File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 226, in get
on_message=on_message,
File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 259, in _wait_for_pending
raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.
[2019-12-17 02:23:57 +0000] [9] [ERROR] ERROR: handling exception The operation timed out.
[2019-12-17 02:24:52,997: INFO/ForkPoolWorker-2] Cleaning up save game for user with id 1The containers all seem up and running since yesterday:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
875c1c3698f3 simoc_celery-worker "/bin/bash start_wor…" 30 hours ago Up 30 hours simoc_celery-worker_1
c85a33855b29 simoc_flask-app "/bin/bash run.sh" 30 hours ago Up 30 hours 0.0.0.0:8000->8000/tcp simoc_flask-app_1
d4c7e4488c61 simoc_celery-worker "/bin/bash start_wor…" 30 hours ago Up 30 hours simoc_celery-worker_2
fbd94d9c72a3 mysql/mysql-server "/entrypoint.sh mysq…" 30 hours ago Up 30 hours (healthy) 0.0.0.0:3306->3306/tcp, 33060/tcp simoc_simoc-db_1
3609e29fc08c redis "docker-entrypoint.s…" 30 hours ago Up 30 hours 0.0.0.0:6379->6379/tcp simoc_redis_1
iurii-milovanov
commented
4 years ago Should be fixed by this https://github.com/kstaats/simoc/commit/4a7bd9b8eca8466deaa239c1d2e0e65833a9f219. Please confirm.
ezio-melotti
commented
4 years ago I got the same error today, by leaving the browser open for ~9 hours on the config wizard screen and then clicking on "Finalize settings" to start a new game. It seems to me that after leaving it there for a while, MySQL disconnects, and this causes OperationalError: MySQL Connection not available.. This failure leaves an uncommited transaction that needs to be rolled back before being able to continue.
The first line of the log is the last operation before leaving the browser, then there's the request to /new_game that caused the OperationalError (followed by a timeout error after 1 minute), and a second attempt to start a new sim results in the InvalidRequestError.
flask-app_1 | 2019-12-19T08:21:14.175899110Z [2019-12-19 08:21:14 +0000] [9] [DEBUG] Closing connection.
flask-app_1 | 2019-12-19T17:16:25.665428630Z [2019-12-19 17:16:25 +0000] [9] [DEBUG] POST /new_game
celery-worker_1 | 2019-12-19T17:16:26.005001868Z [2019-12-19 17:16:26,004: INFO/MainProcess] Received task: celery_worker.tasks.new_game[088b18c5-0c82-48ae-a6f6-0e6e595bc8a6]
celery-worker_1 | 2019-12-19T17:16:26.005233839Z [2019-12-19 17:16:26,004: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f1781449c80> (args:('celery_worker.tasks.new_game', '088b18c5-0c82-48ae-a6f6-0e6e595bc8a6', {'lang': 'py', 'task': 'celery_worker.tasks.new_game', 'id': '088b18c5-0c82-48ae-a6f6-0e6e595bc8a6', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '088b18c5-0c82-48ae-a6f6-0e6e595bc8a6', 'parent_id': None, 'argsrepr': "('eziom', {'agents': {'human_agent': [...], 'solid_waste_aerobic_bioreactor': [...], 'multifiltration_purifier_post_treament': [...], 'oxygen_generation_SFWE': [...], 'urine_recycling_processor_VCD': [...], 'co2_removal_SAWD': [...], 'co2_reduction_sabatier': [...], 'crew_habitat_small': [...], 'solar_pv_array_mars': [...]}, 'storages': {'air_storage': [...], 'water_storage': [...], 'nutrient_storage': [...], 'power_storage': [...], 'food_storage': [...]}, 'termination': [{...}], 'single_agent': 1})", 'kwargsrepr': '{}', 'origin': 'gen9@c85a33855b29', 'reply_to': '339f0422-8bfe-3808-b4e4-ae30c8a7322b', 'correlation_id':... kwargs:{})
celery-worker_1 | 2019-12-19T17:16:26.008052248Z [2019-12-19 17:16:26,007: DEBUG/MainProcess] Task accepted: celery_worker.tasks.new_game[088b18c5-0c82-48ae-a6f6-0e6e595bc8a6] pid:18
celery-worker_1 | 2019-12-19T17:16:26.121131284Z [2019-12-19 17:16:26,098: ERROR/ForkPoolWorker-2] Task celery_worker.tasks.new_game[088b18c5-0c82-48ae-a6f6-0e6e595bc8a6] raised unexpected: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.',)
celery-worker_1 | 2019-12-19T17:16:26.121194594Z Traceback (most recent call last):
celery-worker_1 | 2019-12-19T17:16:26.121211491Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
celery-worker_1 | 2019-12-19T17:16:26.121227751Z context = constructor(dialect, self, conn, *args)
celery-worker_1 | 2019-12-19T17:16:26.121241766Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 768, in _init_compiled
celery-worker_1 | 2019-12-19T17:16:26.121257162Z self.cursor = self.create_cursor()
celery-worker_1 | 2019-12-19T17:16:26.121270632Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 1122, in create_cursor
celery-worker_1 | 2019-12-19T17:16:26.121285412Z return self._dbapi_connection.cursor()
celery-worker_1 | 2019-12-19T17:16:26.121302146Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 965, in cursor
celery-worker_1 | 2019-12-19T17:16:26.121320344Z return self.connection.cursor(*args, **kwargs)
celery-worker_1 | 2019-12-19T17:16:26.121338597Z File "/usr/local/lib/python3.6/dist-packages/mysql/connector/connection_cext.py", line 524, in cursor
celery-worker_1 | 2019-12-19T17:16:26.121356893Z raise errors.OperationalError("MySQL Connection not available.")
celery-worker_1 | 2019-12-19T17:16:26.121371083Z mysql.connector.errors.OperationalError: MySQL Connection not available.
celery-worker_1 | 2019-12-19T17:16:26.121384433Z
celery-worker_1 | 2019-12-19T17:16:26.121398098Z The above exception was the direct cause of the following exception:
celery-worker_1 | 2019-12-19T17:16:26.121412142Z
celery-worker_1 | 2019-12-19T17:16:26.121425566Z Traceback (most recent call last):
celery-worker_1 | 2019-12-19T17:16:26.121442342Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task
celery-worker_1 | 2019-12-19T17:16:26.121484114Z R = retval = fun(*args, **kwargs)
celery-worker_1 | 2019-12-19T17:16:26.121500900Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 648, in __protected_call__
celery-worker_1 | 2019-12-19T17:16:26.121518662Z return self.run(*args, **kwargs)
celery-worker_1 | 2019-12-19T17:16:26.121535432Z File "/simoc/celery_worker/tasks.py", line 64, in new_game
celery-worker_1 | 2019-12-19T17:16:26.121551855Z user = get_user(username)
celery-worker_1 | 2019-12-19T17:16:26.121569232Z File "/simoc/celery_worker/tasks.py", line 34, in get_user
celery-worker_1 | 2019-12-19T17:16:26.121585772Z user = User.query.filter_by(username=username).all()
celery-worker_1 | 2019-12-19T17:16:26.121603748Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3211, in all
celery-worker_1 | 2019-12-19T17:16:26.121620087Z return list(self)
celery-worker_1 | 2019-12-19T17:16:26.121634941Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
celery-worker_1 | 2019-12-19T17:16:26.121651034Z return self._execute_and_instances(context)
celery-worker_1 | 2019-12-19T17:16:26.121670559Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
celery-worker_1 | 2019-12-19T17:16:26.121687732Z result = conn.execute(querycontext.statement, self._params)
celery-worker_1 | 2019-12-19T17:16:26.121703074Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
celery-worker_1 | 2019-12-19T17:16:26.121719912Z return meth(self, multiparams, params)
celery-worker_1 | 2019-12-19T17:16:26.121735740Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
celery-worker_1 | 2019-12-19T17:16:26.121752089Z return connection._execute_clauseelement(self, multiparams, params)
celery-worker_1 | 2019-12-19T17:16:26.121768481Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
celery-worker_1 | 2019-12-19T17:16:26.121785524Z distilled_params,
celery-worker_1 | 2019-12-19T17:16:26.121800296Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
celery-worker_1 | 2019-12-19T17:16:26.121816928Z e, util.text_type(statement), parameters, None, None
celery-worker_1 | 2019-12-19T17:16:26.121832404Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
celery-worker_1 | 2019-12-19T17:16:26.121847883Z util.raise_from_cause(sqlalchemy_exception, exc_info)
celery-worker_1 | 2019-12-19T17:16:26.121863768Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
celery-worker_1 | 2019-12-19T17:16:26.121880291Z reraise(type(exception), exception, tb=exc_tb, cause=cause)
celery-worker_1 | 2019-12-19T17:16:26.121896652Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
celery-worker_1 | 2019-12-19T17:16:26.121913385Z raise value.with_traceback(tb)
celery-worker_1 | 2019-12-19T17:16:26.121929348Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
celery-worker_1 | 2019-12-19T17:16:26.121959071Z context = constructor(dialect, self, conn, *args)
celery-worker_1 | 2019-12-19T17:16:26.121974477Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 768, in _init_compiled
celery-worker_1 | 2019-12-19T17:16:26.121991535Z self.cursor = self.create_cursor()
celery-worker_1 | 2019-12-19T17:16:26.122007070Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 1122, in create_cursor
celery-worker_1 | 2019-12-19T17:16:26.122023641Z return self._dbapi_connection.cursor()
celery-worker_1 | 2019-12-19T17:16:26.122038439Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 965, in cursor
celery-worker_1 | 2019-12-19T17:16:26.122054801Z return self.connection.cursor(*args, **kwargs)
celery-worker_1 | 2019-12-19T17:16:26.122070443Z File "/usr/local/lib/python3.6/dist-packages/mysql/connector/connection_cext.py", line 524, in cursor
celery-worker_1 | 2019-12-19T17:16:26.122085888Z raise errors.OperationalError("MySQL Connection not available.")
celery-worker_1 | 2019-12-19T17:16:26.122103935Z sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available.
celery-worker_1 | 2019-12-19T17:16:26.122121490Z [SQL: SELECT user.id AS user_id, user.username AS user_username, user.password_hash AS user_password_hash, user.date_created AS user_date_created, user.date_modified AS user_date_modified
celery-worker_1 | 2019-12-19T17:16:26.122138006Z FROM user
celery-worker_1 | 2019-12-19T17:16:26.122153938Z WHERE user.username = %(username_1)s]
celery-worker_1 | 2019-12-19T17:16:26.122169866Z [parameters: [immutabledict({})]]
celery-worker_1 | 2019-12-19T17:16:26.122185555Z (Background on this error at: http://sqlalche.me/e/e3q8)
flask-app_1 | 2019-12-19T17:17:26.047373213Z Traceback (most recent call last):
flask-app_1 | 2019-12-19T17:17:26.047444956Z File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
flask-app_1 | 2019-12-19T17:17:26.047462758Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 82, in run
flask-app_1 | 2019-12-19T17:17:26.047477833Z self.result_consumer.drain_events(timeout=1)
flask-app_1 | 2019-12-19T17:17:26.047491810Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 129, in drain_events
flask-app_1 | 2019-12-19T17:17:26.047532854Z self.on_state_change(self._decode_result(message['data']), message)
flask-app_1 | 2019-12-19T17:17:26.047547076Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 305, in decode_result
flask-app_1 | 2019-12-19T17:17:26.047561236Z return self.meta_from_decoded(self.decode(payload))
flask-app_1 | 2019-12-19T17:17:26.047574918Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 301, in meta_from_decoded
flask-app_1 | 2019-12-19T17:17:26.047589072Z meta['result'] = self.exception_to_python(meta['result'])
flask-app_1 | 2019-12-19T17:17:26.047603211Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 281, in exception_to_python
flask-app_1 | 2019-12-19T17:17:26.047617358Z exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
flask-app_1 | 2019-12-19T17:17:26.047630160Z TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'
flask-app_1 | 2019-12-19T17:17:26.047643109Z 2019-12-19T17:16:26Z <Greenlet at 0x7efd45467e48: <bound method greenletDrainer.run of <celery.backends.asynchronous.geventDrainer object at 0x7efd44b55898>>> failed with TypeError
flask-app_1 | 2019-12-19T17:17:26.047661705Z
flask-app_1 | 2019-12-19T17:17:26.047676597Z Traceback (most recent call last):
flask-app_1 | 2019-12-19T17:17:26.047689450Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
flask-app_1 | 2019-12-19T17:17:26.047703803Z on_interval=on_interval):
flask-app_1 | 2019-12-19T17:17:26.047717065Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 54, in drain_events_until
flask-app_1 | 2019-12-19T17:17:26.047728717Z raise socket.timeout()
flask-app_1 | 2019-12-19T17:17:26.047739268Z socket.timeout
flask-app_1 | 2019-12-19T17:17:26.047750965Z
flask-app_1 | 2019-12-19T17:17:26.047762485Z During handling of the above exception, another exception occurred:
flask-app_1 | 2019-12-19T17:17:26.047774773Z
flask-app_1 | 2019-12-19T17:17:26.047786956Z Traceback (most recent call last):
flask-app_1 | 2019-12-19T17:17:26.047799216Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
flask-app_1 | 2019-12-19T17:17:26.047811661Z rv = self.dispatch_request()
flask-app_1 | 2019-12-19T17:17:26.047822671Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
flask-app_1 | 2019-12-19T17:17:26.047834915Z return self.view_functions[rule.endpoint](**req.view_args)
flask-app_1 | 2019-12-19T17:17:26.047846796Z File "/usr/local/lib/python3.6/dist-packages/flask_login/utils.py", line 261, in decorated_view
flask-app_1 | 2019-12-19T17:17:26.047859617Z return func(*args, **kwargs)
flask-app_1 | 2019-12-19T17:17:26.047873889Z File "/simoc/simoc_server/views.py", line 196, in new_game
flask-app_1 | 2019-12-19T17:17:26.047886338Z result = tasks.new_game.delay(get_standard_user_obj().username, game_config).get(timeout=60)
flask-app_1 | 2019-12-19T17:17:26.047898795Z File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 226, in get
flask-app_1 | 2019-12-19T17:17:26.047922069Z on_message=on_message,
flask-app_1 | 2019-12-19T17:17:26.047934090Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
flask-app_1 | 2019-12-19T17:17:26.047946914Z for _ in self._wait_for_pending(result, **kwargs):
flask-app_1 | 2019-12-19T17:17:26.047958970Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 259, in _wait_for_pending
flask-app_1 | 2019-12-19T17:17:26.047971330Z raise TimeoutError('The operation timed out.')
flask-app_1 | 2019-12-19T17:17:26.048044011Z celery.exceptions.TimeoutError: The operation timed out.
flask-app_1 | 2019-12-19T17:17:26.048057952Z [2019-12-19 17:17:26 +0000] [9] [ERROR] ERROR: handling exception The operation timed out.
flask-app_1 | 2019-12-19T17:17:26.060105233Z [2019-12-19 17:17:26 +0000] [9] [DEBUG] Closing connection.
flask-app_1 | 2019-12-19T17:25:32.960851907Z [2019-12-19 17:25:32 +0000] [9] [DEBUG] POST /new_game
celery-worker_1 | 2019-12-19T17:25:33.030124502Z [2019-12-19 17:25:33,029: INFO/MainProcess] Received task: celery_worker.tasks.new_game[57e5f028-1af4-4fae-9f17-69014d227263]
celery-worker_1 | 2019-12-19T17:25:33.030725899Z [2019-12-19 17:25:33,030: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f1781449c80> (args:('celery_worker.tasks.new_game', '57e5f028-1af4-4fae-9f17-69014d227263', {'lang': 'py', 'task': 'celery_worker.tasks.new_game', 'id': '57e5f028-1af4-4fae-9f17-69014d227263', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '57e5f028-1af4-4fae-9f17-69014d227263', 'parent_id': None, 'argsrepr': "('eziom', {'agents': {'human_agent': [...], 'solid_waste_aerobic_bioreactor': [...], 'multifiltration_purifier_post_treament': [...], 'oxygen_generation_SFWE': [...], 'urine_recycling_processor_VCD': [...], 'co2_removal_SAWD': [...], 'co2_reduction_sabatier': [...], 'crew_habitat_small': [...], 'solar_pv_array_mars': [...]}, 'storages': {'air_storage': [...], 'water_storage': [...], 'nutrient_storage': [...], 'power_storage': [...], 'food_storage': [...]}, 'termination': [{...}], 'single_agent': 1})", 'kwargsrepr': '{}', 'origin': 'gen9@c85a33855b29', 'reply_to': '339f0422-8bfe-3808-b4e4-ae30c8a7322b', 'correlation_id':... kwargs:{})
celery-worker_1 | 2019-12-19T17:25:33.033779578Z [2019-12-19 17:25:33,033: DEBUG/MainProcess] Task accepted: celery_worker.tasks.new_game[57e5f028-1af4-4fae-9f17-69014d227263] pid:18
celery-worker_1 | 2019-12-19T17:25:33.038549962Z [2019-12-19 17:25:33,037: ERROR/ForkPoolWorker-2] Task celery_worker.tasks.new_game[57e5f028-1af4-4fae-9f17-69014d227263] raised unexpected: StatementError("(sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back",)
celery-worker_1 | 2019-12-19T17:25:33.038618556Z Traceback (most recent call last):
celery-worker_1 | 2019-12-19T17:25:33.038626460Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
celery-worker_1 | 2019-12-19T17:25:33.038633363Z conn = self._revalidate_connection()
celery-worker_1 | 2019-12-19T17:25:33.038639887Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
celery-worker_1 | 2019-12-19T17:25:33.038646058Z "Can't reconnect until invalid "
celery-worker_1 | 2019-12-19T17:25:33.038652142Z sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
celery-worker_1 | 2019-12-19T17:25:33.038658256Z
celery-worker_1 | 2019-12-19T17:25:33.038664437Z The above exception was the direct cause of the following exception:
celery-worker_1 | 2019-12-19T17:25:33.038670877Z
celery-worker_1 | 2019-12-19T17:25:33.038676895Z Traceback (most recent call last):
celery-worker_1 | 2019-12-19T17:25:33.038682710Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task
celery-worker_1 | 2019-12-19T17:25:33.038689561Z R = retval = fun(*args, **kwargs)
celery-worker_1 | 2019-12-19T17:25:33.038695586Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 648, in __protected_call__
celery-worker_1 | 2019-12-19T17:25:33.038701784Z return self.run(*args, **kwargs)
celery-worker_1 | 2019-12-19T17:25:33.038707572Z File "/simoc/celery_worker/tasks.py", line 64, in new_game
celery-worker_1 | 2019-12-19T17:25:33.038713853Z user = get_user(username)
celery-worker_1 | 2019-12-19T17:25:33.038719735Z File "/simoc/celery_worker/tasks.py", line 34, in get_user
celery-worker_1 | 2019-12-19T17:25:33.038725873Z user = User.query.filter_by(username=username).all()
celery-worker_1 | 2019-12-19T17:25:33.038731446Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3211, in all
celery-worker_1 | 2019-12-19T17:25:33.038737755Z return list(self)
celery-worker_1 | 2019-12-19T17:25:33.038743350Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
celery-worker_1 | 2019-12-19T17:25:33.038749559Z return self._execute_and_instances(context)
celery-worker_1 | 2019-12-19T17:25:33.038755599Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
celery-worker_1 | 2019-12-19T17:25:33.038761869Z result = conn.execute(querycontext.statement, self._params)
celery-worker_1 | 2019-12-19T17:25:33.038767952Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
celery-worker_1 | 2019-12-19T17:25:33.038774129Z return meth(self, multiparams, params)
celery-worker_1 | 2019-12-19T17:25:33.038780904Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
celery-worker_1 | 2019-12-19T17:25:33.038787347Z return connection._execute_clauseelement(self, multiparams, params)
celery-worker_1 | 2019-12-19T17:25:33.038794725Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
celery-worker_1 | 2019-12-19T17:25:33.038801247Z distilled_params,
celery-worker_1 | 2019-12-19T17:25:33.038807224Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
celery-worker_1 | 2019-12-19T17:25:33.038818896Z e, util.text_type(statement), parameters, None, None
celery-worker_1 | 2019-12-19T17:25:33.038825236Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
celery-worker_1 | 2019-12-19T17:25:33.038831881Z util.raise_from_cause(sqlalchemy_exception, exc_info)
celery-worker_1 | 2019-12-19T17:25:33.038838164Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
celery-worker_1 | 2019-12-19T17:25:33.038844837Z reraise(type(exception), exception, tb=exc_tb, cause=cause)
celery-worker_1 | 2019-12-19T17:25:33.038850671Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
celery-worker_1 | 2019-12-19T17:25:33.038857087Z raise value.with_traceback(tb)
celery-worker_1 | 2019-12-19T17:25:33.038863071Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
celery-worker_1 | 2019-12-19T17:25:33.038869472Z conn = self._revalidate_connection()
celery-worker_1 | 2019-12-19T17:25:33.038875422Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
celery-worker_1 | 2019-12-19T17:25:33.038881769Z "Can't reconnect until invalid "
celery-worker_1 | 2019-12-19T17:25:33.038887605Z sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
celery-worker_1 | 2019-12-19T17:25:33.038893785Z [SQL: SELECT user.id AS user_id, user.username AS user_username, user.password_hash AS user_password_hash, user.date_created AS user_date_created, user.date_modified AS user_date_modified
celery-worker_1 | 2019-12-19T17:25:33.038900557Z FROM user
celery-worker_1 | 2019-12-19T17:25:33.038906401Z WHERE user.username = %(username_1)s]
celery-worker_1 | 2019-12-19T17:25:33.038912145Z [parameters: [immutabledict({})]]
flask-app_1 | 2019-12-19T17:26:33.029634407Z Traceback (most recent call last):
flask-app_1 | 2019-12-19T17:26:33.029705615Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
flask-app_1 | 2019-12-19T17:26:33.029715119Z on_interval=on_interval):
flask-app_1 | 2019-12-19T17:26:33.029723639Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 54, in drain_events_until
flask-app_1 | 2019-12-19T17:26:33.029731901Z raise socket.timeout()
flask-app_1 | 2019-12-19T17:26:33.029740936Z socket.timeout
flask-app_1 | 2019-12-19T17:26:33.042155720Z
flask-app_1 | 2019-12-19T17:26:33.042255984Z During handling of the above exception, another exception occurred:
flask-app_1 | 2019-12-19T17:26:33.042277378Z
flask-app_1 | 2019-12-19T17:26:33.042330996Z Traceback (most recent call last):
flask-app_1 | 2019-12-19T17:26:33.042351657Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
flask-app_1 | 2019-12-19T17:26:33.042403265Z rv = self.dispatch_request()
flask-app_1 | 2019-12-19T17:26:33.042423643Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
flask-app_1 | 2019-12-19T17:26:33.042480512Z return self.view_functions[rule.endpoint](**req.view_args)
flask-app_1 | 2019-12-19T17:26:33.042501409Z File "/usr/local/lib/python3.6/dist-packages/flask_login/utils.py", line 261, in decorated_view
flask-app_1 | 2019-12-19T17:26:33.042555878Z return func(*args, **kwargs)
flask-app_1 | 2019-12-19T17:26:33.042576874Z File "/simoc/simoc_server/views.py", line 196, in new_game
flask-app_1 | 2019-12-19T17:26:33.042627248Z result = tasks.new_game.delay(get_standard_user_obj().username, game_config).get(timeout=60)
flask-app_1 | 2019-12-19T17:26:33.042647660Z File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 226, in get
flask-app_1 | 2019-12-19T17:26:33.042701269Z on_message=on_message,
flask-app_1 | 2019-12-19T17:26:33.042720967Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
flask-app_1 | 2019-12-19T17:26:33.042774252Z for _ in self._wait_for_pending(result, **kwargs):
flask-app_1 | 2019-12-19T17:26:33.042811065Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 259, in _wait_for_pending
flask-app_1 | 2019-12-19T17:26:33.042861590Z raise TimeoutError('The operation timed out.')
flask-app_1 | 2019-12-19T17:26:33.042881638Z celery.exceptions.TimeoutError: The operation timed out.
flask-app_1 | 2019-12-19T17:26:33.042929864Z [2019-12-19 17:26:33 +0000] [9] [ERROR] ERROR: handling exception The operation timed out.
flask-app_1 | 2019-12-19T17:26:33.042953888Z [2019-12-19 17:26:33 +0000] [9] [DEBUG] Closing connection. I tried to fix it with my latest commit (https://github.com/kstaats/simoc/commit/02b071c735c0ffc346494e7558ff55a2bb827024).
Could you please test it one more time?
ezio-melotti
commented
4 years ago I got the another error today. I pulled the latest changes you pushed a couple of days ago, restarted the containers, launched SIMOC, run a sim, went back to the config wizard and left it there with the browser open. Now I selected the "one human" preset from the tab I left open, did "Finalize settings" and got the same error message.
The first time I did "Finalize settings" I got the usual OperationalError: MySQL Connection not available.:
flask-app_1 | 2019-12-24T22:20:25.433543694Z [2019-12-24 22:20:25 +0000] [9] [DEBUG] POST /new_game
celery-worker_1 | 2019-12-24T22:20:26.474008962Z [2019-12-24 22:20:26,409: INFO/MainProcess] Received task: celery_worker.tasks.new_game[2e827696-2b05-4bb5-b726-fdac7f46fb02]
celery-worker_1 | 2019-12-24T22:20:26.521595268Z [2019-12-24 22:20:26,410: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7ffa30169c80> (args:('celery_worker.tasks.new_game', '2e827696-2b05-4bb5-b726-fdac7f46fb02', {'lang': 'py', 'task': 'celery_worker.tasks.new_game', 'id': '2e827696-2b05-4bb5-b726-fdac7f46fb02', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2e827696-2b05-4bb5-b726-fdac7f46fb02', 'parent_id': None, 'argsrepr': "('eziom', {'agents': {'human_agent': [...], 'solid_waste_aerobic_bioreactor': [...], 'multifiltration_purifier_post_treament': [...], 'oxygen_generation_SFWE': [...], 'urine_recycling_processor_VCD': [...], 'co2_removal_SAWD': [...], 'co2_reduction_sabatier': [...], 'crew_habitat_small': [...], 'solar_pv_array_mars': [...]}, 'storages': {'air_storage': [...], 'water_storage': [...], 'nutrient_storage': [...], 'power_storage': [...], 'food_storage': [...]}, 'termination': [{...}], 'single_agent': 1})", 'kwargsrepr': '{}', 'origin': 'gen9@c85a33855b29', 'reply_to': '749757ed-da6e-341a-ad80-61221d9877e6', 'correlation_id':... kwargs:{})
celery-worker_1 | 2019-12-24T22:20:26.521658682Z [2019-12-24 22:20:26,413: DEBUG/MainProcess] Task accepted: celery_worker.tasks.new_game[2e827696-2b05-4bb5-b726-fdac7f46fb02] pid:18
celery-worker_1 | 2019-12-24T22:20:26.748598751Z [2019-12-24 22:20:26,725: ERROR/ForkPoolWorker-2] Task celery_worker.tasks.new_game[2e827696-2b05-4bb5-b726-fdac7f46fb02] raised unexpected: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.',)
celery-worker_1 | 2019-12-24T22:20:26.748656875Z Traceback (most recent call last):
celery-worker_1 | 2019-12-24T22:20:26.748673499Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
celery-worker_1 | 2019-12-24T22:20:26.748682459Z context = constructor(dialect, self, conn, *args)
celery-worker_1 | 2019-12-24T22:20:26.748689427Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 768, in _init_compiled
celery-worker_1 | 2019-12-24T22:20:26.748696663Z self.cursor = self.create_cursor()
celery-worker_1 | 2019-12-24T22:20:26.748703219Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 1122, in create_cursor
celery-worker_1 | 2019-12-24T22:20:26.748710946Z return self._dbapi_connection.cursor()
celery-worker_1 | 2019-12-24T22:20:26.748717712Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 965, in cursor
celery-worker_1 | 2019-12-24T22:20:26.748724549Z return self.connection.cursor(*args, **kwargs)
celery-worker_1 | 2019-12-24T22:20:26.748731644Z File "/usr/local/lib/python3.6/dist-packages/mysql/connector/connection_cext.py", line 524, in cursor
celery-worker_1 | 2019-12-24T22:20:26.748739101Z raise errors.OperationalError("MySQL Connection not available.")
celery-worker_1 | 2019-12-24T22:20:26.748765806Z mysql.connector.errors.OperationalError: MySQL Connection not available.
celery-worker_1 | 2019-12-24T22:20:26.748775466Z
celery-worker_1 | 2019-12-24T22:20:26.748782084Z The above exception was the direct cause of the following exception:
celery-worker_1 | 2019-12-24T22:20:26.748815497Z
celery-worker_1 | 2019-12-24T22:20:26.748822927Z Traceback (most recent call last):
celery-worker_1 | 2019-12-24T22:20:26.748857302Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task
celery-worker_1 | 2019-12-24T22:20:26.748864143Z R = retval = fun(*args, **kwargs)
celery-worker_1 | 2019-12-24T22:20:26.748870246Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 648, in __protected_call__
celery-worker_1 | 2019-12-24T22:20:26.748877246Z return self.run(*args, **kwargs)
celery-worker_1 | 2019-12-24T22:20:26.748883256Z File "/simoc/celery_worker/tasks.py", line 64, in new_game
celery-worker_1 | 2019-12-24T22:20:26.748889357Z user = get_user(username)
celery-worker_1 | 2019-12-24T22:20:26.748895153Z File "/simoc/celery_worker/tasks.py", line 34, in get_user
celery-worker_1 | 2019-12-24T22:20:26.748901226Z user = User.query.filter_by(username=username).all()
celery-worker_1 | 2019-12-24T22:20:26.748907223Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3211, in all
celery-worker_1 | 2019-12-24T22:20:26.748913288Z return list(self)
celery-worker_1 | 2019-12-24T22:20:26.748919112Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
celery-worker_1 | 2019-12-24T22:20:26.748926214Z return self._execute_and_instances(context)
celery-worker_1 | 2019-12-24T22:20:26.748935519Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
celery-worker_1 | 2019-12-24T22:20:26.748941951Z result = conn.execute(querycontext.statement, self._params)
celery-worker_1 | 2019-12-24T22:20:26.748948108Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
celery-worker_1 | 2019-12-24T22:20:26.748954516Z return meth(self, multiparams, params)
celery-worker_1 | 2019-12-24T22:20:26.748960638Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
celery-worker_1 | 2019-12-24T22:20:26.748966927Z return connection._execute_clauseelement(self, multiparams, params)
celery-worker_1 | 2019-12-24T22:20:26.748972937Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
celery-worker_1 | 2019-12-24T22:20:26.748979273Z distilled_params,
celery-worker_1 | 2019-12-24T22:20:26.748985201Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
celery-worker_1 | 2019-12-24T22:20:26.748992057Z e, util.text_type(statement), parameters, None, None
celery-worker_1 | 2019-12-24T22:20:26.748997947Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
celery-worker_1 | 2019-12-24T22:20:26.749004204Z util.raise_from_cause(sqlalchemy_exception, exc_info)
celery-worker_1 | 2019-12-24T22:20:26.749010798Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
celery-worker_1 | 2019-12-24T22:20:26.749016923Z reraise(type(exception), exception, tb=exc_tb, cause=cause)
celery-worker_1 | 2019-12-24T22:20:26.749022755Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
celery-worker_1 | 2019-12-24T22:20:26.749029222Z raise value.with_traceback(tb)
celery-worker_1 | 2019-12-24T22:20:26.749035121Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
celery-worker_1 | 2019-12-24T22:20:26.749047522Z context = constructor(dialect, self, conn, *args)
celery-worker_1 | 2019-12-24T22:20:26.749053647Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 768, in _init_compiled
celery-worker_1 | 2019-12-24T22:20:26.749059805Z self.cursor = self.create_cursor()
celery-worker_1 | 2019-12-24T22:20:26.749065649Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 1122, in create_cursor
celery-worker_1 | 2019-12-24T22:20:26.749071850Z return self._dbapi_connection.cursor()
celery-worker_1 | 2019-12-24T22:20:26.749077871Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 965, in cursor
celery-worker_1 | 2019-12-24T22:20:26.749083995Z return self.connection.cursor(*args, **kwargs)
celery-worker_1 | 2019-12-24T22:20:26.749089857Z File "/usr/local/lib/python3.6/dist-packages/mysql/connector/connection_cext.py", line 524, in cursor
celery-worker_1 | 2019-12-24T22:20:26.749096099Z raise errors.OperationalError("MySQL Connection not available.")
celery-worker_1 | 2019-12-24T22:20:26.749103125Z sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available.
celery-worker_1 | 2019-12-24T22:20:26.749109382Z [SQL: SELECT user.id AS user_id, user.username AS user_username, user.password_hash AS user_password_hash, user.date_created AS user_date_created, user.date_modified AS user_date_modified
celery-worker_1 | 2019-12-24T22:20:26.749116105Z FROM user
celery-worker_1 | 2019-12-24T22:20:26.749121874Z WHERE user.username = %(username_1)s]
celery-worker_1 | 2019-12-24T22:20:26.749127562Z [parameters: [immutabledict({})]]
celery-worker_1 | 2019-12-24T22:20:26.749133371Z (Background on this error at: http://sqlalche.me/e/e3q8)When I did it again I got first a TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig', and then an InvalidRequestError: Can't reconnect until invalid transaction is rolled back:
flask-app_1 | 2019-12-24T22:20:54.348815918Z Traceback (most recent call last):
flask-app_1 | 2019-12-24T22:20:54.348865173Z File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
flask-app_1 | 2019-12-24T22:20:54.348874569Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 82, in run
flask-app_1 | 2019-12-24T22:20:54.348882683Z self.result_consumer.drain_events(timeout=1)
flask-app_1 | 2019-12-24T22:20:54.348889679Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/redis.py", line 129, in drain_events
flask-app_1 | 2019-12-24T22:20:54.348896715Z self.on_state_change(self._decode_result(message['data']), message)
flask-app_1 | 2019-12-24T22:20:54.348903785Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 305, in decode_result
flask-app_1 | 2019-12-24T22:20:54.348910685Z return self.meta_from_decoded(self.decode(payload))
flask-app_1 | 2019-12-24T22:20:54.348920671Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 301, in meta_from_decoded
flask-app_1 | 2019-12-24T22:20:54.348930202Z meta['result'] = self.exception_to_python(meta['result'])
flask-app_1 | 2019-12-24T22:20:54.348939286Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/base.py", line 281, in exception_to_python
flask-app_1 | 2019-12-24T22:20:54.348948707Z exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
flask-app_1 | 2019-12-24T22:20:54.348958847Z TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'
flask-app_1 | 2019-12-24T22:20:54.348965721Z 2019-12-24T22:20:26Z <Greenlet at 0x7f0de0f8ee48: <bound method greenletDrainer.run of <celery.backends.asynchronous.geventDrainer object at 0x7f0de06ba048>>> failed with TypeError
flask-app_1 | 2019-12-24T22:20:54.348973551Z
flask-app_1 | 2019-12-24T22:20:54.348979823Z [2019-12-24 22:20:54 +0000] [9] [DEBUG] POST /new_game
celery-worker_1 | 2019-12-24T22:20:54.451641333Z [2019-12-24 22:20:54,451: INFO/MainProcess] Received task: celery_worker.tasks.new_game[932f0b38-ea56-4898-acd0-19892db962e2]
celery-worker_1 | 2019-12-24T22:20:54.454404762Z [2019-12-24 22:20:54,451: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7ffa30169c80> (args:('celery_worker.tasks.new_game', '932f0b38-ea56-4898-acd0-19892db962e2', {'lang': 'py', 'task': 'celery_worker.tasks.new_game', 'id': '932f0b38-ea56-4898-acd0-19892db962e2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '932f0b38-ea56-4898-acd0-19892db962e2', 'parent_id': None, 'argsrepr': "('eziom', {'agents': {'human_agent': [...], 'solid_waste_aerobic_bioreactor': [...], 'multifiltration_purifier_post_treament': [...], 'oxygen_generation_SFWE': [...], 'urine_recycling_processor_VCD': [...], 'co2_removal_SAWD': [...], 'co2_reduction_sabatier': [...], 'crew_habitat_small': [...], 'solar_pv_array_mars': [...]}, 'storages': {'air_storage': [...], 'water_storage': [...], 'nutrient_storage': [...], 'power_storage': [...], 'food_storage': [...]}, 'termination': [{...}], 'single_agent': 1})", 'kwargsrepr': '{}', 'origin': 'gen9@c85a33855b29', 'reply_to': '749757ed-da6e-341a-ad80-61221d9877e6', 'correlation_id':... kwargs:{})
celery-worker_1 | 2019-12-24T22:20:54.455449226Z [2019-12-24 22:20:54,454: DEBUG/MainProcess] Task accepted: celery_worker.tasks.new_game[932f0b38-ea56-4898-acd0-19892db962e2] pid:18
celery-worker_1 | 2019-12-24T22:20:54.461136466Z [2019-12-24 22:20:54,460: ERROR/ForkPoolWorker-2] Task celery_worker.tasks.new_game[932f0b38-ea56-4898-acd0-19892db962e2] raised unexpected: StatementError("(sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back",)
celery-worker_1 | 2019-12-24T22:20:54.461248957Z Traceback (most recent call last):
celery-worker_1 | 2019-12-24T22:20:54.461263105Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
celery-worker_1 | 2019-12-24T22:20:54.461270125Z conn = self._revalidate_connection()
celery-worker_1 | 2019-12-24T22:20:54.461281491Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
celery-worker_1 | 2019-12-24T22:20:54.461292566Z "Can't reconnect until invalid "
celery-worker_1 | 2019-12-24T22:20:54.461303048Z sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
celery-worker_1 | 2019-12-24T22:20:54.461314479Z
celery-worker_1 | 2019-12-24T22:20:54.461323409Z The above exception was the direct cause of the following exception:
celery-worker_1 | 2019-12-24T22:20:54.461334862Z
celery-worker_1 | 2019-12-24T22:20:54.461341619Z Traceback (most recent call last):
celery-worker_1 | 2019-12-24T22:20:54.461352327Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task
celery-worker_1 | 2019-12-24T22:20:54.461362310Z R = retval = fun(*args, **kwargs)
celery-worker_1 | 2019-12-24T22:20:54.461368645Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 648, in __protected_call__
celery-worker_1 | 2019-12-24T22:20:54.461381238Z return self.run(*args, **kwargs)
celery-worker_1 | 2019-12-24T22:20:54.461391241Z File "/simoc/celery_worker/tasks.py", line 64, in new_game
celery-worker_1 | 2019-12-24T22:20:54.461402702Z user = get_user(username)
celery-worker_1 | 2019-12-24T22:20:54.461413602Z File "/simoc/celery_worker/tasks.py", line 34, in get_user
celery-worker_1 | 2019-12-24T22:20:54.461424517Z user = User.query.filter_by(username=username).all()
celery-worker_1 | 2019-12-24T22:20:54.461434487Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3211, in all
celery-worker_1 | 2019-12-24T22:20:54.461441686Z return list(self)
celery-worker_1 | 2019-12-24T22:20:54.461452948Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
celery-worker_1 | 2019-12-24T22:20:54.461462291Z return self._execute_and_instances(context)
celery-worker_1 | 2019-12-24T22:20:54.461468875Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
celery-worker_1 | 2019-12-24T22:20:54.461482665Z result = conn.execute(querycontext.statement, self._params)
celery-worker_1 | 2019-12-24T22:20:54.461490135Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
celery-worker_1 | 2019-12-24T22:20:54.461501492Z return meth(self, multiparams, params)
celery-worker_1 | 2019-12-24T22:20:54.461511482Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
celery-worker_1 | 2019-12-24T22:20:54.461522622Z return connection._execute_clauseelement(self, multiparams, params)
celery-worker_1 | 2019-12-24T22:20:54.461535886Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
celery-worker_1 | 2019-12-24T22:20:54.461543447Z distilled_params,
celery-worker_1 | 2019-12-24T22:20:54.461558655Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
celery-worker_1 | 2019-12-24T22:20:54.461565658Z e, util.text_type(statement), parameters, None, None
celery-worker_1 | 2019-12-24T22:20:54.461578357Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
celery-worker_1 | 2019-12-24T22:20:54.461588049Z util.raise_from_cause(sqlalchemy_exception, exc_info)
celery-worker_1 | 2019-12-24T22:20:54.461594968Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
celery-worker_1 | 2019-12-24T22:20:54.461604822Z reraise(type(exception), exception, tb=exc_tb, cause=cause)
celery-worker_1 | 2019-12-24T22:20:54.461611559Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
celery-worker_1 | 2019-12-24T22:20:54.461618473Z raise value.with_traceback(tb)
celery-worker_1 | 2019-12-24T22:20:54.461624651Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
celery-worker_1 | 2019-12-24T22:20:54.461630716Z conn = self._revalidate_connection()
celery-worker_1 | 2019-12-24T22:20:54.461636819Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
celery-worker_1 | 2019-12-24T22:20:54.461645012Z "Can't reconnect until invalid "
celery-worker_1 | 2019-12-24T22:20:54.461651466Z sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
celery-worker_1 | 2019-12-24T22:20:54.461658260Z [SQL: SELECT user.id AS user_id, user.username AS user_username, user.password_hash AS user_password_hash, user.date_created AS user_date_created, user.date_modified AS user_date_modified
celery-worker_1 | 2019-12-24T22:20:54.461665643Z FROM user
celery-worker_1 | 2019-12-24T22:20:54.461674985Z WHERE user.username = %(username_1)s]
celery-worker_1 | 2019-12-24T22:20:54.461683848Z [parameters: [immutabledict({})]]
flask-app_1 | 2019-12-24T22:21:26.408251611Z Traceback (most recent call last):
flask-app_1 | 2019-12-24T22:21:26.408325693Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
flask-app_1 | 2019-12-24T22:21:26.408335547Z on_interval=on_interval):
flask-app_1 | 2019-12-24T22:21:26.408344096Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 54, in drain_events_until
flask-app_1 | 2019-12-24T22:21:26.408353328Z raise socket.timeout()
flask-app_1 | 2019-12-24T22:21:26.408360155Z socket.timeout
flask-app_1 | 2019-12-24T22:21:26.408367372Z
flask-app_1 | 2019-12-24T22:21:26.408374685Z During handling of the above exception, another exception occurred:
flask-app_1 | 2019-12-24T22:21:26.408382822Z
flask-app_1 | 2019-12-24T22:21:26.408389821Z Traceback (most recent call last):
flask-app_1 | 2019-12-24T22:21:26.408397729Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
flask-app_1 | 2019-12-24T22:21:26.408405915Z rv = self.dispatch_request()
flask-app_1 | 2019-12-24T22:21:26.408413186Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
flask-app_1 | 2019-12-24T22:21:26.408449008Z return self.view_functions[rule.endpoint](**req.view_args)
flask-app_1 | 2019-12-24T22:21:26.408456571Z File "/usr/local/lib/python3.6/dist-packages/flask_login/utils.py", line 261, in decorated_view
flask-app_1 | 2019-12-24T22:21:26.408463728Z return func(*args, **kwargs)
flask-app_1 | 2019-12-24T22:21:26.408470799Z File "/simoc/simoc_server/views.py", line 196, in new_game
flask-app_1 | 2019-12-24T22:21:26.408477679Z result = tasks.new_game.delay(get_standard_user_obj().username, game_config).get(timeout=60)
flask-app_1 | 2019-12-24T22:21:26.408487712Z File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 226, in get
flask-app_1 | 2019-12-24T22:21:26.408494320Z on_message=on_message,
flask-app_1 | 2019-12-24T22:21:26.408501596Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
flask-app_1 | 2019-12-24T22:21:26.408508632Z for _ in self._wait_for_pending(result, **kwargs):
flask-app_1 | 2019-12-24T22:21:26.408515871Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 259, in _wait_for_pending
flask-app_1 | 2019-12-24T22:21:26.408522955Z raise TimeoutError('The operation timed out.')
flask-app_1 | 2019-12-24T22:21:26.408529741Z celery.exceptions.TimeoutError: The operation timed out.
flask-app_1 | 2019-12-24T22:21:26.408536463Z [2019-12-24 22:21:26 +0000] [9] [ERROR] ERROR: handling exception The operation timed out.
flask-app_1 | 2019-12-24T22:21:26.421127656Z [2019-12-24 22:21:26 +0000] [9] [DEBUG] Closing connection.
flask-app_1 | 2019-12-24T22:21:54.457629898Z Traceback (most recent call last):
flask-app_1 | 2019-12-24T22:21:54.457700262Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
flask-app_1 | 2019-12-24T22:21:54.457710245Z on_interval=on_interval):
flask-app_1 | 2019-12-24T22:21:54.457717970Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 54, in drain_events_until
flask-app_1 | 2019-12-24T22:21:54.457726384Z raise socket.timeout()
flask-app_1 | 2019-12-24T22:21:54.457733231Z socket.timeout
flask-app_1 | 2019-12-24T22:21:54.457739869Z
flask-app_1 | 2019-12-24T22:21:54.457745642Z During handling of the above exception, another exception occurred:
flask-app_1 | 2019-12-24T22:21:54.457751452Z
flask-app_1 | 2019-12-24T22:21:54.457756527Z Traceback (most recent call last):
flask-app_1 | 2019-12-24T22:21:54.457761751Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
flask-app_1 | 2019-12-24T22:21:54.457767511Z rv = self.dispatch_request()
flask-app_1 | 2019-12-24T22:21:54.457772740Z File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
flask-app_1 | 2019-12-24T22:21:54.457778194Z return self.view_functions[rule.endpoint](**req.view_args)
flask-app_1 | 2019-12-24T22:21:54.457783482Z File "/usr/local/lib/python3.6/dist-packages/flask_login/utils.py", line 261, in decorated_view
flask-app_1 | 2019-12-24T22:21:54.457789064Z return func(*args, **kwargs)
flask-app_1 | 2019-12-24T22:21:54.457794193Z File "/simoc/simoc_server/views.py", line 196, in new_game
flask-app_1 | 2019-12-24T22:21:54.457829194Z result = tasks.new_game.delay(get_standard_user_obj().username, game_config).get(timeout=60)
flask-app_1 | 2019-12-24T22:21:54.457835166Z File "/usr/local/lib/python3.6/dist-packages/celery/result.py", line 226, in get
flask-app_1 | 2019-12-24T22:21:54.457840717Z on_message=on_message,
flask-app_1 | 2019-12-24T22:21:54.457845823Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
flask-app_1 | 2019-12-24T22:21:54.457851309Z for _ in self._wait_for_pending(result, **kwargs):
flask-app_1 | 2019-12-24T22:21:54.457856430Z File "/usr/local/lib/python3.6/dist-packages/celery/backends/asynchronous.py", line 259, in _wait_for_pending
flask-app_1 | 2019-12-24T22:21:54.457861959Z raise TimeoutError('The operation timed out.')
flask-app_1 | 2019-12-24T22:21:54.457867138Z celery.exceptions.TimeoutError: The operation timed out.
flask-app_1 | 2019-12-24T22:21:54.457872428Z [2019-12-24 22:21:54 +0000] [9] [ERROR] ERROR: handling exception The operation timed out.
flask-app_1 | 2019-12-24T22:21:54.462508640Z [2019-12-24 22:21:54 +0000] [9] [DEBUG] Closing connection.I got this again with the websockets. I ran a sim, left the browser opened on the dashboard for ~10 hours, came back to it and edited some file (that caused the dashboard to reload and re-request the steps) and got the error:
celery-worker_2 | 2020-01-08T15:41:52.061826897Z [2020-01-08 15:41:52,060: INFO/ForkPoolWorker-2] Cleaning up save game for user with id 1
flask-app_1 | 2020-01-09T01:23:47.717703001Z [2020-01-09 01:23:47 +0000] [9] [DEBUG] POST /get_step_to
flask-app_1 | 2020-01-09T01:23:47.794902574Z [2020-01-09 01:23:47 +0000] [9] [DEBUG] POST /kill_game
flask-app_1 | 2020-01-09T01:23:47.833790293Z [2020-01-09 01:23:47 +0000] [9] [INFO] STATUS: Steps requested.
flask-app_1 | 2020-01-09T01:23:47.836561224Z [2020-01-09 01:23:47 +0000] [9] [DEBUG] Closing connection.
flask-app_1 | 2020-01-09T01:23:47.837038365Z [2020-01-09 01:23:47 +0000] [9] [INFO] STATUS: Game 5912925101529154487 killed.
celery-worker_2 | 2020-01-09T01:23:47.833984305Z [2020-01-09 01:23:47,833: INFO/MainProcess] Received task: celery_worker.tasks.get_step_to[51d42bb6-b4c4-46d4-9ff4-c240f1bee08e]
celery-worker_2 | 2020-01-09T01:23:47.834009894Z [2020-01-09 01:23:47,833: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fa036ed0c80> (args:('celery_worker.tasks.get_step_to', '51d42bb6-b4c4-46d4-9ff4-c240f1bee08e', {'lang': 'py', 'task': 'celery_worker.tasks.get_step_to', 'id': '51d42bb6-b4c4-46d4-9ff4-c240f1bee08e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '51d42bb6-b4c4-46d4-9ff4-c240f1bee08e', 'parent_id': None, 'argsrepr': "['eziom', 240]", 'kwargsrepr': '{}', 'origin': 'gen9@c85a33855b29', 'reply_to': '6547237e-245c-3952-bf82-72b03e20f16a', 'correlation_id': '51d42bb6-b4c4-46d4-9ff4-c240f1bee08e', 'delivery_info': {'exchange': '', 'routing_key': 'worker@d4c7e4488c61.dq2', 'priority': 0, 'redelivered': None}}, b'[["eziom", 240], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
celery-worker_2 | 2020-01-09T01:23:47.837753623Z [2020-01-09 01:23:47,837: DEBUG/MainProcess] pidbox received method revoke(task_id='e04a0889-2f14-428d-ab44-3dc8dd46d2a0', terminate=True, signal='SIGKILL') [reply_to:None ticket:None]
celery-worker_1 | 2020-01-09T01:23:47.835386183Z [2020-01-09 01:23:47,835: DEBUG/MainProcess] pidbox received method revoke(task_id='e04a0889-2f14-428d-ab44-3dc8dd46d2a0', terminate=True, signal='SIGKILL') [reply_to:None ticket:None]
flask-app_1 | 2020-01-09T01:23:47.839204036Z [2020-01-09 01:23:47 +0000] [9] [DEBUG] Closing connection.
celery-worker_2 | 2020-01-09T01:23:47.838654627Z [2020-01-09 01:23:47,838: DEBUG/MainProcess] Task accepted: celery_worker.tasks.get_step_to[51d42bb6-b4c4-46d4-9ff4-c240f1bee08e] pid:18
celery-worker_2 | 2020-01-09T01:23:47.913084510Z [2020-01-09 01:23:47,912: ERROR/ForkPoolWorker-2] Task celery_worker.tasks.get_step_to[51d42bb6-b4c4-46d4-9ff4-c240f1bee08e] raised unexpected: OperationalError('(mysql.connector.errors.OperationalError) MySQL Connection not available.',)
celery-worker_2 | 2020-01-09T01:23:47.913125097Z Traceback (most recent call last):
celery-worker_2 | 2020-01-09T01:23:47.913132807Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
celery-worker_2 | 2020-01-09T01:23:47.913139701Z context = constructor(dialect, self, conn, *args)
celery-worker_2 | 2020-01-09T01:23:47.913146409Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 768, in _init_compiled
celery-worker_2 | 2020-01-09T01:23:47.913152897Z self.cursor = self.create_cursor()
celery-worker_2 | 2020-01-09T01:23:47.913159526Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 1122, in create_cursor
celery-worker_2 | 2020-01-09T01:23:47.913166897Z return self._dbapi_connection.cursor()
celery-worker_2 | 2020-01-09T01:23:47.913176977Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 965, in cursor
celery-worker_2 | 2020-01-09T01:23:47.913183995Z return self.connection.cursor(*args, **kwargs)
celery-worker_2 | 2020-01-09T01:23:47.913190234Z File "/usr/local/lib/python3.6/dist-packages/mysql/connector/connection_cext.py", line 524, in cursor
celery-worker_2 | 2020-01-09T01:23:47.913211911Z raise errors.OperationalError("MySQL Connection not available.")
celery-worker_2 | 2020-01-09T01:23:47.913217959Z mysql.connector.errors.OperationalError: MySQL Connection not available.
celery-worker_2 | 2020-01-09T01:23:47.913223600Z
celery-worker_2 | 2020-01-09T01:23:47.913229526Z The above exception was the direct cause of the following exception:
celery-worker_2 | 2020-01-09T01:23:47.913236499Z
celery-worker_2 | 2020-01-09T01:23:47.913242169Z Traceback (most recent call last):
celery-worker_2 | 2020-01-09T01:23:47.913247923Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 385, in trace_task
celery-worker_2 | 2020-01-09T01:23:47.913255092Z R = retval = fun(*args, **kwargs)
celery-worker_2 | 2020-01-09T01:23:47.913261815Z File "/usr/local/lib/python3.6/dist-packages/celery/app/trace.py", line 648, in __protected_call__
celery-worker_2 | 2020-01-09T01:23:47.913267820Z return self.run(*args, **kwargs)
celery-worker_2 | 2020-01-09T01:23:47.913273310Z File "/simoc/celery_worker/tasks.py", line 74, in get_step_to
celery-worker_2 | 2020-01-09T01:23:47.913279077Z user = get_user(username)
celery-worker_2 | 2020-01-09T01:23:47.913285124Z File "/simoc/celery_worker/tasks.py", line 34, in get_user
celery-worker_2 | 2020-01-09T01:23:47.913290220Z user = User.query.filter_by(username=username).all()
celery-worker_2 | 2020-01-09T01:23:47.913295640Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3211, in all
celery-worker_2 | 2020-01-09T01:23:47.913301106Z return list(self)
celery-worker_2 | 2020-01-09T01:23:47.913307718Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3367, in __iter__
celery-worker_2 | 2020-01-09T01:23:47.913313792Z return self._execute_and_instances(context)
celery-worker_2 | 2020-01-09T01:23:47.913321538Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3392, in _execute_and_instances
celery-worker_2 | 2020-01-09T01:23:47.913327771Z result = conn.execute(querycontext.statement, self._params)
celery-worker_2 | 2020-01-09T01:23:47.913333079Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
celery-worker_2 | 2020-01-09T01:23:47.913338274Z return meth(self, multiparams, params)
celery-worker_2 | 2020-01-09T01:23:47.913343695Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
celery-worker_2 | 2020-01-09T01:23:47.913349976Z return connection._execute_clauseelement(self, multiparams, params)
celery-worker_2 | 2020-01-09T01:23:47.913355458Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
celery-worker_2 | 2020-01-09T01:23:47.913360891Z distilled_params,
celery-worker_2 | 2020-01-09T01:23:47.913366124Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
celery-worker_2 | 2020-01-09T01:23:47.913371820Z e, util.text_type(statement), parameters, None, None
celery-worker_2 | 2020-01-09T01:23:47.913377283Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
celery-worker_2 | 2020-01-09T01:23:47.913382919Z util.raise_from_cause(sqlalchemy_exception, exc_info)
celery-worker_2 | 2020-01-09T01:23:47.913393191Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
celery-worker_2 | 2020-01-09T01:23:47.913399131Z reraise(type(exception), exception, tb=exc_tb, cause=cause)
celery-worker_2 | 2020-01-09T01:23:47.913404904Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
celery-worker_2 | 2020-01-09T01:23:47.913410956Z raise value.with_traceback(tb)
celery-worker_2 | 2020-01-09T01:23:47.913416856Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
celery-worker_2 | 2020-01-09T01:23:47.913423808Z context = constructor(dialect, self, conn, *args)
celery-worker_2 | 2020-01-09T01:23:47.913429488Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 768, in _init_compiled
celery-worker_2 | 2020-01-09T01:23:47.913435148Z self.cursor = self.create_cursor()
celery-worker_2 | 2020-01-09T01:23:47.913440464Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 1122, in create_cursor
celery-worker_2 | 2020-01-09T01:23:47.913446252Z return self._dbapi_connection.cursor()
celery-worker_2 | 2020-01-09T01:23:47.913451937Z File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/pool/base.py", line 965, in cursor
celery-worker_2 | 2020-01-09T01:23:47.913457537Z return self.connection.cursor(*args, **kwargs)
celery-worker_2 | 2020-01-09T01:23:47.913462953Z File "/usr/local/lib/python3.6/dist-packages/mysql/connector/connection_cext.py", line 524, in cursor
celery-worker_2 | 2020-01-09T01:23:47.913468836Z raise errors.OperationalError("MySQL Connection not available.")
celery-worker_2 | 2020-01-09T01:23:47.913474765Z sqlalchemy.exc.OperationalError: (mysql.connector.errors.OperationalError) MySQL Connection not available.
celery-worker_2 | 2020-01-09T01:23:47.913480448Z [SQL: SELECT user.id AS user_id, user.username AS user_username, user.password_hash AS user_password_hash, user.date_created AS user_date_created, user.date_modified AS user_date_modified
celery-worker_2 | 2020-01-09T01:23:47.913486716Z FROM user
celery-worker_2 | 2020-01-09T01:23:47.913492025Z WHERE user.username = %(username_1)s]
celery-worker_2 | 2020-01-09T01:23:47.913498485Z [parameters: [immutabledict({})]]
celery-worker_2 | 2020-01-09T01:23:47.913503706Z (Background on this error at: http://sqlalche.me/e/e3q8)
flask-app_1 | 2020-01-09T01:23:48.153331853Z [2020-01-09 01:23:48 +0000] [9] [DEBUG] GET /socket.io/
A few days ago, MySQLconnection was not available and apparently that created an invalid transaction that locked the DB until the transaction was rolled back. Since nothing rolled that transaction back, the DB was stuck until Kai noticed and I restarted the docker.
These are the logs for the day, with an analysis at the bottom:
OperationalError: MySQL Connection not available.InvalidRequestError: Can't reconnect until invalid transaction is rolled backInvalidRequestError: Can't reconnect until invalid transaction is rolled back