Seems like the discovery of e.g. the database at the beginning of the lifetime of a container is a bad idea, if the database changes location afterwards.
Traceback (most recent call last):
File "worker-launcher.py", line 207, in consume
run()
File "/var/python/worker/fontbakery.py", line 354, in run
fonts = _prepare(self._job, self._cache, self._dbOps, None)
File "/var/python/worker/fontbakery.py", line 194, in _prepare
dbOps.update({'preparation_logs': logs})
File "/var/python/worker/fontbakery.py", line 91, in update
return self.q.get(self._docid).update(_doc).run(self.conn)
File "/usr/local/lib/python3.7/dist-packages/rethinkdb/ast.py", line 140, in run
return c._start(self, **global_optargs)
File "/usr/local/lib/python3.7/dist-packages/rethinkdb/net.py", line 678, in _start
self.check_open()
File "/usr/local/lib/python3.7/dist-packages/rethinkdb/net.py", line 652, in check_open
raise ReqlDriverError('Connection is closed.')
rethinkdb.errors.ReqlDriverError: Connection is closed.
The workers seem to catch the error and survive this and hence never reconnect to the database then. Therefore, at least this could lead to a fatal pod failure and kill the container. It will then be restarted, so that the next version can discover the database again.
Seems like the discovery of e.g. the database at the beginning of the lifetime of a container is a bad idea, if the database changes location afterwards.
The workers seem to catch the error and survive this and hence never reconnect to the database then. Therefore, at least this could lead to a fatal pod failure and kill the container. It will then be restarted, so that the next version can discover the database again.