search

crossbario / crossbar

Crossbar.io - WAMP application router
https://crossbar.io/
Other
2.05k stars 274 forks source link

Open zlmdb databases only at most once per process #2033

Open oberstet opened 2 years ago

oberstet commented 2 years ago

zlmdb has recently tightened its run-time checks, one of which ensure a given database is opened at most once within a single process.

https://github.com/crossbario/zlmdb/blob/c952f420212e5bea33690f2678fa161d920fc580/zlmdb/_database.py#L354

apparently, this is currently violated in _agg_metering_mworker_logs, which SHOULD use the DB handle already (somewhere) in the worker:

2022-07-03T08:36:27+0200 [Container   15651] 
Traceback (most recent call last):
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/master/node/controller.py", line 968, in _do_metering
    mrealm_res = yield deferToThread(self._agg_metering_mworker_logs, from_ts, until_ts, mrealm_id)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/master/node/controller.py", line 718, in _agg_metering_mworker_logs
    db = zlmdb.Database(dbpath=dbpath, readonly=False)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/zlmdb/_database.py", line 338, in __init__
    self.__enter__()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/zlmdb/_database.py", line 355, in __enter__
    raise RuntimeError('tried to open same dbpath "{}" twice within '
builtins.RuntimeError: tried to open same dbpath "/home/oberstet/scm/crossbario/crossbar/test/cfc/.crossbar/.db-mrealm-d1aaf64f-ea35-4a08-9b5d-7293edb82113" twice within same process (PID 15651) from <zlmdb._database.Database object at 0x7fc4dcfc0ac0>

2022-07-03T08:36:27+0200 [Container   15651] 
Traceback (most recent call last):
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/master/node/controller.py", line 968, in _do_metering
    mrealm_res = yield deferToThread(self._agg_metering_mworker_logs, from_ts, until_ts, mrealm_id)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/master/node/controller.py", line 718, in _agg_metering_mworker_logs
    db = zlmdb.Database(dbpath=dbpath, readonly=False)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/zlmdb/_database.py", line 338, in __init__
    self.__enter__()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-api-1/lib/python3.9/site-packages/zlmdb/_database.py", line 355, in __enter__
    raise RuntimeError('tried to open same dbpath "{}" twice within '
builtins.RuntimeError: tried to open same dbpath "/home/oberstet/scm/crossbario/crossbar/test/cfc/.crossbar/.db-mrealm-d1aaf64f-ea35-4a08-9b5d-7293edb82113" twice within same process (PID 15651) from <zlmdb._database.Database object at 0x7fc4dd0ee430>

here is another instance

2022-07-03T09:52:42+0200 [Controller  25492] Management realm "mrealm1" backend container "cfcontainer1" already running (skipped starting container)
2022-07-03T09:52:42+0200 [Container   25538] component instantiation failed: tried to open same dbpath "/home/oberstet/scm/crossbario/crossbar/test/cfc/.crossbar/.db-controller" twice within same process (PID 25538) from <zlmdb._database.Database object at 0x7f1cf78bc250>
Traceback (most recent call last):
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/internet/endpoints.py", line 147, in dataReceived
    return self._wrappedProtocol.dataReceived(data)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/autobahn/twisted/rawsocket.py", line 410, in dataReceived
    self._on_handshake_complete()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/autobahn/twisted/rawsocket.py", line 117, in _on_handshake_complete
    self._session = self.factory._factory()
--- <exception caught here> ---
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/worker/container.py", line 364, in create_session
    session = create_component(component_config)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/master/mrealm/controller.py", line 190, in __init__
    self.gdb = zlmdb.Database(dbpath=dbfile, maxsize=maxsize, readonly=False, sync=True)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/zlmdb/_database.py", line 338, in __init__
    self.__enter__()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/zlmdb/_database.py", line 355, in __enter__
    raise RuntimeError('tried to open same dbpath "{}" twice within '
builtins.RuntimeError: tried to open same dbpath "/home/oberstet/scm/crossbario/crossbar/test/cfc/.crossbar/.db-controller" twice within same process (PID 25538) from <zlmdb._database.Database object at 0x7f1cf78bc250>

2022-07-03T09:52:42+0200 [Container   25538] WampRawSocketClientProtocol._on_handshake_complete(): ApplicationSession constructor / onOpen raised (tried to open same dbpath "/home/oberstet/scm/crossbario/crossbar/test/cfc/.crossbar/.db-controller" twice within same process (PID 25538) from <zlmdb._database.Database object at 0x7f1cf78bc250>)
2022-07-03T09:52:42+0200 [Container   25538] Unhandled Error
Traceback (most recent call last):
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/python/log.py", line 96, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/python/log.py", line 80, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
--- <exception caught here> ---
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/internet/posixbase.py", line 683, in _doReadOrWrite
    why = selectable.doRead()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/internet/unix.py", line 193, in doRead
    return self._dataReceived(data)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/twisted/internet/endpoints.py", line 147, in dataReceived
    return self._wrappedProtocol.dataReceived(data)
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/autobahn/twisted/rawsocket.py", line 410, in dataReceived
    self._on_handshake_complete()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/autobahn/twisted/rawsocket.py", line 125, in _on_handshake_complete
    self.abort()
  File "/home/oberstet/scm/crossbario/crossbar/.tox/py39-cli-2/lib/python3.9/site-packages/autobahn/twisted/rawsocket.py", line 240, in abort
    raise TransportLost()
autobahn.wamp.exception.TransportLost:
oberstet commented 2 years ago 
(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ find crossbar -name "*.py" -exec grep -Hi "= zlmdb.Database" {} \;
crossbar/network/_api.py:        self._db = zlmdb.Database(dbpath=self._dbpath, maxsize=2**30, readonly=False, sync=True)
crossbar/network/_authenticator.py:        self._db = zlmdb.Database(dbpath=self._dbpath, maxsize=2**30, readonly=False, sync=True)
crossbar/master/mrealm/controller.py:        self.gdb = zlmdb.Database(dbpath=dbfile, maxsize=maxsize, readonly=False, sync=True)
crossbar/master/mrealm/controller.py:        self.db = zlmdb.Database(dbpath=dbfile, maxsize=maxsize, readonly=False, sync=True)
crossbar/master/node/controller.py:        self.db = zlmdb.Database(dbpath=dbpath, maxsize=maxsize, readonly=False, sync=True)
crossbar/master/node/controller.py:        db = zlmdb.Database(dbpath=dbpath, readonly=False)
crossbar/master/node/controller.py:        db = zlmdb.Database(dbpath=dbpath, readonly=False)
crossbar/master/node/controller.py:        db = zlmdb.Database(dbpath=dbpath, readonly=False)
crossbar/master/node/authenticator.py:        self._db = zlmdb.Database(dbpath=dbpath, maxsize=maxsize, readonly=False, sync=True)
crossbar/master/node/node.py:        self._db = zlmdb.Database(dbpath=dbpath, maxsize=maxsize, readonly=False, sync=True)
crossbar/edge/worker/realmstore.py:        self._db = zlmdb.Database(dbpath=dbpath, maxsize=maxsize, readonly=readonly, sync=sync)
crossbar/edge/worker/xbrmm.py:        self._db = zlmdb.Database(dbpath=self._dbpath,
crossbar/edge/worker/xbr/_marketmaker.py:        self._db = zlmdb.Database(dbpath=dbpath, maxsize=maxsize, readonly=readonly, sync=sync)
crossbar/router/cookiestore.py:        self._db = zlmdb.Database(dbpath=dbpath, maxsize=maxsize, readonly=readonly, sync=sync)
oberstet commented 2 years ago

ok, so this is the problem:

(cpy39_1) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ find crossbar -name "*.py" -exec grep -Hi ".db-controller" {} \;
crossbar/master/node/controller.py:        dbpath = config.get('path', '.db-controller')
crossbar/master/node/authenticator.py:        dbpath = config.get('path', '.db-controller')
crossbar/master/node/node.py:        dbpath = config.get('path', '.db-controller')