[BCDB] Get/Set calls on BCDBModel should go over BCDB connector if not threebot [development_fixes]

[xmonader]

Get/Set calls on BCDBModel should go over BCDB connector if not from threebot otherwise should execute without problems in normal flow

[xmonader]

Relevant:

in workers tab

    OperationalError('database is locked',)
--TRACEBACK------------------
/sandbox/var/cmds/workers_1.py in <module>
    9    j.servers.myjobs._worker_inprocess_start_from_tmux(1)
/sandbox/lib/jumpscale/jumpscale_generated.py in myjobs
    378    self._myjobs =  MyJobsFactory()
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSBase.py in __init__
    64    self._init(**kwargs)
/sandbox/lib/jumpscale/Jumpscale/servers/myjobs/MyJobsFactory.py in _init
    27    self._bcdb = self._bcdb_selector()
/sandbox/lib/jumpscale/Jumpscale/servers/myjobs/MyJobsFactory.py in _bcdb_selector
    78    client = j.clients.rdb.client_get()
/sandbox/lib/jumpscale/Jumpscale/clients/stor_rdb/RDBFactory.py in client_get
    23    client = j.clients.rdb.get(f"{namespace}_myjobs", nsname=namespace)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSConfigsBCDB.py in get
    115    rc, jsconfig = self._get(name=name, id=id, die=needexist, reload=reload)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSConfigsBCDB.py in _get
    164    res = self.find(name=name)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSConfigsBCDB.py in find
    245    ids = self._model.find_ids(**kwargs)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in find_ids
    603    cursor = self._find_query(nid, **kwargs)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in _find_query
    585    return self.query_model([field], whereclause, values)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in query_model
    589    query = f"select {fieldstring} FROM {self.index.sql_table_name} "
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in index
    97    self._index_ = indexklass(model=self, reset=False)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSBase.py in __init__
    64    self._init(**kwargs)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModelIndex.py in _init
    61    self._sql_index_init()
/sandbox/var/codegen/bcdbindex_jumpscale_rdb_client_6469707b5f20a71b512fb12ad6dcfb30.py in _sql_index_init
    49    self.sql.create_table(safe=True)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in create_table
    6473    cls._schema.create_all(safe, **options)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in create_all
    5630    self.create_table(safe, **table_options)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in create_table
    5505    self.database.execute(self._create_table(safe=safe, **options))
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in execute
    3082    return self.execute_sql(sql, params, commit=commit)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in execute_sql
    3076    self.commit()
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in __exit__
    2831    reraise(new_type, new_type(*exc_args), traceback)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in reraise
    196    raise value.with_traceback(tb)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in execute_sql
    3069    cursor.execute(sql, params or ())
-----------------------------

threebot default tab

Tue 10 13:09:48 ab89334c307aa9189.py -  31 - 1_24bc86709efd3dbab89334c307aa9189 : init index:jumpscale.threebot_server.1
Tue 10 13:09:53 nts/peewee/peewee.py -3069 - execute_sql                        : EXCEPTION: 
    OperationalError('database is locked',)
--TRACEBACK------------------
/sandbox/var/cmds/threebot_default.py in <module>
    4    server = j.servers.threebot.get("default", executor='TMUX', web=False)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSConfigsBCDB.py in get
    115    rc, jsconfig = self._get(name=name, id=id, die=needexist, reload=reload)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSConfigsBCDB.py in _get
    164    res = self.find(name=name)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSConfigsBCDB.py in find
    245    ids = self._model.find_ids(**kwargs)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in find_ids
    603    cursor = self._find_query(nid, **kwargs)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in _find_query
    585    return self.query_model([field], whereclause, values)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in query_model
    589    query = f"select {fieldstring} FROM {self.index.sql_table_name} "
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModel.py in index
    97    self._index_ = indexklass(model=self, reset=False)
/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSBase.py in __init__
    64    self._init(**kwargs)
/sandbox/lib/jumpscale/Jumpscale/data/bcdb/BCDBModelIndex.py in _init
    61    self._sql_index_init()
/sandbox/var/codegen/bcdbindex_jumpscale_threebot_server_1_24bc86709efd3dbab89334c307aa9189.py in _sql_index_init
    49    self.sql.create_table(safe=True)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in create_table
    6473    cls._schema.create_all(safe, **options)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in create_all
    5630    self.create_table(safe, **table_options)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in create_table
    5505    self.database.execute(self._create_table(safe=safe, **options))
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in execute
    3082    return self.execute_sql(sql, params, commit=commit)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in execute_sql
    3076    self.commit()
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in __exit__
    2831    reraise(new_type, new_type(*exc_args), traceback)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in reraise
    196    raise value.with_traceback(tb)
/sandbox/lib/jumpscale/Jumpscale/clients/peewee/peewee.py in execute_sql
    3069    cursor.execute(sql, params or ())
-----------------------------

[abom]

Trying to get redis bcdb connector starts with a single process and other processes of jumpscale connect to it to do bcdb operations of set/get/delete.

RedisServer is tied to specific bcdb instance, now trying to refactor this to be more consistent, also to enable a single server to handle requests for any bcdb instance.

https://github.com/threefoldtech/jumpscaleX_core/blob/5e2e297de52c8d39af7731e3547efdb1e37ca9bc/JumpscaleCore/data/bcdb/connectors/redis/RedisServer.py#L33

Currently it's not consistent, as some commands are handled with bcdb passed to server instance and other not, current bcdb is used in hscan, get, delete and hlen, but not in set

https://github.com/threefoldtech/jumpscaleX_core/blob/5e2e297de52c8d39af7731e3547efdb1e37ca9bc/JumpscaleCore/data/bcdb/connectors/redis/RedisServer.py#L238-L247

https://github.com/threefoldtech/jumpscaleX_core/blob/5e2e297de52c8d39af7731e3547efdb1e37ca9bc/JumpscaleCore/data/bcdb/connectors/redis/RedisServer.py#L253-L258

So, bcdb redis connector should automatically start with the main threebot process only.

[xmonader]

cfcc176d30682e2cd6e840b3fe43120d564d75f4

[Dina-Abd-Elrahman]

• verified.