Closed adrianLIrobotics closed 1 year ago
Rethinking the way of filtering is very close to reviewing how we access and store the data. For example, it may turn out that storing data in another database (relational or document) can be a better idea than keeping it in Redis, as we do at the moment. Things to consider in the DB are that it should provide us with a way to scale it to multiple replicas.
We should handle this topic sooner than later. We will need a separate meeting to discuss this. I would suggest a meeting next week when Radu gets better. What we have to discuss:
Possible packages to be used: https://github.com/redis/redis-om-dotnet https://github.com/redis/NRedisStack
The biggest migration challenge is to enable backward compatibility with the data. Some partners are using an older version of the Middleware than the latest. The migration can be complicated within a single database. A new data store will be needed.
Possible migration strategy:
At the meetings, it was decided that we would keep using Redis but would rework the querying system and use different packages, allowing us better querying capabilities. The following things have been decided:
In the future, there will be a situation in which we will have to store historical data. Redis is unsuitable for this. For analytics and historical data, the SQL database will be needed.
Ad.1 When Redis is in cluster mode, the cluster replica will have to be deployed as a part of the Middleware and automatically connect to the existing cluster. Therefore, the cluster should function in the form of Multi-Leader Replication.
Ad.2: The process of switching the data access has to complete a set of steps:
Ad.3 On top of the data migration, the plan for upgrade and migration of the existing Middleware instances has to be prepared to allow the least possible overhead for the end-users.
I am moving this to the "In progress column" as development has started for this one :)
Describe the bug Performance degradation over redis when query for an specific stored plan.
To Reproduce Steps to reproduce the behavior i.e.:
Expected behavior Nice performance over query for plan stored in redis. There is a need to improve the query.
Screenshots N/A
Additional context Add any other context about the problem here.