Open hpatro opened 4 months ago
@valkey-io/core-team WDYT ?
It would be great if anyone wants to add such tests, because we're a bit overloaded at this point. It doesn't have to be in TCL (maybe Python instead?) and it doesn't even have to be in this repo, if we're testing different forks. We can make a separate interop testing repo. It is about
i am willing to spend time. if anyone guide me some starting point.
We can make a separate interop testing repo. It is about
I think it should be this repo, I don't think we want to start introducing tests elsewhere for now.
I'm fine either way, but what id'd like to see is some prototyping to get something running. Use any language, any client lib, but be ready to discard it later.
There are potential benefits of having a completely separate repo for this. It would need to check out various versions of valkey anyway (so it can't just run out of the checked out repo). It could also test valkey against various forks and versions. We can have test a cluster with mixed nodes of KeyDB, Dragonfly, Redict, Redis, Valkey, various versions. We can run redis testsuites on the valkey binary. Etc.
@madolson @zuiderkwast @pragnesh @mattsta Hi everyone! Is anyone already working on this? We currently have a Python test script that provides the same functionality as the TCL tests, but written in Python. It has been working well for the past two years and has significantly improved the efficiency of writing and debugging tests compared to TCL. Here is a relatively old version: https://pypi.org/project/pybbt/ If you're interested, I'm willing to spend 1-2 weeks adapting it for Valkey and writing a few test examples.
@suxb201 how is the test setup/teardown done? How long does a similar tcl test takes in python?
@hpatro
@subcase
are safely destroyed when the @subcase
ends.@case
will be destroyed when the @case
concludes.@subcase
is concurrent, which makes testing efficient.@case
per file, and the tests are composed of multiple files.Here's a example:
from testsuite import *
# Master can replicate command longer than client-query-buffer-limit on replica
@subcase()
def replication_query_buffer_limit():
t0 = Tair() # create a process
t1 = Tair() # create another process
t1.wait_slaveof(t0) # call 'slaveof' and then wait for the replication connection to be set up
t0.do("config", "set", "client-query-buffer-limit", 2000000)
t1.do("config", "set", "client-query-buffer-limit", 1048576) # 1024*1024 = 1mb
value = "x" * 1100000
ASSERT_TRUE(t0.do("set", "k", value)) # 2000000 > 1100000 > 1048576
t0.wait_consistent()
ASSERT_EQ(t1.do("get", "k"), value.encode())
ASSERT_EQ(t1.digest(), t0.digest())
@case(tags=["flag0", "flag1", "flag2"])
def main():
replication_query_buffer_limit()
if __name__ == '__main__':
main()
It seems to me that porting the tests to a different language is a different topic. I think it can be discussed here: #94.
Personally, I think it would be beneficial to create a separate interoperability-testing repo where scripts and CI jobs can
@bjosv
Reference: https://github.com/tair-opensource/resp-compatibility. Alibaba has some compatibility testing already we should investigate.
I am the owner of resp-compatibility. It was originally designed to detect the Redis version compatible with the Redis-Like system from the interface, but it is also very suitable for version compatibility testing of Redis/Valkey itself. From the discussion in the issue, it can currently do
tests | YES or NO |
---|---|
api | YES |
cluster bus (nodes of different versions in the same cluster) | NO(But can improve) |
replication (primary/replica of different versions) | NO(But can improve) |
RDB and AOF files | YES |
We can:
Please share any thoughts you have, thank you.
@valkey-io/core-team We chatted a bit in the core meeting and want to have a vote if we want to pull in the compatibility test listed above and run it as part of the CI.
We still need a separate effort to do interversion operability tests (clusterbus and replication).
@zuiderkwast Did we secretly do this?
@zuiderkwast Did we secretly do this?
I must have pocket-clicked close...
@valkey-io/core-team We chatted a bit in the core meeting and want to have a vote if we want to pull in the compatibility test listed above and run it as part of the CI.
We still need a separate effort to do interversion operability tests (clusterbus and replication).
Agree, but Daily CI is not necessary.
DESCRIPTION
Introduce cross version/cross fork integration testing infrastructure. With the compatibility version release and planned new major version release, it will be good to improve the testing/release certification process. This will help Valkey to prepare for release(s) more confidently and avoid pain for user(s) during migration/upgrade(s).
Example Scenario:
Issue: https://github.com/redis/redis/issues/12685
Redis 7.2 introduced cluster bus message extensions feature by default and it caused failure of engine upgrade from older version (i.e. Redis 6.2 or lower) due to message broadcasted from engine running 7.2 not being compatible in older versions.
PR to fix the issue during upgrade: https://github.com/valkey-io/valkey/pull/52