Open wangting0128 opened 4 hours ago
- Milvus version:2.5-20241031-164ed80e-amd64 - Deployment mode(standalone or cluster):cluster - MQ type(rocksmq, pulsar or kafka):pulsar - SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc97 - OS(Ubuntu or CentOS): - CPU/Memory: - GPU: - Others:
argo task: multi-vector-corn-1-1730383200 test case name: test_hybrid_search_locust_load_release_replica_cluster
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES multi-vector-corn-1-1730383200-2-etcd-0 1/1 Running 0 13h 10.104.15.52 4am-node20 <none> <none> multi-vector-corn-1-1730383200-2-etcd-1 1/1 Running 0 13h 10.104.20.196 4am-node22 <none> <none> multi-vector-corn-1-1730383200-2-etcd-2 1/1 Running 1 (117m ago) 13h 10.104.34.73 4am-node37 <none> <none> multi-vector-corn-1-1730383200-2-milvus-datanode-76cf855cck54gw 1/1 Running 3 (13h ago) 13h 10.104.34.62 4am-node37 <none> <none> multi-vector-corn-1-1730383200-2-milvus-indexnode-85c58754djqsk 1/1 Running 2 (13h ago) 13h 10.104.23.36 4am-node27 <none> <none> multi-vector-corn-1-1730383200-2-milvus-mixcoord-67d9c4746jp59n 1/1 Running 5 (117m ago) 13h 10.104.34.63 4am-node37 <none> <none> multi-vector-corn-1-1730383200-2-milvus-proxy-5b8c64d658-tqt2h 1/1 Running 3 (13h ago) 13h 10.104.32.22 4am-node39 <none> <none> multi-vector-corn-1-1730383200-2-milvus-querynode-84856b4btvq6w 1/1 Running 2 (13h ago) 13h 10.104.32.21 4am-node39 <none> <none> multi-vector-corn-1-1730383200-2-milvus-querynode-84856b4bx796l 1/1 Running 2 (13h ago) 13h 10.104.19.139 4am-node28 <none> <none> multi-vector-corn-1-1730383200-2-minio-0 1/1 Running 0 13h 10.104.15.49 4am-node20 <none> <none> multi-vector-corn-1-1730383200-2-minio-1 1/1 Running 0 13h 10.104.20.194 4am-node22 <none> <none> multi-vector-corn-1-1730383200-2-minio-2 1/1 Running 0 13h 10.104.30.24 4am-node38 <none> <none> multi-vector-corn-1-1730383200-2-minio-3 1/1 Running 0 13h 10.104.34.74 4am-node37 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-bookie-0 1/1 Running 0 13h 10.104.30.22 4am-node38 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-bookie-1 1/1 Running 0 13h 10.104.15.59 4am-node20 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-bookie-2 1/1 Running 0 13h 10.104.18.240 4am-node25 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-bookie-init-sgnzf 0/1 Completed 0 13h 10.104.18.227 4am-node25 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-broker-0 1/1 Running 0 13h 10.104.9.14 4am-node14 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-proxy-0 1/1 Running 0 13h 10.104.18.229 4am-node25 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-pulsar-init-l6kfm 0/1 Completed 0 13h 10.104.18.228 4am-node25 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-recovery-0 1/1 Running 0 13h 10.104.9.13 4am-node14 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-zookeeper-0 1/1 Running 0 13h 10.104.20.192 4am-node22 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-zookeeper-1 1/1 Running 0 13h 10.104.18.15 4am-node25 <none> <none> multi-vector-corn-1-1730383200-2-pulsar-zookeeper-2 1/1 Running 0 13h 10.104.16.121 4am-node21 <none> <none>
rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout load_collection_failed.log
rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
client log:
[2024-11-01 02:01:08,259 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=65535, message=empty grpc client: find no available querycoord, check querycoord state)>, <Time:{'RPC start': '2024-11-01 01:59:45.745227', 'RPC error': '2024-11-01 02:01:08.259286'}> (decorators.py:140) [2024-11-01 02:01:08,260 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=65535, message=empty grpc client: find no available querycoord, check querycoord state)>, <Time:{'RPC start': '2024-11-01 01:58:05.653698', 'RPC error': '2024-11-01 02:01:08.260247'}> (decorators.py:140) [2024-11-01 02:01:08,260 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=65535, message=empty grpc client: find no available querycoord, check querycoord state)>, <Time:{'RPC start': '2024-11-01 01:58:04.468923', 'RPC error': '2024-11-01 02:01:08.260460'}> (decorators.py:140) [2024-11-01 02:01:08,264 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=empty grpc client: find no available querycoord, check querycoord state)>, [requestId: bb766464-97f4-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:01:08,264 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available querycoord, check querycoord state)> (func_check.py:101) [2024-11-01 02:01:24,106 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:01:24,106 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:01:24,106 - INFO - fouram]: grpc load_release 2691 0(0.00%) | 13327 1423 35064 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:01:24,106 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:01:24,106 - INFO - fouram]: Aggregated 2691 0(0.00%) | 13327 1423 35064 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:01:24,106 - INFO - fouram]: (stats.py:790) [2024-11-01 02:01:24,107 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:01:24,107 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:01:24,107 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:01:24,107 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 19000 23000 35000 35000 2691 (stats.py:819) [2024-11-01 02:01:24,107 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:01:24,107 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 19000 23000 35000 35000 2691 (stats.py:819) [2024-11-01 02:01:24,107 - INFO - fouram]: (stats.py:820) [2024-11-01 02:01:44,108 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:01:44,108 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:01:44,108 - INFO - fouram]: grpc load_release 2691 0(0.00%) | 13327 1423 35064 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:01:44,108 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:01:44,108 - INFO - fouram]: Aggregated 2691 0(0.00%) | 13327 1423 35064 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:01:44,108 - INFO - fouram]: (stats.py:790) [2024-11-01 02:01:44,109 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:01:44,109 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:01:44,109 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:01:44,109 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 19000 23000 35000 35000 2691 (stats.py:819) [2024-11-01 02:01:44,109 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:01:44,109 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 19000 23000 35000 35000 2691 (stats.py:819) [2024-11-01 02:01:44,109 - INFO - fouram]: (stats.py:820) [2024-11-01 02:02:04,111 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:02:04,111 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:02:04,111 - INFO - fouram]: grpc load_release 2691 0(0.00%) | 13327 1423 35064 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:02:04,111 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:02:04,111 - INFO - fouram]: Aggregated 2691 0(0.00%) | 13327 1423 35064 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:02:04,111 - INFO - fouram]: (stats.py:790) [2024-11-01 02:02:04,111 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:02:04,112 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:02:04,112 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:02:04,112 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 19000 23000 35000 35000 2691 (stats.py:819) [2024-11-01 02:02:04,112 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:02:04,112 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 19000 23000 35000 35000 2691 (stats.py:819) [2024-11-01 02:02:04,112 - INFO - fouram]: (stats.py:820) [2024-11-01 02:02:09,276 - ERROR - fouram]: RPC error: [release_collection], <MilvusException: (code=10001, message=stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /workspace/source/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...] /workspace/source/internal/distributed/rootcoord/client/client.go:308 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp /workspace/source/internal/proxy/timestamp.go:60 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc /workspace/source/internal/proxy/timestamp.go:85 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne /workspace/source/internal/proxy/task_scheduler.go:185 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue /workspace/source/internal/proxy/task_scheduler.go:367 github.com/milvus-io/milvus/internal/proxy.(*ddTaskQueue).Enqueue /workspace/source/internal/proxy/impl.go:894 github.com/milvus-io/milvus/internal/proxy.(*Proxy).ReleaseCollection: context deadline exceeded)>, <Time:{'RPC start': '2024-11-01 02:01:08.264647', 'RPC error': '2024-11-01 02:02:09.276327'}> (decorators.py:140) [2024-11-01 02:02:09,277 - ERROR - fouram]: (api_response) : [Collection.release] <MilvusException: (code=10001, message=stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /workspace/source/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...] /workspace/source/internal/distributed/rootcoord/client/client.go:308 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp /workspace/source/internal/proxy/timestamp.go:60 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc /workspace/source/internal/proxy/timestamp.go:85 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne /workspace/source/internal/proxy/task_scheduler.go:185 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue /workspace/source/internal/proxy/task_scheduler.go:367 github.com/milvus-io/milvus/internal/proxy.(*ddTaskQueue).Enqueue /workspace/source/internal/proxy/impl.go:894 github.com/milvus-io/milvus/internal/proxy.(*Proxy).ReleaseCollection: context deadline exceeded)>, [requestId: 2903655e-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:02:09,277 - ERROR - fouram]: [CheckFunc] release request check failed, response:<MilvusException: (code=10001, message=stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /workspace/source/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...] /workspace/source/internal/distributed/rootcoord/client/client.go:308 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp /workspace/source/internal/proxy/timestamp.go:60 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc /workspace/source/internal/proxy/timestamp.go:85 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne /workspace/source/internal/proxy/task_scheduler.go:185 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue /workspace/source/internal/proxy/task_scheduler.go:367 github.com/milvus-io/milvus/internal/proxy.(*ddTaskQueue).Enqueue /workspace/source/internal/proxy/impl.go:894 github.com/milvus-io/milvus/internal/proxy.(*Proxy).ReleaseCollection: context deadline exceeded)> (func_check.py:101) [2024-11-01 02:02:09,279 - ERROR - fouram]: [func_time_catch] : [Base] Check concurrent_load_release result failed, load: False, release: False (api_request.py:127) [2024-11-01 02:02:22,521 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, <Time:{'RPC start': '2024-11-01 02:02:09.279994', 'RPC error': '2024-11-01 02:02:22.521176'}> (decorators.py:140) [2024-11-01 02:02:22,521 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, [requestId: 4d61990c-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:02:22,521 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)> (func_check.py:101) [2024-11-01 02:02:24,122 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:02:24,122 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:02:24,122 - INFO - fouram]: grpc load_release 2692 1(0.04%) | 13413 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:02:24,122 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:02:24,123 - INFO - fouram]: Aggregated 2692 1(0.04%) | 13413 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:02:24,123 - INFO - fouram]: (stats.py:790) [2024-11-01 02:02:24,123 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:02:24,123 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:02:24,123 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:02:24,123 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 19000 24000 245000 245000 2692 (stats.py:819) [2024-11-01 02:02:24,123 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:02:24,124 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 19000 24000 245000 245000 2692 (stats.py:819) [2024-11-01 02:02:24,124 - INFO - fouram]: (stats.py:820) [2024-11-01 02:02:32,428 - ERROR - fouram]: RPC error: [release_collection], <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, <Time:{'RPC start': '2024-11-01 02:02:22.522431', 'RPC error': '2024-11-01 02:02:32.428709'}> (decorators.py:140) [2024-11-01 02:02:32,430 - ERROR - fouram]: (api_response) : [Collection.release] <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, [requestId: 55463c36-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:02:32,430 - ERROR - fouram]: [CheckFunc] release request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)> (func_check.py:101) [2024-11-01 02:02:32,431 - ERROR - fouram]: [func_time_catch] : [Base] Check concurrent_load_release result failed, load: False, release: False (api_request.py:127) [2024-11-01 02:02:44,125 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:02:44,125 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:02:44,125 - INFO - fouram]: grpc load_release 2693 2(0.07%) | 13416 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:02:44,125 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:02:44,125 - INFO - fouram]: Aggregated 2693 2(0.07%) | 13416 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:02:44,125 - INFO - fouram]: (stats.py:790) [2024-11-01 02:02:44,126 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:02:44,126 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:02:44,126 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:02:44,126 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 20000 24000 245000 245000 2693 (stats.py:819) [2024-11-01 02:02:44,126 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:02:44,126 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 20000 24000 245000 245000 2693 (stats.py:819) [2024-11-01 02:02:44,126 - INFO - fouram]: (stats.py:820) [2024-11-01 02:02:48,012 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, <Time:{'RPC start': '2024-11-01 02:02:32.431929', 'RPC error': '2024-11-01 02:02:48.012407'}> (decorators.py:140) [2024-11-01 02:02:48,013 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, [requestId: 5b2e4d0a-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:02:48,013 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)> (func_check.py:101) [2024-11-01 02:02:58,321 - ERROR - fouram]: RPC error: [release_collection], <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, <Time:{'RPC start': '2024-11-01 02:02:48.013565', 'RPC error': '2024-11-01 02:02:58.321712'}> (decorators.py:140) [2024-11-01 02:02:58,322 - ERROR - fouram]: (api_response) : [Collection.release] <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, [requestId: 6477e164-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:02:58,322 - ERROR - fouram]: [CheckFunc] release request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)> (func_check.py:101) [2024-11-01 02:02:58,322 - ERROR - fouram]: [func_time_catch] : [Base] Check concurrent_load_release result failed, load: False, release: False (api_request.py:127) [2024-11-01 02:03:04,127 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:03:04,127 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:03:04,127 - INFO - fouram]: grpc load_release 2694 3(0.11%) | 13421 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:03:04,127 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:03:04,127 - INFO - fouram]: Aggregated 2694 3(0.11%) | 13421 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:03:04,127 - INFO - fouram]: (stats.py:790) [2024-11-01 02:03:04,128 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:03:04,128 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:03:04,128 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:03:04,128 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2694 (stats.py:819) [2024-11-01 02:03:04,128 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:03:04,128 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2694 (stats.py:819) [2024-11-01 02:03:04,128 - INFO - fouram]: (stats.py:820) [2024-11-01 02:03:05,818 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, <Time:{'RPC start': '2024-11-01 02:02:58.323355', 'RPC error': '2024-11-01 02:03:05.818228'}> (decorators.py:140) [2024-11-01 02:03:05,818 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, [requestId: 6a9d0682-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:03:05,819 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)> (func_check.py:101) [2024-11-01 02:03:15,095 - ERROR - fouram]: RPC error: [release_collection], <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, <Time:{'RPC start': '2024-11-01 02:03:05.819530', 'RPC error': '2024-11-01 02:03:15.095740'}> (decorators.py:140) [2024-11-01 02:03:15,096 - ERROR - fouram]: (api_response) : [Collection.release] <MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)>, [requestId: 6f14da0a-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:03:15,096 - ERROR - fouram]: [CheckFunc] release request check failed, response:<MilvusException: (code=65535, message=empty grpc client: find no available rootcoord, check rootcoord state)> (func_check.py:101) [2024-11-01 02:03:15,096 - ERROR - fouram]: [func_time_catch] : [Base] Check concurrent_load_release result failed, load: False, release: False (api_request.py:127) [2024-11-01 02:03:24,129 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:03:24,130 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:03:24,130 - INFO - fouram]: grpc load_release 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:03:24,130 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:03:24,130 - INFO - fouram]: Aggregated 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:03:24,130 - INFO - fouram]: (stats.py:790) [2024-11-01 02:03:24,131 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:03:24,131 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:03:24,131 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:03:24,131 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:03:24,131 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:03:24,131 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:03:24,131 - INFO - fouram]: (stats.py:820) [2024-11-01 02:03:44,132 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:03:44,132 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:03:44,132 - INFO - fouram]: grpc load_release 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:03:44,133 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:03:44,133 - INFO - fouram]: Aggregated 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:03:44,133 - INFO - fouram]: (stats.py:790) [2024-11-01 02:03:44,133 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:03:44,133 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:03:44,133 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:03:44,133 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:03:44,133 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:03:44,133 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:03:44,133 - INFO - fouram]: (stats.py:820) [2024-11-01 02:04:04,134 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:04:04,134 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:04:04,135 - INFO - fouram]: grpc load_release 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:04:04,135 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:04:04,135 - INFO - fouram]: Aggregated 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:04:04,135 - INFO - fouram]: (stats.py:790) [2024-11-01 02:04:04,135 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:04:04,135 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:04:04,135 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:04:04,135 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:04:04,136 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:04:04,136 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:04:04,136 - INFO - fouram]: (stats.py:820) [2024-11-01 02:04:08,768 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=65535, message=call query coordinator LoadCollection: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /workspace/source/internal/distributed/querycoord/client/client.go:100 github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...] /workspace/source/internal/distributed/querycoord/client/client.go:152 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).LoadCollection /workspace/source/internal/proxy/task.go:1609 github.com/milvus-io/milvus/internal/proxy.(*loadCollectionTask).Execute /workspace/source/internal/proxy/task_scheduler.go:477 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask /workspace/source/internal/proxy/task_scheduler.go:503 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695 runtime.goexit: empty grpc client: find no available querycoord, check querycoord state)>, <Time:{'RPC start': '2024-11-01 02:03:15.097419', 'RPC error': '2024-11-01 02:04:08.768919'}> (decorators.py:140) [2024-11-01 02:04:08,770 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=call query coordinator LoadCollection: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /workspace/source/internal/distributed/querycoord/client/client.go:100 github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...] /workspace/source/internal/distributed/querycoord/client/client.go:152 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).LoadCollection /workspace/source/internal/proxy/task.go:1609 github.com/milvus-io/milvus/internal/proxy.(*loadCollectionTask).Execute /workspace/source/internal/proxy/task_scheduler.go:477 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask /workspace/source/internal/proxy/task_scheduler.go:503 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695 runtime.goexit: empty grpc client: find no available querycoord, check querycoord state)>, [requestId: 749c8bb2-97f5-11ef-8531-6a960a80238e] (api_request.py:57) [2024-11-01 02:04:08,771 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=65535, message=call query coordinator LoadCollection: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /workspace/source/internal/distributed/querycoord/client/client.go:100 github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...] /workspace/source/internal/distributed/querycoord/client/client.go:152 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).LoadCollection /workspace/source/internal/proxy/task.go:1609 github.com/milvus-io/milvus/internal/proxy.(*loadCollectionTask).Execute /workspace/source/internal/proxy/task_scheduler.go:477 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask /workspace/source/internal/proxy/task_scheduler.go:503 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).definitionLoop /go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695 runtime.goexit: empty grpc client: find no available querycoord, check querycoord state)> (func_check.py:101) [2024-11-01 02:04:24,138 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:04:24,138 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:04:24,138 - INFO - fouram]: grpc load_release 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:04:24,138 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:04:24,139 - INFO - fouram]: Aggregated 2695 4(0.15%) | 13422 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:04:24,139 - INFO - fouram]: (stats.py:790) [2024-11-01 02:04:24,139 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819) [2024-11-01 02:04:24,139 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819) [2024-11-01 02:04:24,139 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:04:24,139 - INFO - fouram]: grpc load_release 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:04:24,141 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819) [2024-11-01 02:04:24,141 - INFO - fouram]: Aggregated 13000 13000 14000 14000 15000 16000 18000 20000 26000 245000 245000 2695 (stats.py:819) [2024-11-01 02:04:24,141 - INFO - fouram]: (stats.py:820) [2024-11-01 02:04:36,868 - ERROR - fouram]: [func_time_catch] : [Base] Check concurrent_load_release result failed, load: False, release: True (api_request.py:127) [2024-11-01 02:04:44,142 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 02:04:44,142 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:04:44,143 - INFO - fouram]: grpc load_release 2696 5(0.19%) | 13448 1423 244809 13000 | 0.00 0.00 (stats.py:789) [2024-11-01 02:04:44,143 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 02:04:44,143 - INFO - fouram]: Aggregated 2696 5(0.19%) | 13448 1423 244809 13000 | 0.00 0.00 (stats.py:789)
No response
[2024-10-31 14:04:33,900 - INFO - fouram]: [setup_method] Start setup test case test_hybrid_search_locust_load_release_replica_cluster, test document: concurrent test and calculation of RT and QPS :purpose: `load -> release collection, replica=2` verify load -> release collection scenario, which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1` :test steps: 1. create collection with fields: 'float_vector': 128dim, 'float_vector_1': 128dim, 'float_vector_2': 128dim, 'float_vector_3': 128dim, scalar field: int64_1, varchar_1 2. build indexes: IVF_FLAT: 'float_vector' HNSW: 'float_vector_1', DISKANN: 'float_vector_2' IVF_SQ8: 'float_vector_3' INVERTED: 'int64_1', 'varchar_1' default scalar index: 'id' 3. insert 5 million data 4. flush collection 5. build indexes again using the same params 6. load collection replica: 2 7. concurrent request: (concurrent_number=1) - load_release
test result:
[2024-11-01 04:00:21,178 - INFO - fouram]: Print locust final stats. (locust_runner.py:56) [2024-11-01 04:00:21,179 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789) [2024-11-01 04:00:21,179 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 04:00:21,179 - INFO - fouram]: grpc load_release 3101 5(0.16%) | 13927 1423 244809 13000 | 0.07 0.00 (stats.py:789) [2024-11-01 04:00:21,179 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789) [2024-11-01 04:00:21,180 - INFO - fouram]: Aggregated 3101 5(0.16%) | 13927 1423 244809 13000 | 0.07 0.00 (stats.py:789) [2024-11-01 04:00:21,180 - INFO - fouram]: (stats.py:790) [2024-11-01 04:00:21,183 - INFO - fouram]: [PerfTemplate] Report data: {'server': {'deploy_tool': 'helm', 'deploy_mode': 'cluster', 'config_name': 'cluster_2c8m', 'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '32Gi'}, 'requests': {'cpu': '5.0', 'memory': '17Gi'}}, 'replicas': 2}, 'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 1}, 'dataNode': {'resources': {'limits': {'cpu': '2.0', 'memory': '8Gi'}, 'requests': {'cpu': '2.0', 'memory': '5Gi'}}}, 'cluster': {'enabled': True}, 'pulsar': {}, 'kafka': {}, 'minio': {'metrics': {'podMonitor': {'enabled': True}}}, 'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}}, 'metrics': {'serviceMonitor': {'enabled': True}}, 'log': {'level': 'debug'}, 'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': '2.5-20241031-164ed80e-amd64'}}}, 'host': 'multi-vector-corn-1-1730383200-2-milvus.qa-milvus.svc.cluster.local', 'port': '19530', 'uri': ''}, 'client': {'test_case_type': 'ConcurrentClientBase', 'test_case_name': 'test_hybrid_search_locust_load_release_replica_cluster', 'test_case_params': {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'scalars_index': {'id': {}, 'int64_1': {'index_type': 'INVERTED'}, 'varchar_1': {'index_type': 'INVERTED'}}, 'vectors_index': {'float_vector_1': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}, 'metric_type': 'L2'}, 'float_vector_2': {'index_type': 'DISKANN', 'index_param': {}, 'metric_type': 'IP'}, 'float_vector_3': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 2048}, 'metric_type': 'L2'}}, 'scalars_params': {'float_vector_1': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}, 'float_vector_2': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}, 'float_vector_3': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}}, 'dataset_name': 'sift', 'dataset_size': 5000000, 'ni_per': 10000}, 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int64_1', 'varchar_1'], 'shards_num': 2}, 'load_params': {'replica_number': 2}, 'resource_groups_params': {'reset': False}, 'database_user_params': {'reset_rbac': False, 'reset_db': False}, 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}}, 'concurrent_params': {'concurrent_number': 1, 'during_time': '12h', 'interval': 20, 'spawn_rate': None}, 'concurrent_tasks': [{'type': 'load_release', 'weight': 1, 'params': {'replica_number': 2, 'timeout': 600, 'check_tasks': None}}]}, 'run_id': 2024103134734924, 'datetime': '2024-10-31 14:04:33.900808', 'client_version': '2.5.0'}, 'result': {'test_result': {'index': {'RT': 2423.9575, 'float_vector_1': {'RT': 1602.0847}, 'float_vector_2': {'RT': 1164.008}, 'float_vector_3': {'RT': 676.8264}, 'id': {'RT': 1.0166}, 'int64_1': {'RT': 1.0443}, 'varchar_1': {'RT': 2.0208}}, 'insert': {'total_time': 611.0126, 'VPS': 8183.1373, 'batch_time': 1.222, 'batch': 10000}, 'flush': {'RT': 3.0245}, 'load': {'RT': 17.6013}, 'Locust': {'Aggregated': {'Requests': 3101, 'Fails': 5, 'RPS': 0.07, 'fail_s': 0.0, 'RT_max': 244809.52, 'RT_avg': 13927.64, 'TP50': 13000.0, 'TP99': 28000.0}, 'load_release': {'Requests': 3101, 'Fails': 5, 'RPS': 0.07, 'fail_s': 0.0, 'RT_max': 244809.52, 'RT_avg': 13927.64, 'TP50': 13000.0, 'TP99': 28000.0}}}}}
/assign @aoiasd /unassign
yanliang567
commented
2 hours ago
Is there an existing issue for this?
Environment
Current Behavior
argo task: multi-vector-corn-1-1730383200 test case name: test_hybrid_search_locust_load_release_replica_cluster
server:
rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeoutload_collection_failed.logclient log:
Expected Behavior
No response
Steps To Reproduce
Milvus Log
No response
Anything else?
test result: