yanliang567
commented
3 months ago /assign @zhagnlu /unassign
Open wangting0128 opened 3 months ago
yanliang567
commented
3 months ago /assign @zhagnlu /unassign
wangting0128
commented
3 months ago argo task: multi-vector-corn-1-1718892000 test case name: test_hybrid_search_locust_dql_dml_partition_cluster image: 2.4-20240620-e55fee6b-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-corn-1-1718892000-1-etcd-0 1/1 Running 0 3h10m 10.104.33.254 4am-node36 <none> <none>
multi-vector-corn-1-1718892000-1-etcd-1 1/1 Running 0 3h10m 10.104.18.199 4am-node25 <none> <none>
multi-vector-corn-1-1718892000-1-etcd-2 1/1 Running 0 3h10m 10.104.17.71 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-datacoord-75b698cf5fcmj 1/1 Running 3 (3h9m ago) 3h10m 10.104.17.64 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-datanode-566d86dd4mfc76 1/1 Running 3 (3h9m ago) 3h10m 10.104.17.63 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-indexcoord-74c4b6d5slkk 1/1 Running 0 3h10m 10.104.1.142 4am-node10 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-indexnode-57d68bc78ffg5 1/1 Running 3 (3h9m ago) 3h10m 10.104.20.98 4am-node22 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-indexnode-57d68bc78lgm5 1/1 Running 3 (3h9m ago) 3h10m 10.104.4.92 4am-node11 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-indexnode-57d68bc7f94bf 1/1 Running 3 (3h9m ago) 3h10m 10.104.6.35 4am-node13 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-indexnode-57d68bc7v49qx 1/1 Running 3 (3h9m ago) 3h10m 10.104.30.8 4am-node38 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-proxy-6c5bbcc884-ncdlc 1/1 Running 3 (3h9m ago) 3h10m 10.104.17.62 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-querycoord-6d9c9f4cpq84 1/1 Running 3 (3h9m ago) 3h10m 10.104.1.141 4am-node10 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-querynode-5fc8979c8f7fk 1/1 Running 3 (3h9m ago) 3h10m 10.104.1.143 4am-node10 <none> <none>
multi-vector-corn-1-1718892000-1-milvus-rootcoord-5f445f4ckdh7k 1/1 Running 3 (3h9m ago) 3h10m 10.104.6.34 4am-node13 <none> <none>
multi-vector-corn-1-1718892000-1-minio-0 1/1 Running 0 3h10m 10.104.18.196 4am-node25 <none> <none>
multi-vector-corn-1-1718892000-1-minio-1 1/1 Running 0 3h10m 10.104.17.69 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-minio-2 1/1 Running 0 3h10m 10.104.16.143 4am-node21 <none> <none>
multi-vector-corn-1-1718892000-1-minio-3 1/1 Running 0 3h10m 10.104.33.4 4am-node36 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-bookie-0 1/1 Running 0 3h10m 10.104.33.2 4am-node36 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-bookie-1 1/1 Running 0 3h10m 10.104.18.198 4am-node25 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-bookie-2 1/1 Running 0 3h10m 10.104.16.148 4am-node21 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-bookie-init-pdskm 0/1 Completed 0 3h10m 10.104.17.61 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-broker-0 1/1 Running 0 3h10m 10.104.33.247 4am-node36 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-proxy-0 1/1 Running 0 3h10m 10.104.33.246 4am-node36 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-pulsar-init-8h5rq 0/1 Completed 0 3h10m 10.104.17.60 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-recovery-0 1/1 Running 0 3h10m 10.104.16.139 4am-node21 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-zookeeper-0 1/1 Running 0 3h10m 10.104.18.197 4am-node25 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-zookeeper-1 1/1 Running 0 3h9m 10.104.17.79 4am-node23 <none> <none>
multi-vector-corn-1-1718892000-1-pulsar-zookeeper-2 1/1 Running 0 3h9m 10.104.34.19 4am-node37 <none> <none>client pod name: multi-vector-corn-1-1718892000-3432833946 client logs:
[2024-06-20 17:08:43,971 - ERROR - fouram]: RPC error: [hybrid_search], <MilvusException: (code=503, message=fail to Query on QueryNode 3: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_450597291929633066v1])>, <Time:{'RPC start': '2024-06-20 17:08:43.263132', 'RPC error': '2024-06-20 17:08:43.971716'}> (decorators.py:146)
[2024-06-20 17:08:43,972 - ERROR - fouram]: (api_response) : [Collection.hybrid_search] <MilvusException: (code=503, message=fail to Query on QueryNode 3: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_450597291929633066v1])>, [requestId: bf5e3f56-2f27-11ef-b968-42a579a79d73] (api_request.py:57)
[2024-06-20 17:08:43,972 - ERROR - fouram]: [CheckFunc] hybrid_search request check failed, response:<MilvusException: (code=503, message=fail to Query on QueryNode 3: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_450597291929633066v1])> (func_check.py:48)test steps:
concurrent test and calculation of RT and QPS
:purpose: `DQL & DML(partition)`
verify concurrent DQL & DML(partition) scenario,
which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'float_vector_1': 128dim,
'float_vector_2': 128dim,
'float_vector_3': 128dim,
scalar field: int64_1, varchar_1
2. build indexes:
IVF_FLAT: 'float_vector'
HNSW: 'float_vector_1',
DISKANN: 'float_vector_2'
IVF_SQ8: 'float_vector_3'
INVERTED: 'int64_1', 'varchar_1'
default scalar index: 'id'
3. insert 1 million data into 10 partitions
4. flush collection
5. build indexes again using the same params
6. load collection
replica: 1
7. concurrent request:
- scene_test_partition
(partition: create->insert->flush->index again->load->search->release->search failed->drop)
- search
- hybrid_search
- querytest result:
[2024-06-20 17:12:30,063 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-06-20 17:12:30,063 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-20 17:12:30,063 - INFO - fouram]: grpc hybrid_search 1114 1(0.09%) | 28736 25 158540 27000 | 0.10 0.00 (stats.py:789)
[2024-06-20 17:12:30,063 - INFO - fouram]: grpc query 127 0(0.00%) | 31414 183 161310 16000 | 0.01 0.00 (stats.py:789)
[2024-06-20 17:12:30,064 - INFO - fouram]: grpc scene_test_partition 126 0(0.00%) | 956617 413009 1827396 890000 | 0.01 0.00 (stats.py:789)
[2024-06-20 17:12:30,064 - INFO - fouram]: grpc search 1044 0(0.00%) | 52480 13033 126048 51000 | 0.10 0.00 (stats.py:789)
[2024-06-20 17:12:30,064 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-20 17:12:30,064 - INFO - fouram]: Aggregated 2411 1(0.04%) | 87650 25 1827396 41000 | 0.22 0.00 (stats.py:789)
[2024-06-20 17:12:30,064 - INFO - fouram]: (stats.py:790)
[2024-06-20 17:12:30,072 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0',
'memory': '64Gi'},
'requests': {'cpu': '17.0',
'memory': '33Gi'}},
'replicas': 1},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': '2.4-20240620-e55fee6b-amd64'}}},
'host': 'multi-vector-corn-1-1718892000-1-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_dql_dml_partition_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift'}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift'}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift'}}},
'extra_partitions': {'partitions': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'data_repeated': False},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '3h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_test_partition',
'weight': 1,
'params': {'data_size': 3000,
'ni': 3000,
'nq': 1,
'search_param': {'nprobe': 64},
'limit': 1,
'expr': None,
'output_fields': ['*'],
'guarantee_timestamp': None,
'timeout': 600}},
{'type': 'search',
'weight': 8,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 8,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024062021686923,
'datetime': '2024-06-20 14:02:48.048522',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 132.9929,
'float_vector_1': {'RT': 0.5187},
'float_vector_2': {'RT': 5.5508},
'float_vector_3': {'RT': 1.0194},
'id': {'RT': 0.5129},
'int64_1': {'RT': 0.5132},
'varchar_1': {'RT': 0.5122}},
'insert': {'total_time': 148.5766,
'VPS': 6733.0546,
'batch_time': 1.4858,
'batch': 10000.0},
'flush': {'RT': 2.537},
'load': {'RT': 5.5589},
'Locust': {'Aggregated': {'Requests': 2411,
'Fails': 1,
'RPS': 0.22,
'fail_s': 0.0,
'RT_max': 1827396.76,
'RT_avg': 87650.74,
'TP50': 41000.0,
'TP99': 1240000.0},
'hybrid_search': {'Requests': 1114,
'Fails': 1,
'RPS': 0.1,
'fail_s': 0.0,
'RT_max': 158540.85,
'RT_avg': 28736.84,
'TP50': 27000.0,
'TP99': 94000.0},
'query': {'Requests': 127,
'Fails': 0,
'RPS': 0.01,
'fail_s': 0.0,
'RT_max': 161310.19,
'RT_avg': 31414.55,
'TP50': 16000.0,
'TP99': 140000.0},
'scene_test_partition': {'Requests': 126,
'Fails': 0,
'RPS': 0.01,
'fail_s': 0.0,
'RT_max': 1827396.76,
'RT_avg': 956617.85,
'TP50': 912000.0,
'TP99': 1797000.0},
'search': {'Requests': 1044,
'Fails': 0,
'RPS': 0.1,
'fail_s': 0.0,
'RT_max': 126048.28,
'RT_avg': 52480.46,
'TP50': 51000.0,
'TP99': 95000.0}}}}}argo task: multi-vector-corn-1-1722866400 test case name: test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster image: 2.4-20240805-1b304e98-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-corn-1-1722866400-1-etcd-0 1/1 Running 0 3h23m 10.104.16.160 4am-node21 <none> <none>
multi-vector-corn-1-1722866400-1-etcd-1 1/1 Running 0 3h23m 10.104.23.195 4am-node27 <none> <none>
multi-vector-corn-1-1722866400-1-etcd-2 1/1 Running 0 3h23m 10.104.33.190 4am-node36 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-datacoord-57d64c44xsqfn 1/1 Running 3 (3h22m ago) 3h23m 10.104.17.134 4am-node23 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-datanode-57fcb5cb5cvnww 1/1 Running 3 (3h22m ago) 3h23m 10.104.13.140 4am-node16 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-indexcoord-6dcb697b9768 1/1 Running 0 3h23m 10.104.16.153 4am-node21 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-indexnode-f7565d8fhkhjh 1/1 Running 3 (3h22m ago) 3h23m 10.104.6.187 4am-node13 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-indexnode-f7565d8fjp9xr 1/1 Running 3 (3h22m ago) 3h23m 10.104.9.240 4am-node14 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-indexnode-f7565d8flczhs 1/1 Running 3 (3h22m ago) 3h23m 10.104.23.187 4am-node27 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-indexnode-f7565d8fn598n 1/1 Running 3 (3h22m ago) 3h23m 10.104.16.154 4am-node21 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-proxy-66997d8f5c-j6j28 1/1 Running 3 (3h22m ago) 3h23m 10.104.17.136 4am-node23 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-querycoord-6884458dsqnn 1/1 Running 3 (3h22m ago) 3h23m 10.104.13.138 4am-node16 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-querynode-7d4fcdb42sdtm 1/1 Running 0 3h23m 10.104.18.146 4am-node25 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-querynode-7d4fcdb4rr29j 1/1 Running 3 (3h22m ago) 3h23m 10.104.17.137 4am-node23 <none> <none>
multi-vector-corn-1-1722866400-1-milvus-rootcoord-69848d5clw9m4 1/1 Running 3 (3h22m ago) 3h23m 10.104.13.139 4am-node16 <none> <none>
multi-vector-corn-1-1722866400-1-minio-0 1/1 Running 0 3h23m 10.104.16.159 4am-node21 <none> <none>
multi-vector-corn-1-1722866400-1-minio-1 1/1 Running 0 3h23m 10.104.18.139 4am-node25 <none> <none>
multi-vector-corn-1-1722866400-1-minio-2 1/1 Running 0 3h23m 10.104.33.186 4am-node36 <none> <none>
multi-vector-corn-1-1722866400-1-minio-3 1/1 Running 0 3h23m 10.104.23.199 4am-node27 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-bookie-0 1/1 Running 0 3h23m 10.104.16.161 4am-node21 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-bookie-1 1/1 Running 0 3h23m 10.104.18.140 4am-node25 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-bookie-2 1/1 Running 0 3h23m 10.104.19.7 4am-node28 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-bookie-init-nt7kr 0/1 Completed 0 3h23m 10.104.17.138 4am-node23 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-broker-0 1/1 Running 0 3h23m 10.104.18.133 4am-node25 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-proxy-0 1/1 Running 0 3h23m 10.104.6.186 4am-node13 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-pulsar-init-bn7nc 0/1 Completed 0 3h23m 10.104.17.135 4am-node23 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-recovery-0 1/1 Running 0 3h23m 10.104.23.189 4am-node27 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-zookeeper-0 1/1 Running 0 3h23m 10.104.18.138 4am-node25 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-zookeeper-1 1/1 Running 0 3h22m 10.104.16.163 4am-node21 <none> <none>
multi-vector-corn-1-1722866400-1-pulsar-zookeeper-2 1/1 Running 0 3h21m 10.104.33.193 4am-node36 <none> <none>client log:
test result:
[2024-08-05 20:45:02,810 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-08-05 20:45:02,811 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: grpc hybrid_search 2768 21(0.76%) | 3912 14 42835 2300 | 0.26 0.00 (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: grpc query 336 1(0.30%) | 3497 30 49197 510 | 0.03 0.00 (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: grpc scene_test_partition_hybrid_search 344 0(0.00%) | 380458 116339 727011 383000 | 0.03 0.00 (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: grpc search 2823 21(0.74%) | 25075 676 55330 25000 | 0.26 0.00 (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: Aggregated 6271 43(0.69%) | 34072 14 727011 14000 | 0.58 0.00 (stats.py:789)
[2024-08-05 20:45:02,811 - INFO - fouram]: (stats.py:790)
[2024-08-05 20:45:02,814 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0',
'memory': '32Gi'},
'requests': {'cpu': '17.0',
'memory': '17Gi'}},
'replicas': 2},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': '2.4-20240805-1b304e98-amd64'}}},
'host': 'multi-vector-corn-1-1722866400-1-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift'}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift'}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift'}}},
'extra_partitions': {'partitions': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'data_repeated': False},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '3h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_test_partition_hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 1,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'timeout': 600,
'random_data': True,
'data_size': 3000,
'ni': 3000}},
{'type': 'search',
'weight': 8,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 8,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': ['_default',
'partition_1',
'partition_2',
'partition_3',
'partition_4',
'partition_5',
'partition_6',
'partition_7',
'partition_8',
'partition_9'],
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024080585433568,
'datetime': '2024-08-05 17:22:23.148494',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 101.3387,
'float_vector_1': {'RT': 0.5151},
'float_vector_2': {'RT': 2.5389},
'float_vector_3': {'RT': 0.5197},
'id': {'RT': 0.5186},
'int64_1': {'RT': 0.5196},
'varchar_1': {'RT': 0.52}},
'insert': {'total_time': 157.084,
'VPS': 6368.2859,
'batch_time': 1.5708,
'batch': 10000.0},
'flush': {'RT': 2.5605},
'load': {'RT': 3.95},
'Locust': {'Aggregated': {'Requests': 6271,
'Fails': 43,
'RPS': 0.58,
'fail_s': 0.01,
'RT_max': 727011.75,
'RT_avg': 34072.87,
'TP50': 14000.0,
'TP99': 458000.0},
'hybrid_search': {'Requests': 2768,
'Fails': 21,
'RPS': 0.26,
'fail_s': 0.01,
'RT_max': 42835.57,
'RT_avg': 3912.88,
'TP50': 2300.0,
'TP99': 24000.0},
'query': {'Requests': 336,
'Fails': 1,
'RPS': 0.03,
'fail_s': 0.0,
'RT_max': 49197.71,
'RT_avg': 3497.17,
'TP50': 580.0,
'TP99': 38000.0},
'scene_test_partition_hybrid_search': {'Requests': 344,
'Fails': 0,
'RPS': 0.03,
'fail_s': 0.0,
'RT_max': 727011.75,
'RT_avg': 380458.46,
'TP50': 384000.0,
'TP99': 693000.0},
'search': {'Requests': 2823,
'Fails': 21,
'RPS': 0.26,
'fail_s': 0.01,
'RT_max': 55330.86,
'RT_avg': 25075.22,
'TP50': 25000.0,
'TP99': 48000.0}}}}}argo task: fouramf-85cth image: master-20240828-a52ba3d0-amd64 test case name: test_bitmap_locust_dql_dml_partitions_cluster
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouramf-85cth-72-5605-etcd-0 1/1 Running 0 4h22m 10.104.20.52 4am-node22 <none> <none>
fouramf-85cth-72-5605-milvus-standalone-58f9444cdc-nhgxc 1/1 Running 0 4h22m 10.104.17.13 4am-node23 <none> <none>
fouramf-85cth-72-5605-minio-55d8f6ffc-k4qcx 1/1 Running 0 4h22m 10.104.20.54 4am-node22 <none> <none>client log: fouram_log.log
[2024-08-28 10:59:43,706 - DEBUG - fouram]: (api_request) : [Collection.search] args: [[[0.7237668359122134, 0.648043421129466, 0.4929686714403122, 0.41550353719605704, 0.7535322083862649, 0.3882265314186817, 0.2066266858127187, 0.6265932179570949, 0.777488351135437, 0.5541087949533525, 0.42416196166276654, 0.6035151148137117, 0.10227936496159118, 0.7976156761817611, 0.29368923395619206, 0.9509778787073774, 0.4990785872993516, 0.43167129097108936, 0.4451736754181129, 0.35101497385783387, 0.0611899852998713, 0.3358900179169769, 0.33630915534811856, 0.6335747992361372, 0.9056977071666841, 0.24092222211125436, 0.38195942745525324, 0.5181360770024148, 0.7076355784074545, 0.12141132264171273, 0.39456265840924465, 0.7376221784828476, 0.20506614371283804, 0.9584059342601796, 0.24679315765236542, 0.2860374825130768, 0.5005431069377602, 0.6656913542032278, 0.478018776774619, 0.9475406216172895, 0.057764987061774575, 0.5718156022234668, 0.05082033827823074, 0.26169580043857466, 0.925303933801244, 0.09902992827574508, 0.01187290339267244, 0.20098390467943827, 0.730228149397519, 0.7114744377163259, 0.9277106271987441, 0.8311958861501743, 0.7865491257058267, 0.5146069656988344, 0.40638197384461483, 0.12829032587045652, 0.0645563590751248, 0.48416426806191926, 0.16192089624676465, 0.47914207018107535, 0.46804159366866693, 0.22886164537020914, 0.379486069640923, 0.6378515699026558, 0.1886258518661701, 0.15896358410388134, 0.47206116816090893, 0.4943850049791644, 0.9978413459441872, 0.4826086769246196, 0.0044595201021364606, 0.36059543900413304, 0.3251206179044528, 0.3989551449540315, 0.6098779316869865, 0.2538205678793508, 0.1622851449145376, 0.5491437053975082, 0.12873870858202485, 0.5590444695755187, 0.06168532538710425, 0.4411134123648115, 0.6292100208292175, 0.44342174877311846, 0.7360380290116103, 0.7241368510156941, 0.5068123880802802, 0.7363878325858056, 0.74829639769815, 0.3449570090337192, 0.7915700019607437, 0.9699585142666377, 0.08560089756512523, 0.6398482310194643, 0.3535408226834059, 0.0991351119598265, 0.7842708949087892, 0.6899632829996758, 0.6072112465827695, 0.6447108997861742, 0.8798515472684227, 0.7564549394264041, 0.06512745797086017, 0.7801051730065407, 0.9312686025108594, 0.1435846156551729, 0.45581018472551393, 0.8144166048046874, 0.03643714305829682, 0.9059225633528635, 0.5972409608143253, 0.9938039792355797, 0.8372115858592578, 0.8755872201997471, 0.9446153322365856, 0.4828942988022089, 0.6265868754366474, 0.37482394516757245, 0.49212409093275056, 0.502523318769924, 0.45874470796158806, 0.9402983728648977, 0.0953326646024859, 0.6114380551553108, 0.37416090231460863, 0.6849343527591302, 0.14389896121429435, 0.9959474172933469], [0.8617711906646015, 0.43159586386376914, 0.349434518744565, 0.010054410416265647, 0.6467857748179233, 0.9059050938948503, 0.4753195406361216, 0.06033354154892534, 0.6307731446764382, 0.09112994978146416, 0.4401501130594322, 0.617060992527357, 0.9915670104334737, 0.1520865400880227, 0.9858038489080814, 0.13283776143154236, 0.9015218241704094, 0.06459150560400617, 0.2872810814154567, 0.661805490037......778866361, 0.9308997340257026, 0.4186401144128842, 0.11699929831544076, 0.019660611669384376, 0.9711947870100999], [0.6603006833348785, 0.5903344349057594, 0.18242347160983563, 0.34510040319518986, 0.5459001724318607, 0.8201285151345135, 0.8480956200402998, 0.8555216064383646, 0.6600774204720747, 0.6405828390650233, 0.8237909780719429, 0.6252980242115891, 0.1598743773772202, 0.6441883626438523, 0.8213097651959012, 0.6058424242109802, 0.8075491484003711, 0.5860706008260271, 0.4049646506819248, 0.8778260174488782, 0.5379174985042802, 0.9509919423113531, 0.9260166068677882, 0.8901320646142035, 0.7817020417542672, 0.9154804654456002, 0.09088861609545373, 0.5253783966797207, 0.358728093681423, 0.26903311180975664, 0.6972330930686408, 0.13099728777937725, 0.013716681113807416, 0.6410284460541524, 0.48414659620588296, 0.04221073445658352, 0.6763298149461207, 0.20468027347532358, 0.1824541603692108, 0.004278677227461114, 0.30220238694719215, 0.029829096935202082, 0.016016428195324783, 0.3775678421209432, 0.11316580850222646, 0.7999611720056753, 0.09885888984601976, 0.8253047257036695, 0.29882944004463874, 0.008529276118882878, 0.11747238176040065, 0.8036342891789162, 0.00629168362275867, 0.15065168872407808, 0.44489375631189276, 0.8876815206808192, 0.8879383612738007, 0.8052249206157476, 0.28839340412739856, 0.26459248551514014, 0.5653483290220287, 0.12269317753185494, 0.7513462699910726, 0.6437239140082646, 0.6171550086703265, 0.7862965863455841, 0.8366072050725535, 0.7472957001953385, 0.48742643816857023, 0.5366055350947154, 0.49435439405547354, 0.6922938094585521, 0.3131422740734765, 0.3215926802568324, 0.024389290438680766, 0.914491033415305, 0.8260326675493123, 0.8910245940154549, 0.9016703325706548, 0.029432883489792228, 0.10259074409291402, 0.9304993185876482, 0.9582871911084085, 0.6858113352180506, 0.5896426513331698, 0.4104130354955652, 0.9543306660466944, 0.4290964721411401, 0.10027251234666401, 0.9313843732655509, 0.31730064852568285, 0.5329878611594173, 0.4933910278430399, 0.07333480768635792, 0.4897473000902247, 0.8524555647688156, 0.14531276274701643, 0.9553457767942688, 0.6841320692462827, 0.8250997077389988, 0.36222793153958444, 0.9354370193786335, 0.586187255664986, 0.9530491554640134, 0.8614887430501613, 0.4375948687389515, 0.4283486729520276, 0.39567065577282634, 0.24874959709404565, 0.5154633311136565, 0.2723534907922769, 0.8048810347719334, 0.623966203874654, 0.636062308111971, 0.9692730206721839, 0.2884849836070842, 0.03431540043695258, 0.5607027669067339, 0.01891993807852932, 0.0019292267610200309, 0.6424147979941566, 0.951657953919519, 0.4434847784228676, 0.9768372288956938, 0.9535888688885901, 0.2912969172936094, 0.3016118512690228, 0.36948515651840275]], 'float_vector', {'metric_type': 'L2', 'params': {'nprobe': 16}}, 10, 'int8_1 == 100', ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4', 'partition_5', 'partition_6', 'partition_7', 'partition_8', 'partition_9'], ['id', 'float_vector', 'int64_1'], None, -1], kwargs: {}, [requestId: a19351a6-652c-11ef-ac43-ba470a5fb028] (api_request.py:77)
[2024-08-28 10:59:45,769 - ERROR - fouram]: (api_response) : [Collection.search] <MilvusException: (code=503, message=fail to Query on QueryNode 1: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_8_452155450596262579v8])>, [requestId: a19351a6-652c-11ef-ac43-ba470a5fb028] (api_request.py:57)test steps:
concurrent test and calculation of RT and QPS
:purpose: `primary key: INT64`, divided into 10 partitions
1. building `BITMAP` index on all supported 12 scalar fields
2. 2 fields of different vector types
3. load and search partial partitions & DQL requests
:test steps:
1. create collection with fields:
'float_vector': 128dim
'float_vector_1': 768dim
'id': primary key type is INT64
all scalar fields: varchar max_length=100, array max_capacity=13
2. build indexes:
IVF_SQ8: 'float_vector'
HNSW: 'float_vector_1'
BITMAP: all scalar fields
3. insert 5 million data
4. flush collection
5. build indexes again using the same params
6. load collection
7. concurrent request:
- scene_insert_partition
(partition: create->insert->flush->release->drop)
- scene_test_partition
(partition: create->insert->flush->index again->load->search->release->search failed->drop)
- scene_test_partition_hybrid_search
(partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop)
- search
- query
- hybrid_searchtest result:
[2024-08-28 13:29:25,321 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-08-28 13:29:25,322 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: grpc hybrid_search 246 0(0.00%) | 2177 624 8312 1300 | 0.02 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: grpc query 240 0(0.00%) | 1648 15 11873 280 | 0.02 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: grpc scene_insert_partition 229 0(0.00%) | 46708 3634 233616 40000 | 0.02 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: grpc scene_test_partition 230 0(0.00%) | 667868 43059 1264769 709000 | 0.02 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: grpc scene_test_partition_hybrid_search 210 0(0.00%) | 690265 53355 1260876 715000 | 0.02 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: grpc search 224 1(0.45%) | 2967 0 8799 1900 | 0.02 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: Aggregated 1379 1(0.07%) | 225422 0 1264769 7900 | 0.13 0.00 (stats.py:789)
[2024-08-28 13:29:25,322 - INFO - fouram]: (stats.py:790)
[2024-08-28 13:29:25,327 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'standalone',
'config_name': 'standalone_8c16m',
'config': {'standalone': {'resources': {'limits': {'cpu': '16', 'memory': '64Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}},
'cluster': {'enabled': False},
'etcd': {'replicaCount': 1, 'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
'minio': {'mode': 'standalone', 'metrics': {'podMonitor': {'enabled': True}}},
'pulsar': {'enabled': False},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'indexNode': {'replicas': 2},
'queryNode': {'replicas': 2},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20240828-a52ba3d0-amd64'}}},
'host': 'fouramf-85cth-72-5605-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_bitmap_locust_dql_dml_partitions_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'max_length': 100,
'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
'int16_1': {'index_type': 'BITMAP'},
'int32_1': {'index_type': 'BITMAP'},
'int64_1': {'index_type': 'BITMAP'},
'varchar_1': {'index_type': 'BITMAP'},
'bool_1': {'index_type': 'BITMAP'},
'array_int8_1': {'index_type': 'BITMAP'},
'array_int16_1': {'index_type': 'BITMAP'},
'array_int32_1': {'index_type': 'BITMAP'},
'array_int64_1': {'index_type': 'BITMAP'},
'array_varchar_1': {'index_type': 'BITMAP'},
'array_bool_1': {'index_type': 'BITMAP'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8, 'efConstruction': 200},
'metric_type': 'L2'}},
'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-128, 128],
'max_capacity': 13}}},
'array_int16_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-200, 200],
'max_capacity': 13}}},
'array_int32_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'specify_scope',
'specify_range': [-300, 300],
'max_capacity': 13}}},
'array_int64_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'fixed_value_range',
'specify_range': [-400, 432],
'batch': 50,
'max_capacity': 13}}},
'array_varchar_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-1500, 1500],
'max_capacity': 13}}},
'array_bool_1': {'params': {'max_capacity': 13}},
'int8_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-128, 128],
'max_capacity': 13}}},
'int16_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-200, 200],
'max_capacity': 13}}},
'int32_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'specify_scope',
'specify_range': [-300, 300],
'max_capacity': 13}}},
'int64_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'fixed_value_range',
'specify_range': [-400, 432],
'batch': 50,
'max_capacity': 13}}},
'varchar_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-1500, 1500],
'max_capacity': 13}}}},
'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'data_repeated': False},
'dataset_name': 'sift',
'dataset_size': 5000000,
'ni_per': 5000},
'collection_params': {'other_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
'array_bool_1'],
'shards_num': 16},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False, 'reset_db': False},
'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 30, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_insert_partition',
'weight': 1,
'params': {'data_size': 3000, 'ni': 1000, 'with_flush': True, 'timeout': 600}},
{'type': 'scene_test_partition',
'weight': 1,
'params': {'data_size': 3000,
'ni': 3000,
'nq': 1,
'search_param': {'nprobe': 64},
'limit': 1,
'expr': None,
'output_fields': ['*'],
'guarantee_timestamp': None,
'timeout': 600}},
{'type': 'scene_test_partition_hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 1,
'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
{'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'timeout': 600,
'random_data': True,
'data_size': 3000,
'ni': 3000}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 10,
'search_param': {'nprobe': 16},
'expr': 'int8_1 == 100',
'guarantee_timestamp': None,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'output_fields': ['id', 'float_vector', 'int64_1'],
'ignore_growing': False,
'group_by_field': None,
'timeout': None,
'random_data': True,
'check_task': 'check_search_output',
'check_items': {'nq': 1000}}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 > -1',
'output_fields': ['*'],
'offset': None,
'limit': 10,
'ignore_growing': False,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'timeout': None,
'random_data': False,
'random_count': 0,
'random_range': [0, 1],
'field_name': 'id',
'field_type': 'int64',
'check_task': 'check_query_output',
'check_items': {'expect_length': 10}}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 10,
'top_k': 10,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': '(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, '
'1)) || ((varchar_1 like "1%") and (bool_1 == True))',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, '
'1])'}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'timeout': None,
'random_data': True,
'check_task': 'check_search_output',
'check_items': {'output_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1',
'varchar_1', 'bool_1', 'array_int8_1', 'array_int16_1',
'array_int32_1', 'array_int64_1', 'array_varchar_1',
'array_bool_1', 'id', 'float_vector'],
'nq': 10}}}]},
'run_id': 2024082860324811,
'datetime': '2024-08-28 09:07:12.271400',
'client_version': '2.2'},
'result': {'test_result': {'index': {'RT': 3765.4554,
'float_vector_1': {'RT': 0.519},
'int8_1': {'RT': 0.5173},
'int16_1': {'RT': 0.5249},
'int32_1': {'RT': 0.5187},
'int64_1': {'RT': 0.5209},
'varchar_1': {'RT': 0.5207},
'bool_1': {'RT': 0.521},
'array_int8_1': {'RT': 0.5224},
'array_int16_1': {'RT': 0.5217},
'array_int32_1': {'RT': 0.5204},
'array_int64_1': {'RT': 0.5219},
'array_varchar_1': {'RT': 0.518},
'array_bool_1': {'RT': 0.6875}},
'insert': {'total_time': 663.7768, 'VPS': 7540.4515, 'batch_time': 0.6638, 'batch': 5000.0},
'flush': {'RT': 5.2698},
'load': {'RT': 9.1158},
'Locust': {'Aggregated': {'Requests': 1379,
'Fails': 1,
'RPS': 0.13,
'fail_s': 0.0,
'RT_max': 1264769.2,
'RT_avg': 225422.68,
'TP50': 7900.0,
'TP99': 1230000.0},
'hybrid_search': {'Requests': 246,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 8312.08,
'RT_avg': 2177.23,
'TP50': 1300.0,
'TP99': 8000.0},
'query': {'Requests': 240,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 11873.5,
'RT_avg': 1648.34,
'TP50': 280.0,
'TP99': 11000.0},
'scene_insert_partition': {'Requests': 229,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 233616.65,
'RT_avg': 46708.28,
'TP50': 40000.0,
'TP99': 128000.0},
'scene_test_partition': {'Requests': 230,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 1264769.2,
'RT_avg': 667868.79,
'TP50': 710000.0,
'TP99': 1260000.0},
'scene_test_partition_hybrid_search': {'Requests': 210,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 1260876.86,
'RT_avg': 690265.85,
'TP50': 715000.0,
'TP99': 1260000.0},
'search': {'Requests': 224,
'Fails': 1,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 8799.57,
'RT_avg': 2967.99,
'TP50': 1900.0,
'TP99': 8400.0}}}}}argo task: fouramf-hkh54 test case name: test_bitmap_locust_dql_dml_partitions_cluster image: master-20240906-7859faf8-amd64
server:
[2024-09-07 05:41:14,625 - INFO - fouram]: [Base] Deploy initial state:
I0907 02:22:51.423609 2040 request.go:665] Waited for 1.17858041s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/helm.toolkit.fluxcd.io/v2beta2?timeout=32s
I0907 02:23:01.423819 2040 request.go:665] Waited for 3.997740261s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/events.k8s.io/v1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouramf-hkh54-78-5249-etcd-0 1/1 Running 0 4m52s 10.104.21.168 4am-node24 <none> <none>
fouramf-hkh54-78-5249-etcd-1 1/1 Running 0 4m52s 10.104.17.145 4am-node23 <none> <none>
fouramf-hkh54-78-5249-etcd-2 1/1 Running 0 4m52s 10.104.32.215 4am-node39 <none> <none>
fouramf-hkh54-78-5249-milvus-datanode-86d747c85c-nc4w6 1/1 Running 2 (4m29s ago) 4m53s 10.104.13.184 4am-node16 <none> <none>
fouramf-hkh54-78-5249-milvus-indexnode-595c56c756-glhrf 1/1 Running 2 (4m30s ago) 4m53s 10.104.9.241 4am-node14 <none> <none>
fouramf-hkh54-78-5249-milvus-indexnode-595c56c756-mfrg4 1/1 Running 1 (4m28s ago) 4m53s 10.104.6.151 4am-node13 <none> <none>
fouramf-hkh54-78-5249-milvus-mixcoord-688df4bc59-ssbrt 1/1 Running 2 (4m30s ago) 4m53s 10.104.9.240 4am-node14 <none> <none>
fouramf-hkh54-78-5249-milvus-proxy-b866bc67c-vmph6 1/1 Running 2 (4m29s ago) 4m53s 10.104.13.187 4am-node16 <none> <none>
fouramf-hkh54-78-5249-milvus-querynode-6dccfc6c86-4pkzh 1/1 Running 1 (4m29s ago) 4m53s 10.104.5.183 4am-node12 <none> <none>
fouramf-hkh54-78-5249-milvus-querynode-6dccfc6c86-rrrrf 1/1 Running 2 (4m30s ago) 4m53s 10.104.4.187 4am-node11 <none> <none>
fouramf-hkh54-78-5249-minio-0 1/1 Running 0 4m53s 10.104.21.167 4am-node24 <none> <none>
fouramf-hkh54-78-5249-minio-1 1/1 Running 0 4m52s 10.104.17.143 4am-node23 <none> <none>
fouramf-hkh54-78-5249-minio-2 1/1 Running 0 4m52s 10.104.32.214 4am-node39 <none> <none>
fouramf-hkh54-78-5249-minio-3 1/1 Running 0 4m52s 10.104.18.12 4am-node25 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-0 1/1 Running 0 4m52s 10.104.21.169 4am-node24 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-1 1/1 Running 0 4m52s 10.104.17.147 4am-node23 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-2 1/1 Running 0 4m51s 10.104.30.73 4am-node38 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-init-dtm7v 0/1 Completed 0 4m53s 10.104.13.188 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-broker-0 1/1 Running 0 4m53s 10.104.13.189 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-proxy-0 1/1 Running 0 4m53s 10.104.13.186 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-pulsar-init-7wxjk 0/1 Completed 0 4m53s 10.104.21.160 4am-node24 <none> <none>
fouramf-hkh54-78-5249-pulsar-recovery-0 1/1 Running 0 4m53s 10.104.13.185 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-zookeeper-0 1/1 Running 0 4m53s 10.104.21.166 4am-node24 <none> <none>
fouramf-hkh54-78-5249-pulsar-zookeeper-1 1/1 Running 0 4m9s 10.104.17.149 4am-node23 <none> <none>
fouramf-hkh54-78-5249-pulsar-zookeeper-2 1/1 Running 0 3m32s 10.104.18.14 4am-node25 <none> <none> (base.py:261)
[2024-09-07 05:41:14,625 - INFO - fouram]: [Cmd Exe] kubectl get pods -n qa-milvus -o wide | grep -E 'NAME|fouramf-hkh54-78-5249-milvus|fouramf-hkh54-78-5249-minio|fouramf-hkh54-78-5249-etcd|fouramf-hkh54-78-5249-pulsar|fouramf-hkh54-78-5249-zookeeper|fouramf-hkh54-78-5249-kafka|fouramf-hkh54-78-5249-log|fouramf-hkh54-78-5249-tikv' (util_cmd.py:14)
[2024-09-07 05:41:35,210 - INFO - fouram]: [CliClient] pod details of release(fouramf-hkh54-78-5249):
I0907 05:41:15.922517 2113 request.go:665] Waited for 1.169743062s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/policy/v1beta1?timeout=32s
I0907 05:41:25.922736 2113 request.go:665] Waited for 3.997497169s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/source.toolkit.fluxcd.io/v1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouramf-hkh54-78-5249-etcd-0 1/1 Running 0 3h23m 10.104.21.168 4am-node24 <none> <none>
fouramf-hkh54-78-5249-etcd-1 1/1 Running 0 3h23m 10.104.17.145 4am-node23 <none> <none>
fouramf-hkh54-78-5249-etcd-2 1/1 Running 0 3h23m 10.104.32.215 4am-node39 <none> <none>
fouramf-hkh54-78-5249-milvus-datanode-86d747c85c-nc4w6 1/1 Running 2 (3h22m ago) 3h23m 10.104.13.184 4am-node16 <none> <none>
fouramf-hkh54-78-5249-milvus-indexnode-595c56c756-glhrf 1/1 Running 2 (3h22m ago) 3h23m 10.104.9.241 4am-node14 <none> <none>
fouramf-hkh54-78-5249-milvus-indexnode-595c56c756-mfrg4 1/1 Running 1 (3h22m ago) 3h23m 10.104.6.151 4am-node13 <none> <none>
fouramf-hkh54-78-5249-milvus-mixcoord-688df4bc59-ssbrt 1/1 Running 2 (3h22m ago) 3h23m 10.104.9.240 4am-node14 <none> <none>
fouramf-hkh54-78-5249-milvus-proxy-b866bc67c-vmph6 1/1 Running 2 (3h22m ago) 3h23m 10.104.13.187 4am-node16 <none> <none>
fouramf-hkh54-78-5249-milvus-querynode-6dccfc6c86-4pkzh 1/1 Running 1 (3h22m ago) 3h23m 10.104.5.183 4am-node12 <none> <none>
fouramf-hkh54-78-5249-milvus-querynode-6dccfc6c86-rrrrf 1/1 Running 2 (3h22m ago) 3h23m 10.104.4.187 4am-node11 <none> <none>
fouramf-hkh54-78-5249-minio-0 1/1 Running 0 3h23m 10.104.21.167 4am-node24 <none> <none>
fouramf-hkh54-78-5249-minio-1 1/1 Running 0 3h23m 10.104.17.143 4am-node23 <none> <none>
fouramf-hkh54-78-5249-minio-2 1/1 Running 0 3h23m 10.104.32.214 4am-node39 <none> <none>
fouramf-hkh54-78-5249-minio-3 1/1 Running 0 3h23m 10.104.18.12 4am-node25 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-0 1/1 Running 0 3h23m 10.104.21.169 4am-node24 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-1 1/1 Running 0 3h23m 10.104.17.147 4am-node23 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-2 1/1 Running 0 3h23m 10.104.30.73 4am-node38 <none> <none>
fouramf-hkh54-78-5249-pulsar-bookie-init-dtm7v 0/1 Completed 0 3h23m 10.104.13.188 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-broker-0 1/1 Running 0 3h23m 10.104.13.189 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-proxy-0 1/1 Running 0 3h23m 10.104.13.186 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-pulsar-init-7wxjk 0/1 Completed 0 3h23m 10.104.21.160 4am-node24 <none> <none>
fouramf-hkh54-78-5249-pulsar-recovery-0 1/1 Running 0 3h23m 10.104.13.185 4am-node16 <none> <none>
fouramf-hkh54-78-5249-pulsar-zookeeper-0 1/1 Running 0 3h23m 10.104.21.166 4am-node24 <none> <none>
fouramf-hkh54-78-5249-pulsar-zookeeper-1 1/1 Running 0 3h22m 10.104.17.149 4am-node23 <none> <none>
fouramf-hkh54-78-5249-pulsar-zookeeper-2 1/1 Running 0 3h21m 10.104.18.14 4am-node25 <none> <none>client log: client.log
test result:
[2024-09-07 05:40:51,039 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-09-07 05:40:51,040 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: grpc hybrid_search 260 1(0.38%) | 560 0 3073 470 | 0.02 0.00 (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: grpc query 255 0(0.00%) | 85 15 1731 20 | 0.02 0.00 (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: grpc scene_insert_partition 272 30(11.03%) | 71031 3715 216002 45000 | 0.03 0.00 (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: grpc scene_test_partition 233 25(10.73%) | 283044 19183 863920 213000 | 0.02 0.00 (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: grpc scene_test_partition_hybrid_search 234 4(1.71%) | 309433 15188 873586 328000 | 0.02 0.00 (stats.py:789)
[2024-09-07 05:40:51,040 - INFO - fouram]: grpc search 235 0(0.00%) | 1254 884 3120 1100 | 0.02 0.00 (stats.py:789)
[2024-09-07 05:40:51,041 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-09-07 05:40:51,041 - INFO - fouram]: Aggregated 1489 60(4.03%) | 106205 0 873586 2500 | 0.14 0.01 (stats.py:789)
[2024-09-07 05:40:51,041 - INFO - fouram]: (stats.py:790)
[2024-09-07 05:40:51,045 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_8c16m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '16.0', 'memory': '32Gi'}, 'requests': {'cpu': '9.0', 'memory': '17Gi'}}, 'replicas': 2},
'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 2},
'dataNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '16Gi'}, 'requests': {'cpu': '5.0', 'memory': '9Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20240906-7859faf8-amd64'}}},
'host': 'fouramf-hkh54-78-5249-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_bitmap_locust_dql_dml_partitions_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'max_length': 100,
'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
'int16_1': {'index_type': 'BITMAP'},
'int32_1': {'index_type': 'BITMAP'},
'int64_1': {'index_type': 'BITMAP'},
'varchar_1': {'index_type': 'BITMAP'},
'bool_1': {'index_type': 'BITMAP'},
'array_int8_1': {'index_type': 'BITMAP'},
'array_int16_1': {'index_type': 'BITMAP'},
'array_int32_1': {'index_type': 'BITMAP'},
'array_int64_1': {'index_type': 'BITMAP'},
'array_varchar_1': {'index_type': 'BITMAP'},
'array_bool_1': {'index_type': 'BITMAP'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8, 'efConstruction': 200},
'metric_type': 'L2'}},
'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-128, 128],
'max_capacity': 13}}},
'array_int16_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-200, 200],
'max_capacity': 13}}},
'array_int32_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'specify_scope',
'specify_range': [-300, 300],
'max_capacity': 13}}},
'array_int64_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'fixed_value_range',
'specify_range': [-400, 432],
'batch': 50,
'max_capacity': 13}}},
'array_varchar_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-1500, 1500],
'max_capacity': 13}}},
'array_bool_1': {'params': {'max_capacity': 13}},
'int8_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-128, 128],
'max_capacity': 13}}},
'int16_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-200, 200],
'max_capacity': 13}}},
'int32_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'specify_scope',
'specify_range': [-300, 300],
'max_capacity': 13}}},
'int64_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'fixed_value_range',
'specify_range': [-400, 432],
'batch': 50,
'max_capacity': 13}}},
'varchar_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-1500, 1500],
'max_capacity': 13}}}},
'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'data_repeated': False},
'dataset_name': 'sift',
'dataset_size': 5000000,
'ni_per': 5000},
'collection_params': {'other_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
'array_bool_1'],
'shards_num': 16},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False, 'reset_db': False},
'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 15, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_insert_partition',
'weight': 1,
'params': {'data_size': 3000, 'ni': 1000, 'with_flush': True, 'timeout': 600}},
{'type': 'scene_test_partition',
'weight': 1,
'params': {'data_size': 3000,
'ni': 3000,
'nq': 1,
'search_param': {'nprobe': 64},
'limit': 1,
'expr': None,
'output_fields': ['*'],
'guarantee_timestamp': None,
'timeout': 600}},
{'type': 'scene_test_partition_hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 1,
'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
{'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'timeout': 600,
'random_data': True,
'data_size': 3000,
'ni': 3000}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 10,
'search_param': {'nprobe': 16},
'expr': 'int8_1 == 100',
'guarantee_timestamp': None,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'output_fields': ['id', 'float_vector', 'int64_1'],
'ignore_growing': False,
'group_by_field': None,
'timeout': 30,
'random_data': True,
'check_task': 'check_search_output',
'check_items': {'nq': 1000}}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 > -1',
'output_fields': ['*'],
'offset': None,
'limit': 10,
'ignore_growing': False,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'timeout': 30,
'consistency_level': None,
'random_data': False,
'random_count': 0,
'random_range': [0, 1],
'field_name': 'id',
'field_type': 'int64',
'check_task': 'check_query_output',
'check_items': {'expect_length': 10}}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 10,
'top_k': 10,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': '(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, '
'1)) || ((varchar_1 like "1%") and (bool_1 == True))',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, '
'1])'}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'timeout': 60,
'random_data': True,
'check_task': 'check_search_output',
'check_items': {'output_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1',
'varchar_1', 'bool_1', 'array_int8_1', 'array_int16_1',
'array_int32_1', 'array_int64_1', 'array_varchar_1',
'array_bool_1', 'id', 'float_vector'],
'nq': 10}}}]},
'run_id': 2024090754936377,
'datetime': '2024-09-07 02:18:13.808144',
'client_version': '2.2'},
'result': {'test_result': {'index': {'RT': 207.0157,
'float_vector_1': {'RT': 0.5206},
'int8_1': {'RT': 0.5196},
'int16_1': {'RT': 0.5197},
'int32_1': {'RT': 0.5195},
'int64_1': {'RT': 0.5219},
'varchar_1': {'RT': 0.5195},
'bool_1': {'RT': 0.5197},
'array_int8_1': {'RT': 0.5202},
'array_int16_1': {'RT': 0.5193},
'array_int32_1': {'RT': 0.5189},
'array_int64_1': {'RT': 0.5209},
'array_varchar_1': {'RT': 0.5195},
'array_bool_1': {'RT': 0.5211}},
'insert': {'total_time': 659.2684, 'VPS': 7587.5656, 'batch_time': 0.6593, 'batch': 5000.0},
'flush': {'RT': 5.7398},
'load': {'RT': 8.1003},
'Locust': {'Aggregated': {'Requests': 1489,
'Fails': 60,
'RPS': 0.14,
'fail_s': 0.04,
'RT_max': 873586.68,
'RT_avg': 106205.32,
'TP50': 2500.0,
'TP99': 768000.0},
'hybrid_search': {'Requests': 260,
'Fails': 1,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 3073.78,
'RT_avg': 560.7,
'TP50': 470.0,
'TP99': 2200.0},
'query': {'Requests': 255,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 1731.48,
'RT_avg': 85.3,
'TP50': 20,
'TP99': 1300.0},
'scene_insert_partition': {'Requests': 272,
'Fails': 30,
'RPS': 0.03,
'fail_s': 0.11,
'RT_max': 216002.66,
'RT_avg': 71031.44,
'TP50': 45000.0,
'TP99': 213000.0},
'scene_test_partition': {'Requests': 233,
'Fails': 25,
'RPS': 0.02,
'fail_s': 0.11,
'RT_max': 863920.34,
'RT_avg': 283044.0,
'TP50': 213000.0,
'TP99': 835000.0},
'scene_test_partition_hybrid_search': {'Requests': 234,
'Fails': 4,
'RPS': 0.02,
'fail_s': 0.02,
'RT_max': 873586.68,
'RT_avg': 309433.85,
'TP50': 336000.0,
'TP99': 864000.0},
'search': {'Requests': 235,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 3120.8,
'RT_avg': 1254.73,
'TP50': 1100.0,
'TP99': 2800.0}}}}} @weiliu1031 Please help take a look
wangting0128
commented
1 week ago argo task：multi-vector-corn-1-1725890400 test case name: test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster image: 2.4-20240909-14fe3213-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-corn-1-1725890400-1-etcd-0 1/1 Running 0 3h10m 10.104.19.3 4am-node28 <none> <none>
multi-vector-corn-1-1725890400-1-etcd-1 1/1 Running 0 3h10m 10.104.15.183 4am-node20 <none> <none>
multi-vector-corn-1-1725890400-1-etcd-2 1/1 Running 0 3h10m 10.104.16.232 4am-node21 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-datanode-6c6dcc7667mkcn 1/1 Running 2 (3h10m ago) 3h10m 10.104.32.151 4am-node39 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-indexnode-86bc67bb2wg9d 1/1 Running 2 (3h9m ago) 3h10m 10.104.20.128 4am-node22 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-indexnode-86bc67bbnhqn2 1/1 Running 2 (3h10m ago) 3h10m 10.104.32.152 4am-node39 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-indexnode-86bc67bbq6vfh 1/1 Running 2 (3h10m ago) 3h10m 10.104.25.167 4am-node30 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-indexnode-86bc67bbsbm6m 1/1 Running 2 (3h9m ago) 3h10m 10.104.6.155 4am-node13 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-mixcoord-657c576d6vnqft 1/1 Running 2 (3h10m ago) 3h10m 10.104.6.156 4am-node13 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-proxy-5df578d89b-q799k 1/1 Running 2 (3h9m ago) 3h10m 10.104.4.219 4am-node11 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-querynode-6cf8747fjzbzd 1/1 Running 2 (3h10m ago) 3h10m 10.104.6.157 4am-node13 <none> <none>
multi-vector-corn-1-1725890400-1-milvus-querynode-6cf8747fkcgdj 1/1 Running 2 (3h9m ago) 3h10m 10.104.4.220 4am-node11 <none> <none>
multi-vector-corn-1-1725890400-1-minio-0 1/1 Running 0 3h10m 10.104.15.181 4am-node20 <none> <none>
multi-vector-corn-1-1725890400-1-minio-1 1/1 Running 0 3h10m 10.104.19.2 4am-node28 <none> <none>
multi-vector-corn-1-1725890400-1-minio-2 1/1 Running 0 3h10m 10.104.16.229 4am-node21 <none> <none>
multi-vector-corn-1-1725890400-1-minio-3 1/1 Running 0 3h10m 10.104.21.158 4am-node24 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-bookie-0 1/1 Running 0 3h10m 10.104.15.182 4am-node20 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-bookie-1 1/1 Running 0 3h10m 10.104.19.4 4am-node28 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-bookie-2 1/1 Running 0 3h10m 10.104.16.230 4am-node21 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-bookie-init-nrd9h 0/1 Completed 0 3h10m 10.104.19.248 4am-node28 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-broker-0 1/1 Running 0 3h10m 10.104.34.218 4am-node37 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-proxy-0 1/1 Running 0 3h10m 10.104.19.247 4am-node28 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-pulsar-init-gghj7 0/1 Completed 0 3h10m 10.104.34.217 4am-node37 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-recovery-0 1/1 Running 0 3h10m 10.104.9.40 4am-node14 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-zookeeper-0 1/1 Running 0 3h10m 10.104.19.254 4am-node28 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-zookeeper-1 1/1 Running 0 3h9m 10.104.30.150 4am-node38 <none> <none>
multi-vector-corn-1-1725890400-1-pulsar-zookeeper-2 1/1 Running 0 3h9m 10.104.15.185 4am-node20 <none> <none>client log:
[2024-09-09 20:26:39,843 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: grpc hybrid_search 2704 0(0.00%) | 4171 20 59788 2500 | 0.40 0.00 (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: grpc query 335 0(0.00%) | 4333 35 85852 430 | 0.10 0.00 (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: grpc scene_test_partition_hybrid_search 330 0(0.00%) | 393927 160338 690339 378000 | 0.10 0.00 (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: grpc search 2604 0(0.00%) | 26572 2884 61882 26000 | 0.40 0.00 (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: Aggregated 5973 0(0.00%) | 35480 20 690339 15000 | 1.00 0.00 (stats.py:789)
[2024-09-09 20:26:39,843 - INFO - fouram]: (stats.py:790)
[2024-09-09 20:26:39,845 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-09-09 20:26:39,845 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-09-09 20:26:39,845 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-09-09 20:26:39,845 - INFO - fouram]: grpc hybrid_search 2500 3400 4400 5300 9800 15000 24000 27000 59000 60000 60000 2704 (stats.py:819)
[2024-09-09 20:26:39,845 - INFO - fouram]: grpc query 430 1100 1600 2600 17000 27000 34000 48000 86000 86000 86000 335 (stats.py:819)
[2024-09-09 20:26:39,846 - INFO - fouram]: grpc scene_test_partition_hybrid_search 379000 433000 470000 491000 533000 566000 612000 625000 690000 690000 690000 330 (stats.py:819)
[2024-09-09 20:26:39,846 - INFO - fouram]: grpc search 26000 29000 30000 31000 35000 42000 50000 52000 61000 62000 62000 2604 (stats.py:819)
[2024-09-09 20:26:39,846 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-09-09 20:26:39,846 - INFO - fouram]: Aggregated 15000 24000 27000 29000 35000 280000 427000 497000 617000 690000 690000 5973 (stats.py:819)
[2024-09-09 20:26:39,846 - INFO - fouram]: (stats.py:820)
[2024-09-09 20:26:47,577 - ERROR - fouram]: RPC error: [hybrid_search], <MilvusException: (code=503, message=fail to Query on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])>, <Time:{'RPC start': '2024-09-09 20:26:35.698007', 'RPC error': '2024-09-09 20:26:47.577248'}> (decorators.py:146)
[2024-09-09 20:26:47,577 - ERROR - fouram]: (api_response) : [Collection.hybrid_search] <MilvusException: (code=503, message=fail to Query on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])>, [requestId: cf59cbf8-6ee9-11ef-ad39-aaa779003951] (api_request.py:57)
[2024-09-09 20:26:47,577 - ERROR - fouram]: [CheckFunc] hybrid_search request check failed, response:<MilvusException: (code=503, message=fail to Query on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])> (func_check.py:101)
[2024-09-09 20:26:48,375 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=503, message=fail to search on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])>, <Time:{'RPC start': '2024-09-09 20:26:47.663868', 'RPC error': '2024-09-09 20:26:48.375514'}> (decorators.py:146)
[2024-09-09 20:26:48,376 - ERROR - fouram]: (api_response) : [Collection.search] <MilvusException: (code=503, message=fail to search on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])>, [requestId: d670cc16-6ee9-11ef-ad39-aaa779003951] (api_request.py:57)
[2024-09-09 20:26:48,376 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=503, message=fail to search on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])> (func_check.py:101)
[2024-09-09 20:26:49,277 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=503, message=fail to search on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])>, <Time:{'RPC start': '2024-09-09 20:26:48.457082', 'RPC error': '2024-09-09 20:26:49.277828'}> (decorators.py:146)
[2024-09-09 20:26:49,278 - ERROR - fouram]: (api_response) : [Collection.search] <MilvusException: (code=503, message=fail to search on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])>, [requestId: d6ea9d5c-6ee9-11ef-ad39-aaa779003951] (api_request.py:57)
[2024-09-09 20:26:49,278 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=503, message=fail to search on QueryNode 2: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_1_452434948645454155v1])> (func_check.py:101)
[2024-09-09 20:27:00,119 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: grpc hybrid_search 2706 1(0.04%) | 4173 20 59788 2500 | 0.20 0.10 (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: grpc query 336 0(0.00%) | 4320 35 85852 430 | 0.10 0.00 (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: grpc scene_test_partition_hybrid_search 330 0(0.00%) | 393927 160338 690339 378000 | 0.00 0.00 (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: grpc search 2612 2(0.08%) | 26556 783 61882 26000 | 0.70 0.20 (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: Aggregated 5984 3(0.05%) | 35445 20 690339 15000 | 1.00 0.30 (stats.py:789)
[2024-09-09 20:27:00,120 - INFO - fouram]: (stats.py:790)
[2024-09-09 20:27:00,121 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: grpc hybrid_search 2500 3400 4400 5300 9800 15000 24000 27000 59000 60000 60000 2706 (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: grpc query 430 1100 1600 2400 17000 27000 34000 48000 86000 86000 86000 336 (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: grpc scene_test_partition_hybrid_search 379000 433000 470000 491000 533000 566000 612000 625000 690000 690000 690000 330 (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: grpc search 26000 28000 30000 31000 35000 42000 50000 52000 61000 62000 62000 2612 (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-09-09 20:27:00,122 - INFO - fouram]: Aggregated 15000 24000 27000 29000 35000 280000 427000 497000 617000 690000 690000 5984 (stats.py:819)@weiliu1031 Please help take a look
Is there an existing issue for this?
Environment
Current Behavior
argo task: multi-vector-corn-1-1717336800 test case name: test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster
server:
client pod name: multi-vector-corn-1-1717336800-161976730 client logs:
hybrid search request raises error
Expected Behavior
No response
Steps To Reproduce
Milvus Log
No response
Anything else?
test result: