Open wangting0128 opened 5 months ago
/assign @XuanYang-cn /unassign
argo task: multi-vector-corn-1-1717682400 test case name: test_hybrid_search_locust_dml_dql_cluster image: 2.4-20240605-443197bd-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-corn-1-1717682400-4-etcd-0 1/1 Running 0 12h 10.104.23.78 4am-node27 <none> <none>
multi-vector-corn-1-1717682400-4-etcd-1 1/1 Running 0 12h 10.104.27.92 4am-node31 <none> <none>
multi-vector-corn-1-1717682400-4-etcd-2 1/1 Running 0 12h 10.104.25.222 4am-node30 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-datacoord-9bc4947dpvvgp 1/1 Running 5 (12h ago) 12h 10.104.13.183 4am-node16 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-datanode-744fc96dfpnvb2 1/1 Running 5 (12h ago) 12h 10.104.4.203 4am-node11 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-indexcoord-5f5bd45bfggc 1/1 Running 0 12h 10.104.23.39 4am-node27 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-indexnode-cfbff6b9pqt96 1/1 Running 5 (12h ago) 12h 10.104.1.135 4am-node10 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-proxy-6f896db784-ldk47 1/1 Running 5 (12h ago) 12h 10.104.13.184 4am-node16 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-querycoord-788885ctgs44 1/1 Running 5 (12h ago) 12h 10.104.13.185 4am-node16 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-querynode-5ccd885fnwc8h 1/1 Running 5 (12h ago) 12h 10.104.18.9 4am-node25 <none> <none>
multi-vector-corn-1-1717682400-4-milvus-rootcoord-6f6c487cf6pq2 1/1 Running 5 (12h ago) 12h 10.104.4.202 4am-node11 <none> <none>
multi-vector-corn-1-1717682400-4-minio-0 1/1 Running 0 12h 10.104.25.215 4am-node30 <none> <none>
multi-vector-corn-1-1717682400-4-minio-1 1/1 Running 0 12h 10.104.23.77 4am-node27 <none> <none>
multi-vector-corn-1-1717682400-4-minio-2 1/1 Running 0 12h 10.104.15.43 4am-node20 <none> <none>
multi-vector-corn-1-1717682400-4-minio-3 1/1 Running 0 12h 10.104.27.91 4am-node31 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-0 1/1 Running 0 12h 10.104.15.38 4am-node20 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-1 1/1 Running 0 12h 10.104.25.218 4am-node30 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-2 1/1 Running 0 12h 10.104.21.210 4am-node24 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-init-57gl2 0/1 Completed 0 12h 10.104.4.200 4am-node11 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-broker-0 1/1 Running 0 12h 10.104.23.40 4am-node27 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-proxy-0 1/1 Running 0 12h 10.104.13.186 4am-node16 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-pulsar-init-8mbtk 0/1 Completed 0 12h 10.104.4.199 4am-node11 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-recovery-0 1/1 Running 0 12h 10.104.14.241 4am-node18 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.27.81 4am-node31 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.21.217 4am-node24 <none> <none>
multi-vector-corn-1-1717682400-4-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.23.85 4am-node27 <none> <none>
client pod name: multi-vector-corn-1-1717682400-1777966671 client logs:
test result:
[2024-06-07 02:38:11,307 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-06-07 02:38:11,308 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc delete 2152 0(0.00%) | 20745 10 254032 1900 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc flush 2201 3(0.14%) | 53477 281 284799 47000 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc hybrid_search 2253 0(0.00%) | 92728 557 291654 89000 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc insert 2236 0(0.00%) | 23130 157 145139 16000 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc load 2264 0(0.00%) | 34656 7 284603 26000 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc query 2213 0(0.00%) | 77643 210 315524 74000 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: grpc search 2239 0(0.00%) | 84631 16585 237241 81000 | 0.05 0.00 (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-07 02:38:11,308 - INFO - fouram]: Aggregated 15558 3(0.02%) | 55454 7 315524 51000 | 0.36 0.00 (stats.py:789)
[2024-06-07 02:38:11,309 - INFO - fouram]: (stats.py:790)
[2024-06-07 02:38:11,312 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '64Gi'},
'requests': {'cpu': '5.0',
'memory': '33Gi'}},
'replicas': 1},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 1},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': '2.4-20240605-443197bd-amd64'}}},
'host': 'multi-vector-corn-1-1717682400-4-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_dml_dql_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}}},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'insert',
'weight': 1,
'params': {'nb': 1000,
'timeout': 30,
'random_id': True,
'random_vector': True,
'varchar_filled': False,
'start_id': 1000000}},
{'type': 'delete',
'weight': 1,
'params': {'expr': '',
'delete_length': 1000,
'timeout': 30}},
{'type': 'flush',
'weight': 1,
'params': {'timeout': 180}},
{'type': 'load',
'weight': 1,
'params': {'replica_number': 1,
'timeout': 600}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'-1',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'id '
'> '
'-1',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'expr': 'varchar_1 '
'> '
'"1"',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024060625304491,
'datetime': '2024-06-06 14:02:10.758994',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 461.1983,
'float_vector_1': {'RT': 345.9712},
'float_vector_2': {'RT': 263.8506},
'float_vector_3': {'RT': 241.1576},
'id': {'RT': 282.8067},
'int64_1': {'RT': 1.0222},
'varchar_1': {'RT': 1.0199}},
'insert': {'total_time': 158.9733,
'VPS': 6290.3645,
'batch_time': 1.5897,
'batch': 10000},
'flush': {'RT': 2.5245},
'load': {'RT': 7.0519},
'Locust': {'Aggregated': {'Requests': 15558,
'Fails': 3,
'RPS': 0.36,
'fail_s': 0.0,
'RT_max': 315524.01,
'RT_avg': 55454.58,
'TP50': 51000.0,
'TP99': 176000.0},
'delete': {'Requests': 2152,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 254032.74,
'RT_avg': 20745.53,
'TP50': 1900.0,
'TP99': 111000.0},
'flush': {'Requests': 2201,
'Fails': 3,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 284799.27,
'RT_avg': 53477.73,
'TP50': 47000.0,
'TP99': 169000.0},
'hybrid_search': {'Requests': 2253,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 291654.94,
'RT_avg': 92728.76,
'TP50': 89000.0,
'TP99': 208000.0},
'insert': {'Requests': 2236,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 145139.42,
'RT_avg': 23130.14,
'TP50': 16000.0,
'TP99': 107000.0},
'load': {'Requests': 2264,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 284603.66,
'RT_avg': 34656.54,
'TP50': 26000.0,
'TP99': 155000.0},
'query': {'Requests': 2213,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 315524.01,
'RT_avg': 77643.42,
'TP50': 74000.0,
'TP99': 186000.0},
'search': {'Requests': 2239,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 237241.91,
'RT_avg': 84631.22,
'TP50': 81000.0,
'TP99': 170000.0}}}}}
Is there an existing issue for this?
Environment
Current Behavior
argo task: multi-vector-corn-1-1717336800 test case name: test_hybrid_search_locust_dml_dql_cluster
server:
client pod name: multi-vector-corn-1-1717336800-3646130279 client logs: flush timeout
Expected Behavior
No response
Steps To Reproduce
Milvus Log
No response
Anything else?
test result: