yanliang567
commented
8 months ago /assign @MrPresent-Han /unassign
Open wangting0128 opened 8 months ago
yanliang567
commented
8 months ago /assign @MrPresent-Han /unassign
MrPresent-Han
commented
8 months ago rootcoord describe collection timeout, cannot figure out root casue for lacking some logs
wangting0128
commented
8 months ago argo task: multi-vector-scene-mix2-dtpjw test case name: test_hybrid_search_locust_ddl_dql_search_cluster image: master-20240203-20c9cfc5
server:
[2024-02-04 15:02:39,748 - INFO - fouram]: [Base] Deploy initial state:
I0204 02:56:29.747873 397 request.go:665] Waited for 1.158596217s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/discovery.k8s.io/v1beta1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-dtpjw-1-etcd-0 1/1 Running 0 7m42s 10.104.16.153 4am-node21 <none> <none>
multi-vector-scene-mix2-dtpjw-1-etcd-1 1/1 Running 0 7m41s 10.104.32.200 4am-node39 <none> <none>
multi-vector-scene-mix2-dtpjw-1-etcd-2 1/1 Running 0 7m41s 10.104.30.104 4am-node38 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-datacoord-748cbb48457mmd 1/1 Running 1 (3m13s ago) 7m44s 10.104.25.44 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-datanode-568f7d9b544qg9z 1/1 Running 1 (3m13s ago) 7m44s 10.104.17.147 4am-node23 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexcoord-7d5dfdf7qmnjr 1/1 Running 0 7m44s 10.104.17.146 4am-node23 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856frzmr 1/1 Running 1 (3m13s ago) 7m43s 10.104.23.89 4am-node27 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856hltl2 1/1 Running 1 (3m13s ago) 7m43s 10.104.25.47 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856lwndx 1/1 Running 1 (3m13s ago) 7m44s 10.104.13.122 4am-node16 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856m7mmr 1/1 Running 1 (3m12s ago) 7m43s 10.104.14.228 4am-node18 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-proxy-77855cdfb4-l4tx5 1/1 Running 1 (3m13s ago) 7m44s 10.104.25.45 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-querycoord-5f54bdc6ww6t7 1/1 Running 1 (3m13s ago) 7m44s 10.104.25.46 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-querynode-f99b9ccbfrwnk7 1/1 Running 1 (3m13s ago) 7m43s 10.104.33.24 4am-node36 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-querynode-f99b9ccbfz7v2w 1/1 Running 0 7m43s 10.104.31.117 4am-node34 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-rootcoord-6586c5fd4xb4f7 1/1 Running 1 (3m13s ago) 7m43s 10.104.31.116 4am-node34 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-0 1/1 Running 0 7m43s 10.104.18.199 4am-node25 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-1 1/1 Running 0 7m42s 10.104.16.154 4am-node21 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-2 1/1 Running 0 7m42s 10.104.30.95 4am-node38 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-3 1/1 Running 0 7m41s 10.104.32.201 4am-node39 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-0 1/1 Running 0 7m42s 10.104.18.202 4am-node25 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-1 1/1 Running 0 7m42s 10.104.16.157 4am-node21 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-2 1/1 Running 0 7m41s 10.104.30.105 4am-node38 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-init-pq7vh 0/1 Completed 0 7m44s 10.104.14.227 4am-node18 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-broker-0 1/1 Running 0 7m43s 10.104.4.182 4am-node11 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-proxy-0 1/1 Running 0 7m42s 10.104.12.191 4am-node17 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-pulsar-init-7l8bl 0/1 Completed 0 7m44s 10.104.14.226 4am-node18 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-recovery-0 1/1 Running 0 7m43s 10.104.12.189 4am-node17 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-zookeeper-0 1/1 Running 0 7m43s 10.104.18.200 4am-node25 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-zookeeper-1 1/1 Running 0 4m37s 10.104.32.214 4am-node39 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-zookeeper-2 1/1 Running 0 3m10s 10.104.21.5 4am-node24 <none> <none> (base.py:257)
[2024-02-04 15:02:39,748 - INFO - fouram]: [Cmd Exe] kubectl get pods -n qa-milvus -o wide | grep -E 'STATUS|multi-vector-scene-mix2-dtpjw-1-milvus|multi-vector-scene-mix2-dtpjw-1-minio|multi-vector-scene-mix2-dtpjw-1-etcd|multi-vector-scene-mix2-dtpjw-1-pulsar|multi-vector-scene-mix2-dtpjw-1-kafka|multi-vector-scene-mix2-dtpjw-1-log|multi-vector-scene-mix2-dtpjw-1-tikv' (util_cmd.py:14)
[2024-02-04 15:02:49,849 - INFO - fouram]: [CliClient] pod details of release(multi-vector-scene-mix2-dtpjw-1):
I0204 15:02:41.011203 540 request.go:665] Waited for 1.160408613s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/networking.k8s.io/v1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-dtpjw-1-etcd-0 1/1 Running 0 12h 10.104.16.153 4am-node21 <none> <none>
multi-vector-scene-mix2-dtpjw-1-etcd-1 1/1 Running 0 12h 10.104.32.200 4am-node39 <none> <none>
multi-vector-scene-mix2-dtpjw-1-etcd-2 1/1 Running 0 12h 10.104.30.104 4am-node38 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-datacoord-748cbb48457mmd 1/1 Running 1 (12h ago) 12h 10.104.25.44 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-datanode-568f7d9b544qg9z 1/1 Running 1 (12h ago) 12h 10.104.17.147 4am-node23 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexcoord-7d5dfdf7qmnjr 1/1 Running 0 12h 10.104.17.146 4am-node23 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856frzmr 1/1 Running 1 (12h ago) 12h 10.104.23.89 4am-node27 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856hltl2 1/1 Running 1 (12h ago) 12h 10.104.25.47 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856lwndx 1/1 Running 1 (12h ago) 12h 10.104.13.122 4am-node16 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-indexnode-7f76b9856m7mmr 1/1 Running 1 (12h ago) 12h 10.104.14.228 4am-node18 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-proxy-77855cdfb4-l4tx5 1/1 Running 1 (12h ago) 12h 10.104.25.45 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-querycoord-5f54bdc6ww6t7 1/1 Running 1 (12h ago) 12h 10.104.25.46 4am-node30 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-querynode-f99b9ccbfrwnk7 1/1 Running 1 (12h ago) 12h 10.104.33.24 4am-node36 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-querynode-f99b9ccbfz7v2w 1/1 Running 0 12h 10.104.31.117 4am-node34 <none> <none>
multi-vector-scene-mix2-dtpjw-1-milvus-rootcoord-6586c5fd4xb4f7 1/1 Running 1 (12h ago) 12h 10.104.31.116 4am-node34 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-0 1/1 Running 0 12h 10.104.18.199 4am-node25 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-1 1/1 Running 0 12h 10.104.16.154 4am-node21 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-2 1/1 Running 0 12h 10.104.30.95 4am-node38 <none> <none>
multi-vector-scene-mix2-dtpjw-1-minio-3 1/1 Running 0 12h 10.104.32.201 4am-node39 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-0 1/1 Running 0 12h 10.104.18.202 4am-node25 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-1 1/1 Running 0 12h 10.104.16.157 4am-node21 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-2 1/1 Running 0 12h 10.104.30.105 4am-node38 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-bookie-init-pq7vh 0/1 Completed 0 12h 10.104.14.227 4am-node18 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-broker-0 1/1 Running 0 12h 10.104.4.182 4am-node11 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-proxy-0 1/1 Running 0 12h 10.104.12.191 4am-node17 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-pulsar-init-7l8bl 0/1 Completed 0 12h 10.104.14.226 4am-node18 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-recovery-0 1/1 Running 0 12h 10.104.12.189 4am-node17 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.18.200 4am-node25 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.32.214 4am-node39 <none> <none>
multi-vector-scene-mix2-dtpjw-1-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.21.5 4am-node24 <none> <none>client pod name: multi-vector-scene-mix2-dtpjw-2157039955 client log:
test steps:
concurrent test and calculation of RT and QPS
:purpose: `DDL & DQL`
verify DDL & DQL scenario,
which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'float_vector_1': 128dim,
'float_vector_2': 128dim,
'float_vector_3': 128dim,
scalar field: int64_1, varchar_1
2. build indexes:
IVF_FLAT: 'float_vector'
HNSW: 'float_vector_1',
DISKANN: 'float_vector_2'
IVF_SQ8: 'float_vector_3'
INVERTED: 'int64_1', 'varchar_1'
default scalar index: 'id'
3. insert 1m data
4. flush collection
5. build indexes again using the same params
6. load collection
replica: 1
7. concurrent request:
- scene_search_test
(collection: create->insert->flush->index->load->search->drop)
- search
- hybrid_search
- querytest result:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0',
'memory': '32Gi'},
'requests': {'cpu': '17.0',
'memory': '17Gi'}},
'replicas': 2},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': 'master-20240203-20c9cfc5'}}},
'host': 'multi-vector-scene-mix2-dtpjw-1-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_ddl_dql_search_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}}},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_search_test',
'weight': 1,
'params': {'dataset': 'local',
'dim': 128,
'shards_num': 2,
'data_size': 3000,
'nb': 3000,
'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2',
'other_fields': [],
'replica_number': 1,
'nq': 1,
'top_k': 10,
'search_param': {'nprobe': 16},
'search_counts': 1,
'scalars_params': {},
'scalars_index': {},
'vectors_index': {},
'prepare_before_insert': False,
'new_connect': False,
'new_user': False}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'100000',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'id '
'< '
'900000',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'expr': 'varchar_1 '
'> '
'"1"',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024020449309036,
'datetime': '2024-02-04 02:48:50.616172',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 89.1384,
'float_vector_1': {'RT': 0.5156},
'float_vector_2': {'RT': 1.5281},
'float_vector_3': {'RT': 0.5157},
'id': {'RT': 0.521},
'int64_1': {'RT': 0.5474},
'varchar_1': {'RT': 0.5132}},
'insert': {'total_time': 136.2817,
'VPS': 7337.7423,
'batch_time': 1.3628,
'batch': 10000},
'flush': {'RT': 2.5235},
'load': {'RT': 7.0497},
'Locust': {'Aggregated': {'Requests': 130409,
'Fails': 1,
'RPS': 3.02,
'fail_s': 0.0,
'RT_max': 114593.6,
'RT_avg': 6598.9,
'TP50': 390.0,
'TP99': 55000.0},
'hybrid_search': {'Requests': 32814,
'Fails': 0,
'RPS': 0.76,
'fail_s': 0.0,
'RT_max': 3593.29,
'RT_avg': 269.99,
'TP50': 220.0,
'TP99': 820.0},
'query': {'Requests': 32425,
'Fails': 0,
'RPS': 0.75,
'fail_s': 0.0,
'RT_max': 14801.76,
'RT_avg': 258.18,
'TP50': 200.0,
'TP99': 950.0},
'scene_search_test': {'Requests': 32453,
'Fails': 1,
'RPS': 0.75,
'fail_s': 0.0,
'RT_max': 114593.6,
'RT_avg': 25472.7,
'TP50': 22000.0,
'TP99': 68000.0},
'search': {'Requests': 32717,
'Fails': 0,
'RPS': 0.76,
'fail_s': 0.0,
'RT_max': 18323.86,
'RT_avg': 509.21,
'TP50': 430.0,
'TP99': 1100.0}}}}}@MrPresent-Han
HuaJieHappy
commented
7 months ago When will the issue be fixed = =，I meet same problem
wangting0128
commented
7 months ago argo task: multi-vector-scene-mix2-vjwzc test case name: test_hybrid_search_locust_ddl_dql_search_cluster image: master-20240227-7f78e9d4-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-vjwzc-1-etcd-0 1/1 Running 0 12h 10.104.24.249 4am-node29 <none> <none>
multi-vector-scene-mix2-vjwzc-1-etcd-1 1/1 Running 0 12h 10.104.29.45 4am-node35 <none> <none>
multi-vector-scene-mix2-vjwzc-1-etcd-2 1/1 Running 0 12h 10.104.31.194 4am-node34 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-datacoord-57f5c698c7z778 1/1 Running 0 12h 10.104.14.111 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-datanode-6fcb4f7c7-5xj22 1/1 Running 1 (12h ago) 12h 10.104.1.175 4am-node10 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-indexcoord-6c7b6d66bmlzk 1/1 Running 0 12h 10.104.14.115 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-indexnode-9c4f86d4468zwk 1/1 Running 0 12h 10.104.24.242 4am-node29 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-indexnode-9c4f86d44nl698 1/1 Running 0 12h 10.104.12.198 4am-node17 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-indexnode-9c4f86d44rz2m8 1/1 Running 0 12h 10.104.26.93 4am-node32 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-indexnode-9c4f86d44tj727 1/1 Running 0 12h 10.104.4.132 4am-node11 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-proxy-764bb74555-zqdf9 1/1 Running 1 (12h ago) 12h 10.104.14.116 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-querycoord-7cd696f5h2lrz 1/1 Running 1 (12h ago) 12h 10.104.14.113 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-querynode-6c884b886p8tb9 1/1 Running 0 12h 10.104.25.142 4am-node30 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-querynode-6c884b886qr68g 1/1 Running 0 12h 10.104.5.228 4am-node12 <none> <none>
multi-vector-scene-mix2-vjwzc-1-milvus-rootcoord-6c7b86bc5t5njx 1/1 Running 1 (12h ago) 12h 10.104.14.110 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-minio-0 1/1 Running 0 12h 10.104.31.190 4am-node34 <none> <none>
multi-vector-scene-mix2-vjwzc-1-minio-1 1/1 Running 0 12h 10.104.29.46 4am-node35 <none> <none>
multi-vector-scene-mix2-vjwzc-1-minio-2 1/1 Running 0 12h 10.104.34.187 4am-node37 <none> <none>
multi-vector-scene-mix2-vjwzc-1-minio-3 1/1 Running 0 12h 10.104.23.234 4am-node27 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-bookie-0 1/1 Running 0 12h 10.104.29.47 4am-node35 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-bookie-1 1/1 Running 0 12h 10.104.31.195 4am-node34 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-bookie-2 1/1 Running 0 12h 10.104.23.246 4am-node27 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-bookie-init-zrhcf 0/1 Completed 0 12h 10.104.14.118 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-broker-0 1/1 Running 0 12h 10.104.14.117 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-proxy-0 1/1 Running 0 12h 10.104.5.227 4am-node12 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-pulsar-init-lh7m7 0/1 Completed 0 12h 10.104.14.114 4am-node18 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-recovery-0 1/1 Running 0 12h 10.104.20.5 4am-node22 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.20.17 4am-node22 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.29.49 4am-node35 <none> <none>
multi-vector-scene-mix2-vjwzc-1-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.26.123 4am-node32 <none> <none> Common labels: {"app":"milvus","cluster":"4am","component":"proxy","container":"proxy","job":"qa-milvus/milvus","namespace":"qa-milvus","node_name":"4am-node18","pod":"multi-vector-scene-mix2-vjwzc-1-milvus-proxy-764bb74555-zqdf9"}
Line limit: "5000 (30 returned)"
Total bytes processed: "2.89 GB"
2024-02-27 15:47:51.607 Welcome to use Milvus!
2024-02-27 15:52:14.705 Welcome to use Milvus!
2024-02-27 17:56:36.909 [2024/02/27 09:56:36.909 +00:00] [DEBUG] [proxy/impl.go:764] ["DescribeCollection received"] [traceID=fcbafb4d61218a0bc345ab6a1b7c5494] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My]
2024-02-27 17:56:36.909 [2024/02/27 09:56:36.909 +00:00] [DEBUG] [proxy/impl.go:777] ["DescribeCollection enqueued"] [traceID=fcbafb4d61218a0bc345ab6a1b7c5494] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [BeginTS=448011382784524298] [EndTS=448011382784524298]
2024-02-27 17:56:36.910 [2024/02/27 09:56:36.910 +00:00] [DEBUG] [proxy/impl.go:795] ["DescribeCollection done"] [traceID=fcbafb4d61218a0bc345ab6a1b7c5494] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [BeginTS=448011382784524298] [EndTS=448011382784524298] [db=default] [collection=scene_search_test_eMSi91My]
2024-02-27 17:56:36.911 [2024/02/27 09:56:36.911 +00:00] [DEBUG] [proxy/impl.go:388] ["CreateCollection received"] [traceID=9e5b9f999077695a0423d687818f3dc7] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [len(schema)=68] [shards_num=2] [consistency_level=Bounded]
2024-02-27 17:56:36.912 [2024/02/27 09:56:36.912 +00:00] [DEBUG] [proxy/impl.go:399] ["CreateCollection enqueued"] [traceID=9e5b9f999077695a0423d687818f3dc7] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [len(schema)=68] [shards_num=2] [consistency_level=Bounded] [BeginTs=448011382784524303] [EndTs=448011382784524303] [timestamp=448011382784524303]
2024-02-27 17:56:36.917 [2024/02/27 09:56:36.917 +00:00] [INFO] [proxy/impl.go:114] ["received request to invalidate collection meta cache"] [traceID=9e5b9f999077695a0423d687818f3dc7] [module=Proxy] [role=proxy] [db=default] [collectionName=scene_search_test_eMSi91My] [collectionID=0] [msgType=DropCollection] [partitionName=]
2024-02-27 17:56:36.917 [2024/02/27 09:56:36.917 +00:00] [INFO] [proxy/meta_cache.go:875] ["clearing shard cache for collection"] [collectionName=scene_search_test_eMSi91My]
2024-02-27 17:56:36.917 [2024/02/27 09:56:36.917 +00:00] [INFO] [proxy/impl.go:130] ["complete to invalidate collection meta cache with collection name"] [traceID=9e5b9f999077695a0423d687818f3dc7] [module=Proxy] [role=proxy] [db=default] [collectionName=scene_search_test_eMSi91My] [collectionID=0] [msgType=DropCollection] [partitionName=] [collectionName=scene_search_test_eMSi91My]
2024-02-27 17:56:36.917 [2024/02/27 09:56:36.917 +00:00] [INFO] [proxy/impl.go:163] ["complete to invalidate collection meta cache"] [traceID=9e5b9f999077695a0423d687818f3dc7] [module=Proxy] [role=proxy] [db=default] [collectionName=scene_search_test_eMSi91My] [collectionID=0] [msgType=DropCollection] [partitionName=]
2024-02-27 17:56:36.927 [2024/02/27 09:56:36.927 +00:00] [DEBUG] [proxy/impl.go:417] ["CreateCollection done"] [traceID=9e5b9f999077695a0423d687818f3dc7] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [len(schema)=68] [shards_num=2] [consistency_level=Bounded] [BeginTs=448011382784524303] [EndTs=448011382784524303]
2024-02-27 17:56:38.086 [2024/02/27 09:56:38.086 +00:00] [INFO] [proxy/meta_cache.go:378] ["meta update success"] [database=default] [collectionName=scene_search_test_eMSi91My] [collectionID=448009426460608327]
2024-02-27 17:56:38.086 [2024/02/27 09:56:38.086 +00:00] [DEBUG] [proxy/impl.go:2358] ["Enqueue insert request in Proxy"] [traceID=8c964b67a8956a9a987fa6caf7953037] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [partition=] [len(FieldsData)=2] [len(HashKeys)=0] [NumRows=3000]
2024-02-27 17:56:38.087 [2024/02/27 09:56:38.087 +00:00] [DEBUG] [proxy/impl.go:2367] ["Detail of insert request in Proxy"] [traceID=8c964b67a8956a9a987fa6caf7953037] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [partition=] [len(FieldsData)=2] [len(HashKeys)=0] [NumRows=3000]
2024-02-27 17:56:38.089 [2024/02/27 09:56:38.089 +00:00] [DEBUG] [proxy/task_insert.go:207] ["Proxy Insert PreExecute done"] [traceID=8c964b67a8956a9a987fa6caf7953037] [collectionName=scene_search_test_eMSi91My]
2024-02-27 17:56:43.099 [2024/02/27 09:56:43.099 +00:00] [WARN] [allocator/cached_allocator.go:258] ["Allocator has some reqs to fail"] [Role=SegmentIDAllocator] [reqLen=1]
2024-02-27 17:56:43.099 [2024/02/27 09:56:43.099 +00:00] [ERROR] [proxy/msg_pack.go:132] ["allocate segmentID for insert data failed"] [collectionName=scene_search_test_eMSi91My] [channelName=by-dev-rootcoord-dml_7_448009426460608327v0] ["allocate count"=1503] [error="getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error"] [stack="github.com/milvus-io/milvus/internal/proxy.repackInsertDataByPartition\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/msg_pack.go:132\ngithub.com/milvus-io/milvus/internal/proxy.repackInsertData\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/msg_pack.go:196\ngithub.com/milvus-io/milvus/internal/proxy.(*insertTask).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_insert.go:252\ngithub.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:466\ngithub.com/milvus-io/milvus/internal/proxy.(*taskScheduler).manipulationLoop.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:526\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1\n\t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67"]
2024-02-27 17:56:43.099 [2024/02/27 09:56:43.099 +00:00] [WARN] [proxy/msg_pack.go:198] ["repack insert data to msg pack failed"] [collectionName=scene_search_test_eMSi91My] ["partition name"=_default] [error="getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error"]
2024-02-27 17:56:43.099 [2024/02/27 09:56:43.099 +00:00] [WARN] [proxy/task_insert.go:257] ["assign segmentID and repack insert data failed"] [traceID=8c964b67a8956a9a987fa6caf7953037] [error="getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error"]
2024-02-27 17:56:43.099 [2024/02/27 09:56:43.099 +00:00] [WARN] [proxy/task_scheduler.go:469] ["Failed to execute task: "] [traceID=8c964b67a8956a9a987fa6caf7953037] [error="getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error"]
2024-02-27 17:56:43.099 [2024/02/27 09:56:43.099 +00:00] [WARN] [proxy/impl.go:2370] ["Failed to execute insert task in task scheduler: getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error"] [traceID=8c964b67a8956a9a987fa6caf7953037] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [partition=] [len(FieldsData)=2] [len(HashKeys)=0] [NumRows=3000]
2024-02-28 04:04:54.146 [2024/02/27 20:04:54.146 +00:00] [INFO] [proxy/impl.go:467] ["DropCollection received"] [traceID=c4b776d3b84dadf04b65d1bda208f186] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My]
2024-02-28 04:04:54.146 [2024/02/27 20:04:54.146 +00:00] [DEBUG] [proxy/impl.go:477] ["DropCollection enqueued"] [traceID=c4b776d3b84dadf04b65d1bda208f186] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [BeginTs=448020950319628291] [EndTs=448020950319628291]
2024-02-28 04:04:54.147 [2024/02/27 20:04:54.147 +00:00] [INFO] [proxy/impl.go:114] ["received request to invalidate collection meta cache"] [traceID=c4b776d3b84dadf04b65d1bda208f186] [module=Proxy] [role=proxy] [db=default] [collectionName=scene_search_test_eMSi91My] [collectionID=448009426460608327] [msgType=DropCollection] [partitionName=]
2024-02-28 04:04:54.147 [2024/02/27 20:04:54.147 +00:00] [INFO] [proxy/meta_cache.go:875] ["clearing shard cache for collection"] [collectionName=scene_search_test_eMSi91My]
2024-02-28 04:04:54.147 [2024/02/27 20:04:54.147 +00:00] [INFO] [proxy/impl.go:130] ["complete to invalidate collection meta cache with collection name"] [traceID=c4b776d3b84dadf04b65d1bda208f186] [module=Proxy] [role=proxy] [db=default] [collectionName=scene_search_test_eMSi91My] [collectionID=448009426460608327] [msgType=DropCollection] [partitionName=] [collectionName=scene_search_test_eMSi91My]
2024-02-28 04:04:54.148 [2024/02/27 20:04:54.148 +00:00] [INFO] [proxy/impl.go:163] ["complete to invalidate collection meta cache"] [traceID=c4b776d3b84dadf04b65d1bda208f186] [module=Proxy] [role=proxy] [db=default] [collectionName=scene_search_test_eMSi91My] [collectionID=448009426460608327] [msgType=DropCollection] [partitionName=]
2024-02-28 04:04:54.149 [2024/02/27 20:04:54.149 +00:00] [INFO] [proxy/impl.go:493] ["DropCollection done"] [traceID=c4b776d3b84dadf04b65d1bda208f186] [role=proxy] [db=default] [collection=scene_search_test_eMSi91My] [BeginTs=448020950319628291] [EndTs=448020950319628291]
2024-02-28 04:05:22.727 [2024/02/27 20:05:22.727 +00:00] [ERROR] [config/refresher.go:73] ["can not pull configs"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically\n\t/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:73"]
client pod name: multi-vector-scene-mix2-vjwzc-290373602 client log:
[2024-02-27 09:56:26,994 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: grpc hybrid_search 9290 0(0.00%) | 407 107 3070 350 | 1.50 0.00 (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: grpc query 9229 0(0.00%) | 326 93 5640 290 | 2.20 0.00 (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: grpc scene_search_test 9180 0(0.00%) | 13268 7609 109288 12000 | 1.30 0.00 (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: grpc search 9327 0(0.00%) | 724 387 8142 640 | 1.30 0.00 (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: Aggregated 37026 0(0.00%) | 3655 93 109288 540 | 6.30 0.00 (stats.py:789)
[2024-02-27 09:56:26,995 - INFO - fouram]: (stats.py:790)
[2024-02-27 09:56:26,996 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: grpc hybrid_search 350 450 520 580 740 900 1200 1300 1800 3100 3100 9290 (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: grpc query 290 370 420 460 570 680 810 920 1500 5600 5600 9229 (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: grpc scene_search_test 12000 13000 14000 14000 16000 19000 39000 54000 78000 109000 109000 9180 (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: grpc search 640 750 840 900 1100 1300 1500 1700 7000 8100 8100 9327 (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: Aggregated 540 810 1900 9900 12000 14000 17000 20000 67000 94000 109000 37026 (stats.py:819)
[2024-02-27 09:56:26,996 - INFO - fouram]: (stats.py:820)
[2024-02-27 09:56:42,913 - ERROR - fouram]: RPC error: [batch_insert], <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, <Time:{'RPC start': '2024-02-27 09:56:37.812552', 'RPC error': '2024-02-27 09:56:42.912936'}> (decorators.py:134)
[2024-02-27 09:56:42,914 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, [requestId: 7f7ebd1e-d556-11ee-a180-de005efb052a] (api_request.py:57)
[2024-02-27 09:56:42,914 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)> (func_check.py:54)
[2024-02-27 09:56:42,915 - ERROR - fouram]: [func_time_catch] : (api_request.py:125)
[2024-02-27 09:56:46,997 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-02-27 09:56:46,997 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-02-27 09:56:46,997 - INFO - fouram]: grpc hybrid_search 9300 0(0.00%) | 407 107 3070 350 | 0.00 0.00 (stats.py:789)
[2024-02-27 09:56:46,997 - INFO - fouram]: grpc query 9240 0(0.00%) | 327 93 5640 290 | 0.20 0.00 (stats.py:789)
[2024-02-27 09:56:46,998 - INFO - fouram]: grpc scene_search_test 9199 1(0.01%) | 13279 6194 109288 12000 | 0.70 0.10 (stats.py:789)
[2024-02-27 09:56:46,998 - INFO - fouram]: grpc search 9339 0(0.00%) | 723 387 8142 640 | 0.70 0.00 (stats.py:789)
[2024-02-27 09:56:46,998 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-02-27 09:56:46,998 - INFO - fouram]: Aggregated 37078 1(0.00%) | 3660 93 109288 540 | 1.60 0.10 (stats.py:789)
[2024-02-27 09:56:46,998 - INFO - fouram]: (stats.py:790)
[2024-02-27 09:56:46,998 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-02-27 09:56:46,998 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: grpc hybrid_search 350 450 520 580 740 900 1200 1300 1800 3100 3100 9300 (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: grpc query 290 370 420 460 570 680 810 920 1900 5600 5600 9240 (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: grpc scene_search_test 12000 13000 14000 15000 16000 19000 39000 54000 78000 109000 109000 9199 (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: grpc search 640 750 840 900 1100 1300 1500 1700 7000 8100 8100 9339 (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-02-27 09:56:46,999 - INFO - fouram]: Aggregated 540 810 1900 9900 12000 14000 17000 20000 67000 94000 109000 37078 (stats.py:819)test steps:
concurrent test and calculation of RT and QPS
:purpose: `DDL & DQL`
verify DDL & DQL scenario,
which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'float_vector_1': 128dim,
'float_vector_2': 128dim,
'float_vector_3': 128dim,
scalar field: int64_1, varchar_1
2. build indexes:
IVF_FLAT: 'float_vector'
HNSW: 'float_vector_1',
DISKANN: 'float_vector_2'
IVF_SQ8: 'float_vector_3'
INVERTED: 'int64_1', 'varchar_1'
default scalar index: 'id'
3. insert 1m data
4. flush collection
5. build indexes again using the same params
6. load collection
replica: 1
7. concurrent request:
- scene_search_test
(collection: create->insert->flush->index->load->search->drop) <- insert raises error
- search
- hybrid_search
- querytest result:
[2024-02-27 20:03:16,141 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-02-27 20:03:16,141 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-02-27 20:03:16,141 - INFO - fouram]: grpc hybrid_search 32994 0(0.00%) | 355 107 3070 280 | 0.76 0.00 (stats.py:789)
[2024-02-27 20:03:16,141 - INFO - fouram]: grpc query 32865 0(0.00%) | 283 93 5640 230 | 0.76 0.00 (stats.py:789)
[2024-02-27 20:03:16,142 - INFO - fouram]: grpc scene_search_test 32948 1(0.00%) | 24855 6194 118121 21000 | 0.76 0.00 (stats.py:789)
[2024-02-27 20:03:16,142 - INFO - fouram]: grpc search 32962 0(0.00%) | 674 368 8142 580 | 0.76 0.00 (stats.py:789)
[2024-02-27 20:03:16,142 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-02-27 20:03:16,142 - INFO - fouram]: Aggregated 131769 1(0.00%) | 6543 93 118121 500 | 3.05 0.00 (stats.py:789)
[2024-02-27 20:03:16,142 - INFO - fouram]: (stats.py:790)
[2024-02-27 20:03:16,146 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0',
'memory': '32Gi'},
'requests': {'cpu': '17.0',
'memory': '17Gi'}},
'replicas': 2},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': 'master-20240227-7f78e9d4-amd64'}}},
'host': 'multi-vector-scene-mix2-vjwzc-1-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_ddl_dql_search_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}}},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_search_test',
'weight': 1,
'params': {'dataset': 'local',
'dim': 128,
'shards_num': 2,
'data_size': 3000,
'nb': 3000,
'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2',
'other_fields': [],
'replica_number': 1,
'nq': 1,
'top_k': 10,
'search_param': {'nprobe': 16},
'search_counts': 1,
'scalars_params': {},
'scalars_index': {},
'vectors_index': {},
'prepare_before_insert': False,
'new_connect': False,
'new_user': False}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'100000',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'id '
'< '
'900000',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'expr': 'varchar_1 '
'> '
'"1"',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024022700608367,
'datetime': '2024-02-27 07:47:40.700081',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 50.3476,
'float_vector_1': {'RT': 36.7551},
'float_vector_2': {'RT': 147.9569},
'float_vector_3': {'RT': 53.3526},
'id': {'RT': 0.5685},
'int64_1': {'RT': 0.5133},
'varchar_1': {'RT': 0.5142}},
'insert': {'total_time': 124.8424,
'VPS': 8010.0991,
'batch_time': 1.2484,
'batch': 10000},
'flush': {'RT': 3.5351},
'load': {'RT': 9.6711},
'Locust': {'Aggregated': {'Requests': 131769,
'Fails': 1,
'RPS': 3.05,
'fail_s': 0.0,
'RT_max': 118121.38,
'RT_avg': 6543.47,
'TP50': 500.0,
'TP99': 55000.0},
'hybrid_search': {'Requests': 32994,
'Fails': 0,
'RPS': 0.76,
'fail_s': 0.0,
'RT_max': 3070.8,
'RT_avg': 355.81,
'TP50': 280.0,
'TP99': 1300.0},
'query': {'Requests': 32865,
'Fails': 0,
'RPS': 0.76,
'fail_s': 0.0,
'RT_max': 5640.89,
'RT_avg': 283.39,
'TP50': 230.0,
'TP99': 880.0},
'scene_search_test': {'Requests': 32948,
'Fails': 1,
'RPS': 0.76,
'fail_s': 0.0,
'RT_max': 118121.38,
'RT_avg': 24855.89,
'TP50': 21000.0,
'TP99': 67000.0},
'search': {'Requests': 32962,
'Fails': 0,
'RPS': 0.76,
'fail_s': 0.0,
'RT_max': 8142.49,
'RT_avg': 674.17,
'TP50': 580.0,
'TP99': 1700.0}}}}}@MrPresent-Han
wangting0128
commented
7 months ago argo task: multi-vector-scene-mix2-zdc55 test case name: test_hybrid_search_locust_ddl_dql_search_cluster image: master-20240313-5220005a-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-zdc55-1-etcd-0 1/1 Running 1 (12h ago) 12h 10.104.19.6 4am-node28 <none> <none>
multi-vector-scene-mix2-zdc55-1-etcd-1 1/1 Running 1 (12h ago) 12h 10.104.31.7 4am-node34 <none> <none>
multi-vector-scene-mix2-zdc55-1-etcd-2 1/1 Running 0 12h 10.104.23.225 4am-node27 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-datacoord-98764bc44gngcp 1/1 Running 0 12h 10.104.6.157 4am-node13 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-datanode-f6774479f-7wdtv 1/1 Running 0 12h 10.104.30.203 4am-node38 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-indexcoord-69d8dc66wmrvb 1/1 Running 0 12h 10.104.30.207 4am-node38 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-indexnode-7984dff4c5pvnb 1/1 Running 0 12h 10.104.32.240 4am-node39 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-indexnode-7984dff4c8jv2f 1/1 Running 0 12h 10.104.23.220 4am-node27 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-indexnode-7984dff4c8x9cc 1/1 Running 0 12h 10.104.14.86 4am-node18 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-indexnode-7984dff4ckgf84 1/1 Running 0 12h 10.104.4.237 4am-node11 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-proxy-859c4bbb69-kgxqm 1/1 Running 0 12h 10.104.30.205 4am-node38 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-querycoord-6b8bc66bd6h4h 1/1 Running 0 12h 10.104.30.206 4am-node38 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-querynode-6566d4978bc2k7 1/1 Running 0 12h 10.104.13.175 4am-node16 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-querynode-6566d4978t84jh 1/1 Running 0 12h 10.104.6.160 4am-node13 <none> <none>
multi-vector-scene-mix2-zdc55-1-milvus-rootcoord-776944d8-ncshj 1/1 Running 0 12h 10.104.30.204 4am-node38 <none> <none>
multi-vector-scene-mix2-zdc55-1-minio-0 1/1 Running 0 12h 10.104.32.241 4am-node39 <none> <none>
multi-vector-scene-mix2-zdc55-1-minio-1 1/1 Running 0 12h 10.104.18.241 4am-node25 <none> <none>
multi-vector-scene-mix2-zdc55-1-minio-2 1/1 Running 0 12h 10.104.19.5 4am-node28 <none> <none>
multi-vector-scene-mix2-zdc55-1-minio-3 1/1 Running 0 12h 10.104.29.66 4am-node35 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-bookie-0 1/1 Running 0 12h 10.104.31.12 4am-node34 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-bookie-1 1/1 Running 0 12h 10.104.18.246 4am-node25 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-bookie-2 1/1 Running 0 12h 10.104.29.70 4am-node35 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-bookie-init-fsptq 0/1 Completed 0 12h 10.104.19.248 4am-node28 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-broker-0 1/1 Running 0 12h 10.104.27.67 4am-node31 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-proxy-0 1/1 Running 0 12h 10.104.19.250 4am-node28 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-pulsar-init-fsrtk 0/1 Completed 0 12h 10.104.19.253 4am-node28 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-recovery-0 1/1 Running 0 12h 10.104.19.3 4am-node28 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.31.9 4am-node34 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.34.59 4am-node37 <none> <none>
multi-vector-scene-mix2-zdc55-1-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.33.46 4am-node36 <none> <none> client pod name: multi-vector-scene-mix2-zdc55-903548361 client log:
test steps:
concurrent test and calculation of RT and QPS
:purpose: `DDL & DQL`
verify DDL & DQL scenario,
which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'float_vector_1': 128dim,
'float_vector_2': 128dim,
'float_vector_3': 128dim,
scalar field: int64_1, varchar_1
2. build indexes:
IVF_FLAT: 'float_vector'
HNSW: 'float_vector_1',
DISKANN: 'float_vector_2'
IVF_SQ8: 'float_vector_3'
INVERTED: 'int64_1', 'varchar_1'
default scalar index: 'id'
3. insert 1 million data
4. flush collection
5. build indexes again using the same params
6. load collection
replica: 1
7. concurrent request:
- scene_search_test
(collection: create->insert->flush->index->load->search->drop)
- search
- hybrid_search
- querytest result:
[2024-03-13 23:30:31,587 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-03-13 23:30:31,588 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: grpc hybrid_search 33824 0(0.00%) | 461 137 8274 390 | 0.78 0.00 (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: grpc query 33686 0(0.00%) | 318 84 17270 250 | 0.78 0.00 (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: grpc scene_search_test 33835 1(0.00%) | 23891 7477 112787 21000 | 0.78 0.00 (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: grpc search 33969 0(0.00%) | 818 405 18179 690 | 0.79 0.00 (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: Aggregated 135314 1(0.00%) | 6374 84 112787 600 | 3.13 0.00 (stats.py:789)
[2024-03-13 23:30:31,588 - INFO - fouram]: (stats.py:790)
[2024-03-13 23:30:31,592 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0',
'memory': '32Gi'},
'requests': {'cpu': '17.0',
'memory': '17Gi'}},
'replicas': 2},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': 'master-20240313-5220005a-amd64'}}},
'host': 'multi-vector-scene-mix2-zdc55-1-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_ddl_dql_search_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}}},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_search_test',
'weight': 1,
'params': {'dataset': 'local',
'dim': 128,
'shards_num': 2,
'data_size': 3000,
'nb': 3000,
'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2',
'other_fields': [],
'replica_number': 1,
'nq': 1,
'top_k': 10,
'search_param': {'nprobe': 16},
'search_counts': 1,
'scalars_params': {},
'scalars_index': {},
'vectors_index': {},
'prepare_before_insert': False,
'new_connect': False,
'new_user': False}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'100000',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'id '
'< '
'900000',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'expr': 'varchar_1 '
'> '
'"1"',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024031385932876,
'datetime': '2024-03-13 11:16:33.956089',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 83.5938,
'float_vector_1': {'RT': 0.516},
'float_vector_2': {'RT': 44.3243},
'float_vector_3': {'RT': 78.0581},
'id': {'RT': 0.5159},
'int64_1': {'RT': 0.5154},
'varchar_1': {'RT': 0.5145}},
'insert': {'total_time': 128.7422,
'VPS': 7767.4609,
'batch_time': 1.2874,
'batch': 10000},
'flush': {'RT': 3.0247},
'load': {'RT': 145.7088},
'Locust': {'Aggregated': {'Requests': 135314,
'Fails': 1,
'RPS': 3.13,
'fail_s': 0.0,
'RT_max': 112787.13,
'RT_avg': 6374.16,
'TP50': 600.0,
'TP99': 49000.0},
'hybrid_search': {'Requests': 33824,
'Fails': 0,
'RPS': 0.78,
'fail_s': 0.0,
'RT_max': 8274.12,
'RT_avg': 461.87,
'TP50': 390.0,
'TP99': 1500.0},
'query': {'Requests': 33686,
'Fails': 0,
'RPS': 0.78,
'fail_s': 0.0,
'RT_max': 17270.47,
'RT_avg': 318.39,
'TP50': 250.0,
'TP99': 1000.0},
'scene_search_test': {'Requests': 33835,
'Fails': 1,
'RPS': 0.78,
'fail_s': 0.0,
'RT_max': 112787.13,
'RT_avg': 23891.22,
'TP50': 21000.0,
'TP99': 60000.0},
'search': {'Requests': 33969,
'Fails': 0,
'RPS': 0.79,
'fail_s': 0.0,
'RT_max': 18179.21,
'RT_avg': 818.58,
'TP50': 690.0,
'TP99': 2100.0}}}}} @MrPresent-Han
wangting0128
commented
7 months ago argo task：multi-vector-scene-mix-sb65c test case name: test_hybrid_search_locust_ddl_dql_cluster image: master-20240313-5220005a-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix-sb65c-5-etcd-0 1/1 Running 0 12h 10.104.17.119 4am-node23 <none> <none>
multi-vector-scene-mix-sb65c-5-etcd-1 1/1 Running 0 12h 10.104.15.48 4am-node20 <none> <none>
multi-vector-scene-mix-sb65c-5-etcd-2 1/1 Running 0 12h 10.104.24.128 4am-node29 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-datacoord-7c58b79cdbh8h49 1/1 Running 0 12h 10.104.31.17 4am-node34 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-datanode-7ccbd5b54c-7dwr6 1/1 Running 2 (12h ago) 12h 10.104.31.16 4am-node34 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-indexcoord-cc568d475j4rwp 1/1 Running 0 12h 10.104.20.118 4am-node22 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-indexnode-5584d89c6b8z858 1/1 Running 0 12h 10.104.20.119 4am-node22 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-indexnode-5584d89c6bhmz5g 1/1 Running 0 12h 10.104.18.4 4am-node25 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-indexnode-5584d89c6blr8dk 1/1 Running 0 12h 10.104.29.94 4am-node35 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-indexnode-5584d89c6bnnk8q 1/1 Running 0 12h 10.104.31.18 4am-node34 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-proxy-796f87d7df-bkn9w 1/1 Running 1 (12h ago) 12h 10.104.4.246 4am-node11 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-querycoord-f4f9d9cb-2df96 1/1 Running 2 (12h ago) 12h 10.104.9.126 4am-node14 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-querynode-6687f774985nh5b 1/1 Running 0 12h 10.104.29.96 4am-node35 <none> <none>
multi-vector-scene-mix-sb65c-5-milvus-rootcoord-5ff578dfb648xs5 1/1 Running 1 (12h ago) 12h 10.104.29.95 4am-node35 <none> <none>
multi-vector-scene-mix-sb65c-5-minio-0 1/1 Running 0 12h 10.104.23.250 4am-node27 <none> <none>
multi-vector-scene-mix-sb65c-5-minio-1 1/1 Running 0 12h 10.104.33.85 4am-node36 <none> <none>
multi-vector-scene-mix-sb65c-5-minio-2 1/1 Running 0 12h 10.104.32.18 4am-node39 <none> <none>
multi-vector-scene-mix-sb65c-5-minio-3 1/1 Running 0 12h 10.104.15.47 4am-node20 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-bookie-0 1/1 Running 0 12h 10.104.28.96 4am-node33 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-bookie-1 1/1 Running 0 12h 10.104.16.224 4am-node21 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-bookie-2 1/1 Running 0 12h 10.104.23.253 4am-node27 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-bookie-init-6nck9 0/1 Completed 0 12h 10.104.33.51 4am-node36 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-broker-0 1/1 Running 0 12h 10.104.21.108 4am-node24 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-proxy-0 1/1 Running 0 12h 10.104.34.72 4am-node37 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-pulsar-init-xkg8r 0/1 Completed 0 12h 10.104.34.70 4am-node37 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-recovery-0 1/1 Running 0 12h 10.104.21.107 4am-node24 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.23.252 4am-node27 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.30.218 4am-node38 <none> <none>
multi-vector-scene-mix-sb65c-5-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.15.72 4am-node20 <none> <none>client pod name: multi-vector-scene-mix-sb65c-2574400242 client log:
test steps:
concurrent test and calculation of RT and QPS
:purpose: `DDL & DQL`
verify DDL & DQL scenario,
which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'float_vector_1': 128dim,
'float_vector_2': 128dim,
'float_vector_3': 128dim,
scalar field: int64_1, varchar_1
2. build indexes:
IVF_FLAT: 'float_vector'
HNSW: 'float_vector_1',
DISKANN: 'float_vector_2'
IVF_SQ8: 'float_vector_3'
INVERTED: 'int64_1', 'varchar_1'
default scalar index: 'id'
3. insert 1 million data
4. flush collection
5. build indexes again using the same params
6. load collection
replica: 1
7. concurrent request:
- scene_test
(collection: create->insert->flush->index->drop)
- search
- hybrid_search
- querytest result:
[2024-03-13 23:37:02,870 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-03-13 23:37:02,871 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: grpc hybrid_search 11864 0(0.00%) | 1732 103 37132 570 | 0.27 0.00 (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: grpc query 11862 0(0.00%) | 896 85 30712 390 | 0.27 0.00 (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: grpc scene_test 11621 3(0.03%) | 65651 7161 138466 65000 | 0.27 0.00 (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: grpc search 11883 0(0.00%) | 5793 1304 44083 4900 | 0.28 0.00 (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: Aggregated 47230 3(0.01%) | 18271 85 138466 2400 | 1.09 0.00 (stats.py:789)
[2024-03-13 23:37:02,872 - INFO - fouram]: (stats.py:790)
[2024-03-13 23:37:02,875 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '32Gi'},
'requests': {'cpu': '5.0',
'memory': '17Gi'}},
'replicas': 1},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': 'master-20240313-5220005a-amd64'}}},
'host': 'multi-vector-scene-mix-sb65c-5-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_ddl_dql_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}}},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_test',
'weight': 1,
'params': {'dim': 128,
'data_size': 3000,
'nb': 3000,
'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2',
'other_fields': [],
'scalars_params': {},
'scalars_index': {},
'vectors_index': {}}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'100000',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'id '
'< '
'900000',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'expr': 'varchar_1 '
'> '
'"1"',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024031387043084,
'datetime': '2024-03-13 11:18:24.411414',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 125.8894,
'float_vector_1': {'RT': 79.8306},
'float_vector_2': {'RT': 33.0875},
'float_vector_3': {'RT': 0.5251},
'id': {'RT': 0.5245},
'int64_1': {'RT': 0.5253},
'varchar_1': {'RT': 0.5271}},
'insert': {'total_time': 172.9516,
'VPS': 5781.9644,
'batch_time': 1.7295,
'batch': 10000},
'flush': {'RT': 7.9167},
'load': {'RT': 8.5803},
'Locust': {'Aggregated': {'Requests': 47230,
'Fails': 3,
'RPS': 1.09,
'fail_s': 0.0,
'RT_max': 138466.47,
'RT_avg': 18271.67,
'TP50': 2400.0,
'TP99': 71000.0},
'hybrid_search': {'Requests': 11864,
'Fails': 0,
'RPS': 0.27,
'fail_s': 0.0,
'RT_max': 37132.72,
'RT_avg': 1732.59,
'TP50': 570.0,
'TP99': 13000.0},
'query': {'Requests': 11862,
'Fails': 0,
'RPS': 0.27,
'fail_s': 0.0,
'RT_max': 30712.89,
'RT_avg': 896.77,
'TP50': 390.0,
'TP99': 8600.0},
'scene_test': {'Requests': 11621,
'Fails': 3,
'RPS': 0.27,
'fail_s': 0.0,
'RT_max': 138466.47,
'RT_avg': 65651.34,
'TP50': 65000.0,
'TP99': 79000.0},
'search': {'Requests': 11883,
'Fails': 0,
'RPS': 0.28,
'fail_s': 0.0,
'RT_max': 44083.03,
'RT_avg': 5793.48,
'TP50': 4900.0,
'TP99': 18000.0}}}}}
xiaofan-luan
commented
7 months ago /assign @czs007 pleas help on this
wangting0128
commented
6 months ago argo task: fouramf-multi-vector-rxb7c test case name: test_inverted_locust_varchar_dml_dql_cluster
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouramf-multi-vr-rxb7c-45-7721-etcd-0 1/1 Running 0 10h 10.104.32.56 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-etcd-1 1/1 Running 0 10h 10.104.23.150 4am-node27 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-etcd-2 1/1 Running 0 10h 10.104.19.202 4am-node28 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-datacoord-7d4b57c8d59m4rg 1/1 Running 0 10h 10.104.4.207 4am-node11 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-datanode-75b47db6df-j956l 1/1 Running 0 10h 10.104.4.211 4am-node11 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-indexcoord-789c68f6bqhg5h 1/1 Running 0 10h 10.104.32.43 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-indexnode-56b946b5976j6ss 1/1 Running 0 10h 10.104.4.209 4am-node11 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-proxy-68cccbcfb8-6gq5w 1/1 Running 0 10h 10.104.4.206 4am-node11 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-querycoord-8549f447c4wsw7 1/1 Running 0 10h 10.104.4.208 4am-node11 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-querynode-55c7f6966cjwc24 1/1 Running 0 10h 10.104.32.42 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-milvus-rootcoord-6c89b78cfcp6lpq 1/1 Running 1 (10h ago) 10h 10.104.32.44 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-minio-0 1/1 Running 0 10h 10.104.32.58 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-minio-1 1/1 Running 0 10h 10.104.23.151 4am-node27 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-minio-2 1/1 Running 0 10h 10.104.20.229 4am-node22 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-minio-3 1/1 Running 0 10h 10.104.19.199 4am-node28 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-bookie-0 1/1 Running 0 10h 10.104.32.57 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-bookie-1 1/1 Running 0 10h 10.104.20.233 4am-node22 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-bookie-2 1/1 Running 0 10h 10.104.19.203 4am-node28 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-bookie-init-xnj9t 0/1 Completed 0 10h 10.104.6.132 4am-node13 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-broker-0 1/1 Running 0 10h 10.104.32.45 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-proxy-0 1/1 Running 0 10h 10.104.4.210 4am-node11 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-pulsar-init-9klk6 0/1 Completed 0 10h 10.104.6.131 4am-node13 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-recovery-0 1/1 Running 0 10h 10.104.1.98 4am-node10 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-zookeeper-0 1/1 Running 0 10h 10.104.32.54 4am-node39 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-zookeeper-1 1/1 Running 0 10h 10.104.23.155 4am-node27 <none> <none>
fouramf-multi-vr-rxb7c-45-7721-pulsar-zookeeper-2 1/1 Running 0 10h 10.104.29.2 4am-node35 <none> <none>client pod name: fouramf-multi-vector-rxb7c-850891244 client log: client.log
[2024-03-21 12:47:33,065 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc delete 3566 0(0.00%) | 3929 3 20777 3500 | 0.60 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc flush 3544 0(0.00%) | 15025 1090 43764 14000 | 0.80 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc hybrid_search 3550 0(0.00%) | 4419 1911 11622 3900 | 0.70 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc insert 3602 0(0.00%) | 4126 13 19968 3800 | 1.10 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc load 3592 0(0.00%) | 7716 9 30409 7100 | 0.30 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc query 3727 0(0.00%) | 5003 6 21888 4400 | 0.60 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: grpc search 3684 0(0.00%) | 3614 1945 9864 3300 | 0.90 0.00 (stats.py:789)
[2024-03-21 12:47:33,065 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-21 12:47:33,066 - INFO - fouram]: Aggregated 25265 0(0.00%) | 6233 3 43764 4400 | 5.00 0.00 (stats.py:789)
[2024-03-21 12:47:33,066 - INFO - fouram]: (stats.py:790)
[2024-03-21 12:47:33,068 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc delete 3500 4500 5200 5700 7400 9200 12000 13000 18000 21000 21000 3566 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc flush 14000 17000 19000 21000 25000 28000 34000 37000 43000 44000 44000 3544 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc hybrid_search 3900 4300 4800 5500 7100 8200 9100 9500 11000 12000 12000 3550 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc insert 3800 4800 5500 6000 7500 9200 12000 13000 17000 20000 20000 3602 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc load 7100 8700 9800 11000 14000 18000 21000 23000 29000 30000 30000 3592 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc query 4400 5500 6400 7000 9300 12000 14000 16000 20000 22000 22000 3727 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: grpc search 3300 3600 3900 4200 5300 5900 6600 7000 8700 9900 9900 3684 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: Aggregated 4400 6000 7600 8700 13000 18000 23000 27000 38000 43000 44000 25265 (stats.py:819)
[2024-03-21 12:47:33,068 - INFO - fouram]: (stats.py:820)
[2024-03-21 12:47:48,375 - ERROR - fouram]: RPC error: [batch_insert], <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, <Time:{'RPC start': '2024-03-21 12:47:36.673136', 'RPC error': '2024-03-21 12:47:48.375516'}> (decorators.py:146)
[2024-03-21 12:47:48,377 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, [requestId: 31c21bb0-e781-11ee-8f85-5a3df0bd6527] (api_request.py:57)
[2024-03-21 12:47:48,377 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)> (func_check.py:48)
[2024-03-21 12:47:48,382 - ERROR - fouram]: RPC error: [batch_insert], <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, <Time:{'RPC start': '2024-03-21 12:47:36.682402', 'RPC error': '2024-03-21 12:47:48.382051'}> (decorators.py:146)
[2024-03-21 12:47:48,382 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, [requestId: 31c395c6-e781-11ee-8f85-5a3df0bd6527] (api_request.py:57)
[2024-03-21 12:47:48,382 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)> (func_check.py:48)
[2024-03-21 12:47:53,069 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-03-21 12:47:53,069 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc delete 3591 0(0.00%) | 3924 3 20777 3500 | 1.40 0.00 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc flush 3546 0(0.00%) | 15028 1090 43764 14000 | 0.00 0.00 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc hybrid_search 3561 0(0.00%) | 4433 1911 11985 3900 | 0.20 0.00 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc insert 3617 2(0.06%) | 4149 13 19968 3800 | 0.20 0.20 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc load 3612 0(0.00%) | 7737 9 30409 7100 | 0.70 0.00 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc query 3739 0(0.00%) | 5007 6 21888 4400 | 0.60 0.00 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: grpc search 3696 0(0.00%) | 3621 1945 10210 3300 | 0.50 0.00 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: Aggregated 25362 2(0.01%) | 6239 3 43764 4400 | 3.60 0.20 (stats.py:789)
[2024-03-21 12:47:53,070 - INFO - fouram]: (stats.py:790)
[2024-03-21 12:47:53,072 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: grpc delete 3500 4500 5200 5700 7400 9200 12000 13000 18000 21000 21000 3591 (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: grpc flush 14000 17000 19000 21000 25000 28000 34000 37000 43000 44000 44000 3546 (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: grpc hybrid_search 3900 4300 4800 5500 7200 8200 9100 9600 12000 12000 12000 3561 (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: grpc insert 3800 4800 5500 6000 7600 9300 12000 13000 17000 20000 20000 3617 (stats.py:819)
[2024-03-21 12:47:53,072 - INFO - fouram]: grpc load 7100 8700 9900 11000 14000 18000 21000 23000 29000 30000 30000 3612 (stats.py:819)
[2024-03-21 12:47:53,073 - INFO - fouram]: grpc query 4400 5600 6400 7000 9400 12000 14000 16000 20000 22000 22000 3739 (stats.py:819)
[2024-03-21 12:47:53,073 - INFO - fouram]: grpc search 3300 3600 3900 4200 5300 5900 6600 7000 8800 10000 10000 3696 (stats.py:819)
[2024-03-21 12:47:53,073 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-03-21 12:47:53,073 - INFO - fouram]: Aggregated 4400 6100 7600 8700 13000 18000 23000 27000 38000 43000 44000 25362 (stats.py:819)test steps:
concurrent test and calculation of RT and QPS
:purpose: `varchar: different max_length`
verify concurrent DML & DQL scenario which has 3 VARCHAR scalars fields and creating INVERTED index
:test steps:
1. create collection with fields:
'float_vector': 3dim,
'varchar_1': max_length=256, varchar_filled=True
'varchar_2': max_length=32768, varchar_filled=True
'varchar_3': max_length=65535, varchar_filled=True
2. build indexes:
IVF_FLAT: 'float_vector'
INVERTED: 'varchar_1', 'varchar_2', 'varchar_3'
3. insert 300k data
4. flush collection
5. build indexes again using the same params
6. load collection
7. concurrent request:
- insert
- delete
- flush
- load
- search
- hybrid_search
- querytest result:
[2024-03-21 12:54:53,258 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-03-21 12:54:53,258 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc delete 3874 0(0.00%) | 4250 3 28231 3700 | 1.08 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc flush 3796 0(0.00%) | 15391 1090 44526 14000 | 1.05 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc hybrid_search 3828 0(0.00%) | 4712 1911 28781 4000 | 1.06 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc insert 3882 2(0.05%) | 4493 13 30221 4000 | 1.08 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc load 3874 0(0.00%) | 8345 9 42500 7400 | 1.08 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc query 4003 0(0.00%) | 5437 6 31628 4600 | 1.11 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: grpc search 3956 0(0.00%) | 3791 1945 22441 3400 | 1.10 0.00 (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-03-21 12:54:53,259 - INFO - fouram]: Aggregated 27213 2(0.01%) | 6595 3 44526 4700 | 7.56 0.00 (stats.py:789)
[2024-03-21 12:54:53,260 - INFO - fouram]: (stats.py:790)
[2024-03-21 12:54:53,263 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c4m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '8',
'memory': '32Gi'},
'requests': {'cpu': '8',
'memory': '32Gi'}},
'replicas': 1},
'indexNode': {'resources': {'limits': {'cpu': '4.0',
'memory': '16Gi'},
'requests': {'cpu': '3.0',
'memory': '9Gi'}},
'replicas': 1},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '4Gi'},
'requests': {'cpu': '2.0',
'memory': '3Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': '2.4-20240321-775129bd-amd64'}}},
'host': 'fouramf-multi-vr-rxb7c-45-7721-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_inverted_locust_varchar_dml_dql_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 3,
'scalars_index': {'varchar_1': {'index_type': 'INVERTED'},
'varchar_2': {'index_type': 'INVERTED'},
'varchar_3': {'index_type': 'INVERTED'}},
'scalars_params': {'varchar_1': {'params': {'max_length': 256},
'other_params': {'varchar_filled': True}},
'varchar_2': {'params': {'max_length': 32768},
'other_params': {'varchar_filled': True}},
'varchar_3': {'params': {'max_length': 65535},
'other_params': {'varchar_filled': True}}},
'dataset_name': 'local',
'dataset_size': 300000,
'ni_per': 50},
'collection_params': {'other_fields': ['varchar_1',
'varchar_2',
'varchar_3'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 50,
'during_time': '1h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'insert',
'weight': 1,
'params': {'nb': 10,
'timeout': 30,
'random_id': True,
'random_vector': True,
'varchar_filled': False,
'start_id': 300000}},
{'type': 'delete',
'weight': 1,
'params': {'expr': '',
'delete_length': 10,
'timeout': 30}},
{'type': 'flush',
'weight': 1,
'params': {'timeout': 600}},
{'type': 'load',
'weight': 1,
'params': {'replica_number': 1,
'timeout': 30}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 32},
'expr': 'varchar_1 '
'like '
'"a%" '
'&& '
'varchar_2 '
'like '
'"A%" '
'&& '
'varchar_3 '
'like '
'"0%" '
'&& '
'id '
'> 0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 60,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 10,
'reqs': [{'search_param': {'nprobe': 16},
'anns_field': 'float_vector',
'expr': 'varchar_1 '
'like '
'"0%"',
'top_k': 2000},
{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'varchar_2 '
'like '
'"9%"'}],
'rerank': {'WeightedRanker': [0.5,
0.5]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 60,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'varchar_3 '
'like '
'"a%" '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 60,
'random_data': True,
'random_count': 20,
'random_range': [0,
150000.0],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024032188745402,
'datetime': '2024-03-21 02:41:14.957001',
'client_version': '2.2'},
'result': {'test_result': {'index': {'RT': 7171.1202,
'varchar_1': {'RT': 4940.5694},
'varchar_2': {'RT': 6759.0299},
'varchar_3': {'RT': 2635.1862}},
'insert': {'total_time': 870.8737,
'VPS': 344.4816,
'batch_time': 0.1451,
'batch': 50},
'flush': {'RT': 4.0307},
'load': {'RT': 69.704},
'Locust': {'Aggregated': {'Requests': 27213,
'Fails': 2,
'RPS': 7.56,
'fail_s': 0.0,
'RT_max': 44526.39,
'RT_avg': 6595.04,
'TP50': 4700.0,
'TP99': 28000.0},
'delete': {'Requests': 3874,
'Fails': 0,
'RPS': 1.08,
'fail_s': 0.0,
'RT_max': 28231.71,
'RT_avg': 4250.65,
'TP50': 3700.0,
'TP99': 15000.0},
'flush': {'Requests': 3796,
'Fails': 0,
'RPS': 1.05,
'fail_s': 0.0,
'RT_max': 44526.39,
'RT_avg': 15391.36,
'TP50': 14000.0,
'TP99': 38000.0},
'hybrid_search': {'Requests': 3828,
'Fails': 0,
'RPS': 1.06,
'fail_s': 0.0,
'RT_max': 28781.42,
'RT_avg': 4712.18,
'TP50': 4000.0,
'TP99': 11000.0},
'insert': {'Requests': 3882,
'Fails': 2,
'RPS': 1.08,
'fail_s': 0.0,
'RT_max': 30221.49,
'RT_avg': 4493.02,
'TP50': 4000.0,
'TP99': 15000.0},
'load': {'Requests': 3874,
'Fails': 0,
'RPS': 1.08,
'fail_s': 0.0,
'RT_max': 42500.81,
'RT_avg': 8345.72,
'TP50': 7400.0,
'TP99': 25000.0},
'query': {'Requests': 4003,
'Fails': 0,
'RPS': 1.11,
'fail_s': 0.0,
'RT_max': 31628.49,
'RT_avg': 5437.75,
'TP50': 4600.0,
'TP99': 17000.0},
'search': {'Requests': 3956,
'Fails': 0,
'RPS': 1.1,
'fail_s': 0.0,
'RT_max': 22441.21,
'RT_avg': 3791.56,
'TP50': 3400.0,
'TP99': 7500.0}}}}}@czs007
wangting0128
commented
6 months ago same scene: #31942
client log:
[2024-04-05 23:14:56,911 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-04-05 23:14:56,911 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-04-05 23:14:56,911 - INFO - fouram]: grpc delete 8 21 33 49 180 690 2400 3800 9600 21000 37000 32693 (stats.py:819)
[2024-04-05 23:14:56,911 - INFO - fouram]: grpc hybrid_search 23 31 39 47 88 240 840 1400 3700 6800 13000 650202 (stats.py:819)
[2024-04-05 23:14:56,911 - INFO - fouram]: grpc insert 33 48 71 97 300 950 2700 4100 12000 21000 42000 32732 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: grpc load 81 270 670 1200 3000 4900 9800 14000 26000 32000 62000 32913 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: grpc query 10 20 32 47 170 720 2500 4000 10000 20000 43000 325201 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: grpc scene_hybrid_search_test 171000 242000 327000 400000 613000 621000 736000 887000 1255000 1390000 1531000 64853 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: grpc scene_test 76000 81000 85000 88000 107000 134000 171000 221000 335000 443000 475000 65255 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: grpc search 27 37 49 63 180 350 910 1500 3600 5900 27000 650499 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: Aggregated 25 38 57 88 810 72000 159000 290000 688000 1152000 1531000 1854348 (stats.py:819)
[2024-04-05 23:14:56,912 - INFO - fouram]: (stats.py:820)
[2024-04-05 23:15:11,159 - ERROR - fouram]: RPC error: [batch_insert], <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, <Time:{'RPC start': '2024-04-05 23:15:00.503031', 'RPC error': '2024-04-05 23:15:11.159161'}> (decorators.py:146)
[2024-04-05 23:15:11,160 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)>, [requestId: 53699c52-f3a2-11ee-bb67-26cb299a8700] (api_request.py:57)
[2024-04-05 23:15:11,160 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest unexpected error)> (func_check.py:54)
[2024-04-05 23:15:11,160 - ERROR - fouram]: [func_time_catch] : (api_request.py:125)
[2024-04-05 23:15:16,915 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-04-05 23:15:16,915 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-05 23:15:16,915 - INFO - fouram]: grpc delete 32693 0(0.00%) | 172 2 37023 8 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,915 - INFO - fouram]: grpc hybrid_search 650208 0(0.00%) | 81 8 12920 23 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,915 - INFO - fouram]: grpc insert 32732 0(0.00%) | 224 12 42277 33 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,915 - INFO - fouram]: grpc load 32913 7(0.02%) | 1034 5 61959 81 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,915 - INFO - fouram]: grpc query 325202 0(0.00%) | 176 3 42896 10 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,916 - INFO - fouram]: grpc scene_hybrid_search_test 64854 4524(6.98%) | 241121 14051 1531061 171000 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,916 - INFO - fouram]: grpc scene_test 65256 1(0.00%) | 83155 57623 474859 76000 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,916 - INFO - fouram]: grpc search 650503 0(0.00%) | 97 11 27306 27 | 0.00 0.00 (stats.py:789)
[2024-04-05 23:15:16,916 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-05 23:15:16,916 - INFO - fouram]: Aggregated 1854361 4532(0.24%) | 11478 2 1531061 25 | 0.00 0.00 (stats.py:789)argo task：multi-vector-scene-mix2-fnkhz
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-fnkhz-1-etcd-0 1/1 Running 0 12h 10.104.26.129 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-1-etcd-1 1/1 Running 0 12h 10.104.15.98 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-1-etcd-2 1/1 Running 0 12h 10.104.19.232 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-datacoord-5996bc99dz6mwc 1/1 Running 0 12h 10.104.14.220 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-datanode-6bf946fbfcjlvfz 1/1 Running 1 (12h ago) 12h 10.104.1.189 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-indexcoord-689c86b6xpp82 1/1 Running 0 12h 10.104.14.219 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-indexnode-7f8b594687f7fl 1/1 Running 0 12h 10.104.9.134 4am-node14 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-indexnode-7f8b594687zjmn 1/1 Running 0 12h 10.104.14.221 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-indexnode-7f8b59468nznmj 1/1 Running 0 12h 10.104.5.140 4am-node12 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-indexnode-7f8b59468szmq5 1/1 Running 0 12h 10.104.26.121 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-proxy-5bd6665f89-mnvz7 1/1 Running 1 (12h ago) 12h 10.104.14.218 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-querycoord-c8fb9676vkf86 1/1 Running 1 (12h ago) 12h 10.104.13.26 4am-node16 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-querynode-5676bccbdjw2wp 1/1 Running 0 12h 10.104.13.25 4am-node16 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-querynode-5676bccbdwbdhr 1/1 Running 0 12h 10.104.6.152 4am-node13 <none> <none>
multi-vector-scene-mix2-fnkhz-1-milvus-rootcoord-85698cbddr8jt6 1/1 Running 1 (12h ago) 12h 10.104.4.102 4am-node11 <none> <none>
multi-vector-scene-mix2-fnkhz-1-minio-0 1/1 Running 0 12h 10.104.15.99 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-1-minio-1 1/1 Running 0 12h 10.104.19.234 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-1-minio-2 1/1 Running 0 12h 10.104.33.241 4am-node36 <none> <none>
multi-vector-scene-mix2-fnkhz-1-minio-3 1/1 Running 0 12h 10.104.32.18 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-bookie-0 1/1 Running 0 12h 10.104.32.14 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-bookie-1 1/1 Running 0 12h 10.104.19.237 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-bookie-2 1/1 Running 0 12h 10.104.15.114 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-bookie-init-9qxs8 0/1 Completed 0 12h 10.104.6.151 4am-node13 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-broker-0 1/1 Running 0 12h 10.104.4.103 4am-node11 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-proxy-0 1/1 Running 0 12h 10.104.9.133 4am-node14 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-pulsar-init-njjlr 0/1 Completed 0 12h 10.104.14.217 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-recovery-0 1/1 Running 0 12h 10.104.4.101 4am-node11 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.26.126 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.25.103 4am-node30 <none> <none>
multi-vector-scene-mix2-fnkhz-1-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.31.14 4am-node34 <none> <none>client log:
test case name：test_hybrid_search_locust_multi_ddl_dql_search_cluster server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-fnkhz-2-etcd-0 1/1 Running 0 12h 10.104.32.20 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-2-etcd-1 1/1 Running 0 12h 10.104.31.5 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-2-etcd-2 1/1 Running 0 12h 10.104.15.118 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-datacoord-59ff9dfd9zs5gw 1/1 Running 0 12h 10.104.1.192 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-datanode-69d887d85-cjzwk 1/1 Running 1 (12h ago) 12h 10.104.4.104 4am-node11 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-indexcoord-849cc974f9lfm 1/1 Running 0 12h 10.104.1.190 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-indexnode-657cd9fc4jws67 1/1 Running 0 12h 10.104.19.226 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-indexnode-657cd9fc4n6z67 1/1 Running 0 12h 10.104.32.11 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-indexnode-657cd9fc4w66lg 1/1 Running 0 12h 10.104.4.105 4am-node11 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-indexnode-657cd9fc4wq5xq 1/1 Running 0 12h 10.104.33.235 4am-node36 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-proxy-597948d6ff-ngsxd 1/1 Running 1 (12h ago) 12h 10.104.1.191 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-querycoord-5654c954k4tt2 1/1 Running 1 (12h ago) 12h 10.104.32.8 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-querynode-6f85d9dbfcl2cc 1/1 Running 0 12h 10.104.34.118 4am-node37 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-querynode-6f85d9dbffhzp5 1/1 Running 0 12h 10.104.1.194 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-milvus-rootcoord-5d75b7df867lqt 1/1 Running 0 12h 10.104.1.188 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-minio-0 1/1 Running 0 12h 10.104.15.113 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-2-minio-1 1/1 Running 0 12h 10.104.33.242 4am-node36 <none> <none>
multi-vector-scene-mix2-fnkhz-2-minio-2 1/1 Running 0 12h 10.104.32.19 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-2-minio-3 1/1 Running 0 12h 10.104.31.4 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-bookie-0 1/1 Running 0 12h 10.104.26.132 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-bookie-1 1/1 Running 0 12h 10.104.15.119 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-bookie-2 1/1 Running 0 12h 10.104.31.6 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-bookie-init-pfp24 0/1 Completed 0 12h 10.104.1.187 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-broker-0 1/1 Running 0 12h 10.104.15.92 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-proxy-0 1/1 Running 0 12h 10.104.1.193 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-pulsar-init-h8xgn 0/1 Completed 0 12h 10.104.1.186 4am-node10 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-recovery-0 1/1 Running 0 12h 10.104.19.224 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.15.106 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.31.10 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-2-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.20.170 4am-node22 <none> <none>client log:
test case name：test_hybrid_search_locust_multi_ddl_dql_hybrid_search_cluster server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-fnkhz-3-etcd-0 1/1 Running 0 12h 10.104.25.100 4am-node30 <none> <none>
multi-vector-scene-mix2-fnkhz-3-etcd-1 1/1 Running 0 12h 10.104.15.120 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-3-etcd-2 1/1 Running 0 12h 10.104.21.79 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-datacoord-6bbd5cc66q55tq 1/1 Running 0 12h 10.104.26.125 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-datanode-dc96497f9-lxlt8 1/1 Running 1 (12h ago) 12h 10.104.26.124 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-indexcoord-7f9456d79bjc7 1/1 Running 0 12h 10.104.5.141 4am-node12 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-indexnode-5fb5bbdbcg4fvs 1/1 Running 0 12h 10.104.31.252 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-indexnode-5fb5bbdbcjfssg 1/1 Running 0 12h 10.104.14.225 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-indexnode-5fb5bbdbcwf469 1/1 Running 0 12h 10.104.25.97 4am-node30 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-indexnode-5fb5bbdbcwqtxh 1/1 Running 0 12h 10.104.23.136 4am-node27 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-proxy-84c7b885dc-gp4rq 1/1 Running 1 (12h ago) 12h 10.104.9.137 4am-node14 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-querycoord-57db7959zlfqx 1/1 Running 1 (12h ago) 12h 10.104.9.136 4am-node14 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-querynode-68dd649bfn5t4f 1/1 Running 0 12h 10.104.9.138 4am-node14 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-querynode-68dd649bfs4qfn 1/1 Running 0 12h 10.104.5.142 4am-node12 <none> <none>
multi-vector-scene-mix2-fnkhz-3-milvus-rootcoord-bf588c4548vskr 1/1 Running 1 (12h ago) 12h 10.104.9.135 4am-node14 <none> <none>
multi-vector-scene-mix2-fnkhz-3-minio-0 1/1 Running 0 12h 10.104.25.101 4am-node30 <none> <none>
multi-vector-scene-mix2-fnkhz-3-minio-1 1/1 Running 0 12h 10.104.21.78 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-3-minio-2 1/1 Running 0 12h 10.104.33.244 4am-node36 <none> <none>
multi-vector-scene-mix2-fnkhz-3-minio-3 1/1 Running 0 12h 10.104.24.111 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-bookie-0 1/1 Running 0 12h 10.104.32.23 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-bookie-1 1/1 Running 0 12h 10.104.24.114 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-bookie-2 1/1 Running 0 12h 10.104.21.82 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-bookie-init-wbjp4 0/1 Completed 0 12h 10.104.14.222 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-broker-0 1/1 Running 0 12h 10.104.21.71 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-proxy-0 1/1 Running 0 12h 10.104.15.94 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-pulsar-init-4zbpf 0/1 Completed 0 12h 10.104.14.223 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-recovery-0 1/1 Running 0 12h 10.104.14.224 4am-node18 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.15.121 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.25.105 4am-node30 <none> <none>
multi-vector-scene-mix2-fnkhz-3-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.31.17 4am-node34 <none> <none>client log:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-scene-mix2-fnkhz-4-etcd-0 1/1 Running 0 12h 10.104.19.3 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-4-etcd-1 1/1 Running 0 12h 10.104.24.118 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-4-etcd-2 1/1 Running 0 12h 10.104.23.140 4am-node27 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-datacoord-5976bb786p6djr 1/1 Running 0 12h 10.104.26.127 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-datanode-7656c585c66ms2t 1/1 Running 1 (12h ago) 12h 10.104.21.73 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-indexcoord-c4dfdd5dcgxcl 1/1 Running 0 12h 10.104.33.236 4am-node36 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-indexnode-6f47cc444jqrq8 1/1 Running 0 12h 10.104.15.97 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-indexnode-6f47cc444m66g4 1/1 Running 0 12h 10.104.26.128 4am-node32 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-indexnode-6f47cc444rq87l 1/1 Running 0 12h 10.104.33.237 4am-node36 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-indexnode-6f47cc444wdmz4 1/1 Running 0 12h 10.104.32.13 4am-node39 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-proxy-7777f689b8-m76k9 1/1 Running 1 (12h ago) 12h 10.104.21.75 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-querycoord-6854c8fbqfnxl 1/1 Running 1 (12h ago) 12h 10.104.24.105 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-querynode-6bb84c488knzj4 1/1 Running 0 12h 10.104.27.81 4am-node31 <none> <none>
multi-vector-scene-mix2-fnkhz-4-milvus-rootcoord-84b5487bcpsll7 1/1 Running 1 (12h ago) 12h 10.104.21.74 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-4-minio-0 1/1 Running 0 12h 10.104.24.113 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-4-minio-1 1/1 Running 0 12h 10.104.19.250 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-4-minio-2 1/1 Running 0 12h 10.104.23.138 4am-node27 <none> <none>
multi-vector-scene-mix2-fnkhz-4-minio-3 1/1 Running 0 12h 10.104.31.8 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-bookie-0 1/1 Running 0 12h 10.104.19.5 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-bookie-1 1/1 Running 0 12h 10.104.24.119 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-bookie-2 1/1 Running 0 12h 10.104.21.85 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-bookie-init-q5g2d 0/1 Completed 0 12h 10.104.15.95 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-broker-0 1/1 Running 0 12h 10.104.4.106 4am-node11 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-proxy-0 1/1 Running 0 12h 10.104.15.96 4am-node20 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-pulsar-init-jcxk7 0/1 Completed 0 12h 10.104.21.72 4am-node24 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-recovery-0 1/1 Running 0 12h 10.104.24.106 4am-node29 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.19.252 4am-node28 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.31.12 4am-node34 <none> <none>
multi-vector-scene-mix2-fnkhz-4-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.25.107 4am-node30 <none> <none>client log:
czs007
commented
6 months ago The priority is not high; it will be addressed in the upcoming 2.4.1 update.
wangting0128
commented
4 months ago argo task: multi-vector-corn-1-1717855200 test case name: test_hybrid_search_locust_ddl_dql_cluster image: 2.4-20240605-443197bd-amd64
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
multi-vector-corn-1-1717855200-5-etcd-0 1/1 Running 0 12h 10.104.15.132 4am-node20 <none> <none>
multi-vector-corn-1-1717855200-5-etcd-1 1/1 Running 0 12h 10.104.21.179 4am-node24 <none> <none>
multi-vector-corn-1-1717855200-5-etcd-2 1/1 Running 0 12h 10.104.27.38 4am-node31 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-datacoord-79bd7f94lxpgq 1/1 Running 5 (12h ago) 12h 10.104.18.43 4am-node25 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-datanode-579d5695fl2p5z 1/1 Running 4 (12h ago) 12h 10.104.18.44 4am-node25 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-indexcoord-84d9fb9zr7vd 1/1 Running 0 12h 10.104.18.40 4am-node25 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-indexnode-84c876847fbhq 1/1 Running 5 (12h ago) 12h 10.104.9.117 4am-node14 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-indexnode-84c87684nsw5j 1/1 Running 5 (12h ago) 12h 10.104.14.177 4am-node18 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-indexnode-84c87684w57fj 1/1 Running 5 (12h ago) 12h 10.104.23.120 4am-node27 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-indexnode-84c87684xtkvh 1/1 Running 4 (12h ago) 12h 10.104.34.139 4am-node37 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-proxy-74ffd476bb-ptt9m 1/1 Running 5 (12h ago) 12h 10.104.18.45 4am-node25 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-querycoord-5cdcbc68l52l 1/1 Running 5 (12h ago) 12h 10.104.18.42 4am-node25 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-querynode-555dc655vghql 1/1 Running 5 (12h ago) 12h 10.104.19.83 4am-node28 <none> <none>
multi-vector-corn-1-1717855200-5-milvus-rootcoord-7c4dd7f74bp7s 1/1 Running 5 (12h ago) 12h 10.104.18.41 4am-node25 <none> <none>
multi-vector-corn-1-1717855200-5-minio-0 1/1 Running 0 12h 10.104.15.130 4am-node20 <none> <none>
multi-vector-corn-1-1717855200-5-minio-1 1/1 Running 0 12h 10.104.27.33 4am-node31 <none> <none>
multi-vector-corn-1-1717855200-5-minio-2 1/1 Running 0 12h 10.104.21.180 4am-node24 <none> <none>
multi-vector-corn-1-1717855200-5-minio-3 1/1 Running 0 12h 10.104.23.146 4am-node27 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-bookie-0 1/1 Running 0 12h 10.104.15.133 4am-node20 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-bookie-1 1/1 Running 0 12h 10.104.27.44 4am-node31 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-bookie-2 1/1 Running 0 12h 10.104.23.149 4am-node27 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-bookie-init-pkvl7 0/1 Completed 0 12h 10.104.13.101 4am-node16 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-broker-0 1/1 Running 0 12h 10.104.6.76 4am-node13 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-proxy-0 1/1 Running 0 12h 10.104.5.17 4am-node12 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-pulsar-init-8zkdz 0/1 Completed 0 12h 10.104.9.115 4am-node14 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-recovery-0 1/1 Running 0 12h 10.104.5.16 4am-node12 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-zookeeper-0 1/1 Running 0 12h 10.104.15.126 4am-node20 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-zookeeper-1 1/1 Running 0 12h 10.104.23.164 4am-node27 <none> <none>
multi-vector-corn-1-1717855200-5-pulsar-zookeeper-2 1/1 Running 0 12h 10.104.27.67 4am-node31 <none> <none>client pod name: multi-vector-corn-1-1717855200-1905225754 client logs:
[2024-06-08 23:56:11,858 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: grpc hybrid_search 1747 0(0.00%) | 74423 262 210855 73000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: grpc query 1833 0(0.00%) | 57963 226 338918 45000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: grpc scene_test 1779 0(0.00%) | 157595 63234 507876 143000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: grpc search 1710 0(0.00%) | 103265 15487 230572 93000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-08 23:56:11,858 - INFO - fouram]: Aggregated 7069 0(0.00%) | 98063 226 507876 86000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:11,859 - INFO - fouram]: (stats.py:790)
[2024-06-08 23:56:11,860 - INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: grpc hybrid_search 73000 88000 97000 104000 137000 162000 178000 193000 207000 211000 211000 1747 (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: grpc query 45000 76000 87000 96000 136000 172000 198000 215000 264000 339000 339000 1833 (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: grpc scene_test 143000 173000 199000 219000 269000 319000 383000 402000 487000 508000 508000 1779 (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: grpc search 94000 110000 124000 134000 164000 176000 191000 200000 227000 231000 231000 1710 (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: Aggregated 86000 107000 131000 146000 179000 224000 292000 340000 420000 508000 508000 7069 (stats.py:819)
[2024-06-08 23:56:11,860 - INFO - fouram]: (stats.py:820)
[2024-06-08 23:56:25,966 - ERROR - fouram]: RPC error: [batch_insert], <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest err:failed to get collection 450325548039573059
)>, <Time:{'RPC start': '2024-06-08 23:56:04.359194', 'RPC error': '2024-06-08 23:56:25.966155'}> (decorators.py:146)
[2024-06-08 23:56:26,023 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest err:failed to get collection 450325548039573059
)>, [requestId: aa6e6636-25f2-11ef-a57d-f63b1c27533e] (api_request.py:57)
[2024-06-08 23:56:26,023 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=65535, message=getSegmentID failed: SegmentIDAllocator failRemainRequest err:failed to get collection 450325548039573059
)> (func_check.py:54)
[2024-06-08 23:56:26,053 - ERROR - fouram]: [func_time_catch] : (api_request.py:127)
[2024-06-08 23:56:31,861 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-06-08 23:56:31,861 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-08 23:56:31,861 - INFO - fouram]: grpc hybrid_search 1747 0(0.00%) | 74423 262 210855 73000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:31,861 - INFO - fouram]: grpc query 1833 0(0.00%) | 57963 226 338918 45000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:31,862 - INFO - fouram]: grpc scene_test 1780 1(0.06%) | 157568 63234 507876 142000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:31,862 - INFO - fouram]: grpc search 1710 0(0.00%) | 103265 15487 230572 93000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:31,862 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-08 23:56:31,862 - INFO - fouram]: Aggregated 7070 1(0.01%) | 98065 226 507876 86000 | 0.00 0.00 (stats.py:789)
[2024-06-08 23:56:31,862 - INFO - fouram]: (stats.py:790)test result:
[2024-06-09 02:17:43,626 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-06-09 02:17:43,626 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-06-09 02:17:43,626 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: grpc hybrid_search 2159 0(0.00%) | 75780 262 254472 75000 | 0.05 0.00 (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: grpc query 2267 0(0.00%) | 60595 226 338918 50000 | 0.05 0.00 (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: grpc scene_test 2143 1(0.05%) | 159013 63234 507876 144000 | 0.05 0.00 (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: grpc search 2123 0(0.00%) | 103709 15487 230572 94000 | 0.05 0.00 (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: Aggregated 8692 1(0.01%) | 99162 226 507876 87000 | 0.20 0.00 (stats.py:789)
[2024-06-09 02:17:43,627 - INFO - fouram]: (stats.py:790)
[2024-06-09 02:17:43,630 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'cluster',
'config_name': 'cluster_2c8m',
'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '32Gi'},
'requests': {'cpu': '5.0',
'memory': '17Gi'}},
'replicas': 1},
'indexNode': {'resources': {'limits': {'cpu': '8.0',
'memory': '8Gi'},
'requests': {'cpu': '5.0',
'memory': '5Gi'}},
'replicas': 4},
'dataNode': {'resources': {'limits': {'cpu': '2.0',
'memory': '8Gi'},
'requests': {'cpu': '2.0',
'memory': '5Gi'}}},
'cluster': {'enabled': True},
'pulsar': {},
'kafka': {},
'minio': {'metrics': {'podMonitor': {'enabled': True}}},
'etcd': {'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': '2.4-20240605-443197bd-amd64'}}},
'host': 'multi-vector-corn-1-1717855200-5-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_hybrid_search_locust_ddl_dql_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {},
'int64_1': {'index_type': 'INVERTED'},
'varchar_1': {'index_type': 'INVERTED'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8,
'efConstruction': 200},
'metric_type': 'L2'},
'float_vector_2': {'index_type': 'DISKANN',
'index_param': {},
'metric_type': 'IP'},
'float_vector_3': {'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2'}},
'scalars_params': {'float_vector_1': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_2': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}},
'float_vector_3': {'params': {'dim': 128},
'other_params': {'dataset': 'sift',
'dim': 128}}},
'dataset_name': 'sift',
'dataset_size': 1000000,
'ni_per': 10000},
'collection_params': {'other_fields': ['float_vector_1',
'float_vector_2',
'float_vector_3',
'int64_1',
'varchar_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_test',
'weight': 1,
'params': {'dim': 128,
'data_size': 3000,
'nb': 3000,
'index_type': 'IVF_SQ8',
'index_param': {'nlist': 2048},
'metric_type': 'L2',
'other_fields': [],
'scalars_params': {},
'scalars_index': {},
'vectors_index': {}}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 1,
'search_param': {'nprobe': 1000},
'expr': 'int64_1 '
'>= '
'0',
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 600,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 100,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'100000',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'id '
'< '
'900000',
'top_k': 10},
{'search_param': {'search_list': 32},
'anns_field': 'float_vector_2',
'expr': 'varchar_1 '
'> '
'"1"',
'top_k': 30},
{'search_param': {'nprobe': 16},
'anns_field': 'float_vector_3',
'top_k': 400}],
'rerank': {'WeightedRanker': [0.85,
0.95,
0.51,
0.32]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 600,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&& ',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 600,
'random_data': True,
'random_count': 20,
'random_range': [0,
100000],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024060853971205,
'datetime': '2024-06-08 14:03:17.853801',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 78.5939,
'float_vector_1': {'RT': 36.2843},
'float_vector_2': {'RT': 166.7049},
'float_vector_3': {'RT': 17.1481},
'id': {'RT': 0.518},
'int64_1': {'RT': 0.5216},
'varchar_1': {'RT': 0.5221}},
'insert': {'total_time': 130.8271,
'VPS': 7643.6763,
'batch_time': 1.3083,
'batch': 10000},
'flush': {'RT': 2.5331},
'load': {'RT': 7.0602},
'Locust': {'Aggregated': {'Requests': 8692,
'Fails': 1,
'RPS': 0.2,
'fail_s': 0.0,
'RT_max': 507876.8,
'RT_avg': 99162.4,
'TP50': 87000.0,
'TP99': 338000.0},
'hybrid_search': {'Requests': 2159,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 254472.81,
'RT_avg': 75780.24,
'TP50': 75000.0,
'TP99': 192000.0},
'query': {'Requests': 2267,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 338918.8,
'RT_avg': 60595.35,
'TP50': 50000.0,
'TP99': 222000.0},
'scene_test': {'Requests': 2143,
'Fails': 1,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 507876.8,
'RT_avg': 159013.32,
'TP50': 144000.0,
'TP99': 400000.0},
'search': {'Requests': 2123,
'Fails': 0,
'RPS': 0.05,
'fail_s': 0.0,
'RT_max': 230572.96,
'RT_avg': 103709.28,
'TP50': 94000.0,
'TP99': 199000.0}}}}}test steps:
concurrent test and calculation of RT and QPS
:purpose: `DDL & DQL`
verify DDL & DQL scenario,
which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'float_vector_1': 128dim,
'float_vector_2': 128dim,
'float_vector_3': 128dim,
scalar field: int64_1, varchar_1
2. build indexes:
IVF_FLAT: 'float_vector'
HNSW: 'float_vector_1',
DISKANN: 'float_vector_2'
IVF_SQ8: 'float_vector_3'
INVERTED: 'int64_1', 'varchar_1'
default scalar index: 'id'
3. insert 1 million data
4. flush collection
5. build indexes again using the same params
6. load collection
replica: 1
7. concurrent request:
- scene_test <- Occasionally insertion failure
(collection: create->insert->flush->index->drop)
- search
- hybrid_search
- query
Is there an existing issue for this?
Environment
Current Behavior
argo task: multi-vector-scene-mix-lpwcq test case name: test_hybrid_search_locust_ddl_dql_cluster
server:
client log: client.log
test result:
Expected Behavior
No response
Steps To Reproduce
Milvus Log
No response
Anything else?
No response