wangting0128
commented
4 months ago Different case, same panic
argo task: inverted-corn-1719936000 test case name: test_inverted_locust_partitions_dml_dql_standalone image: 2.4-20240702-6cb0f1ff-amd64
server:
[2024-07-02 19:18:17,135 - INFO - fouram]: [Base] Deploy initial state:
I0702 16:09:27.070528 397 request.go:665] Waited for 1.178985568s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/events.k8s.io/v1beta1?timeout=32s
I0702 16:09:37.270151 397 request.go:665] Waited for 6.997772679s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/discovery.k8s.io/v1beta1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inverted-corn-136000-3-99-3725-etcd-0 1/1 Running 0 4m22s 10.104.26.88 4am-node32 <none> <none>
inverted-corn-136000-3-99-3725-milvus-standalone-7bb77b9f5prm66 1/1 Running 3 (2m27s ago) 4m23s 10.104.20.92 4am-node22 <none> <none>
inverted-corn-136000-3-99-3725-minio-7969c556f4-xhqvl 1/1 Running 0 4m23s 10.104.26.86 4am-node32 <none> <none> (base.py:261)
[2024-07-02 19:18:17,135 - INFO - fouram]: [Cmd Exe] kubectl get pods -n qa-milvus -o wide | grep -E 'NAME|inverted-corn-136000-3-99-3725-milvus|inverted-corn-136000-3-99-3725-minio|inverted-corn-136000-3-99-3725-etcd|inverted-corn-136000-3-99-3725-pulsar|inverted-corn-136000-3-99-3725-zookeeper|inverted-corn-136000-3-99-3725-kafka|inverted-corn-136000-3-99-3725-log|inverted-corn-136000-3-99-3725-tikv' (util_cmd.py:14)
[2024-07-02 19:18:33,655 - INFO - fouram]: [CliClient] pod details of release(inverted-corn-136000-3-99-3725):
I0702 19:18:18.409536 541 request.go:665] Waited for 1.14581067s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/policy/v1?timeout=32s
I0702 19:18:28.409741 541 request.go:665] Waited for 6.798684072s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/notification.kubesphere.io/v2beta2?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inverted-corn-136000-3-99-3725-etcd-0 1/1 Running 0 3h13m 10.104.26.88 4am-node32 <none> <none>
inverted-corn-136000-3-99-3725-milvus-standalone-7bb77b9f5prm66 1/1 Running 6 (30m ago) 3h13m 10.104.20.92 4am-node22 <none> <none>
inverted-corn-136000-3-99-3725-minio-7969c556f4-xhqvl 1/1 Running 0 3h13m 10.104.26.86 4am-node32 <none> <none> (cli_client.py:144)client pod name: inverted-corn-1719936000-4134836050
test steps:
concurrent test and calculation of RT and QPS
:purpose: `partition: collection has many partitions`
verify concurrent DML & DQL scenario which
scalar `id`(pk) & `int64_1` created INVERTED index and collection has 10 partitions
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'int64_1'
2. build indexes:
IVF_FLAT: 'float_vector'
INVERTED: 'id', 'int64_1'
3. insert 5 million data to 10 partitions
4. flush collection
5. build indexes again using the same params
6. load collection
7. concurrent request:
- insert
- delete
- flush
- load
- search
- hybrid_search
- querytest result:
[2024-07-02 19:17:43,140 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-07-02 19:17:43,141 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-07-02 19:17:43,141 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-07-02 19:17:43,141 - INFO - fouram]: grpc delete 3424 37(1.08%) | 343 0 30961 6 | 0.32 0.00 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: grpc flush 3378 145(4.29%) | 38959 287 272989 11000 | 0.31 0.01 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: grpc hybrid_search 3382 16(0.47%) | 9960 287 110941 8100 | 0.31 0.00 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: grpc insert 3414 27(0.79%) | 408 1 32701 58 | 0.32 0.00 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: grpc load 3383 5(0.15%) | 770 3 32372 99 | 0.31 0.00 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: grpc query 3435 0(0.00%) | 5776 97 164545 3900 | 0.32 0.00 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: grpc search 3416 0(0.00%) | 7367 382 160782 5500 | 0.32 0.00 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: Aggregated 23832 230(0.97%) | 9041 0 272989 1700 | 2.21 0.02 (stats.py:789)
[2024-07-02 19:17:43,142 - INFO - fouram]: (stats.py:790)
[2024-07-02 19:17:43,146 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'standalone',
'config_name': 'standalone_16c16m',
'config': {'standalone': {'resources': {'limits': {'cpu': '16.0',
'memory': '16Gi'},
'requests': {'cpu': '9.0',
'memory': '9Gi'}}},
'cluster': {'enabled': False},
'etcd': {'replicaCount': 1,
'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'minio': {'mode': 'standalone',
'metrics': {'podMonitor': {'enabled': True}}},
'pulsar': {'enabled': False},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': '2.4-20240702-6cb0f1ff-amd64'}}},
'host': 'inverted-corn-136000-3-99-3725-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_inverted_locust_partitions_dml_dql_standalone',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {'index_type': 'INVERTED'},
'int64_1': {'index_type': 'INVERTED'}},
'extra_partitions': {'partitions': 10,
'data_repeated': False},
'dataset_name': 'sift',
'dataset_size': 5000000,
'ni_per': 50000},
'collection_params': {'other_fields': ['int64_1'],
'shards_num': 2},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '3h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'insert',
'weight': 1,
'params': {'nb': 10,
'timeout': 30,
'random_id': True,
'random_vector': True,
'varchar_filled': False,
'start_id': 5000000}},
{'type': 'delete',
'weight': 1,
'params': {'expr': '',
'delete_length': 9,
'timeout': 30}},
{'type': 'flush',
'weight': 1,
'params': {'timeout': 180}},
{'type': 'load',
'weight': 1,
'params': {'replica_number': 1,
'timeout': 30}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 10,
'search_param': {'nprobe': 16},
'expr': None,
'guarantee_timestamp': None,
'partition_names': None,
'output_fields': None,
'ignore_growing': False,
'group_by_field': None,
'timeout': 180,
'random_data': True}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 10,
'reqs': [{'search_param': {'nprobe': 16},
'anns_field': 'float_vector',
'top_k': 2000},
{'search_param': {'nprobe': 32},
'anns_field': 'float_vector',
'expr': 'int64_1 '
'> '
'-1 '
'&& '
'id '
'> '
'-1'},
{'search_param': {'nprobe': 64},
'anns_field': 'float_vector',
'expr': 'id '
'> '
'10',
'top_k': 60}],
'rerank': {'WeightedRanker': [0.3,
0.4,
0.3]},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': None,
'timeout': 60,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 '
'> '
'-1 '
'&&',
'output_fields': ['*'],
'offset': None,
'limit': None,
'ignore_growing': False,
'partition_names': None,
'timeout': 180,
'random_data': True,
'random_count': 20,
'random_range': [0,
1000000.0],
'field_name': 'id',
'field_type': 'int64'}}]},
'run_id': 2024070263168668,
'datetime': '2024-07-02 16:05:16.951633',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 295.4749,
'id': {'RT': 1.0141},
'int64_1': {'RT': 0.5106}},
'insert': {'total_time': 154.4347,
'VPS': 32409.404,
'batch_time': 1.5443,
'batch': 50000.0},
'flush': {'RT': 3.029},
'load': {'RT': 3.0301},
'Locust': {'Aggregated': {'Requests': 23832,
'Fails': 230,
'RPS': 2.21,
'fail_s': 0.01,
'RT_max': 272989.31,
'RT_avg': 9041.73,
'TP50': 1700.0,
'TP99': 157000.0},
'delete': {'Requests': 3424,
'Fails': 37,
'RPS': 0.32,
'fail_s': 0.01,
'RT_max': 30961.56,
'RT_avg': 343.67,
'TP50': 6,
'TP99': 7600.0},
'flush': {'Requests': 3378,
'Fails': 145,
'RPS': 0.31,
'fail_s': 0.04,
'RT_max': 272989.31,
'RT_avg': 38959.86,
'TP50': 11000.0,
'TP99': 182000.0},
'hybrid_search': {'Requests': 3382,
'Fails': 16,
'RPS': 0.31,
'fail_s': 0.0,
'RT_max': 110941.34,
'RT_avg': 9960.62,
'TP50': 8100.0,
'TP99': 41000.0},
'insert': {'Requests': 3414,
'Fails': 27,
'RPS': 0.32,
'fail_s': 0.01,
'RT_max': 32701.55,
'RT_avg': 408.27,
'TP50': 58,
'TP99': 7700.0},
'load': {'Requests': 3383,
'Fails': 5,
'RPS': 0.31,
'fail_s': 0.0,
'RT_max': 32372.1,
'RT_avg': 770.34,
'TP50': 99,
'TP99': 13000.0},
'query': {'Requests': 3435,
'Fails': 0,
'RPS': 0.32,
'fail_s': 0.0,
'RT_max': 164545.4,
'RT_avg': 5776.95,
'TP50': 3900.0,
'TP99': 29000.0},
'search': {'Requests': 3416,
'Fails': 0,
'RPS': 0.32,
'fail_s': 0.0,
'RT_max': 160782.19,
'RT_avg': 7367.94,
'TP50': 5500.0,
'TP99': 26000.0}}}}}
weiliu1031
xiaocai2333
Is there an existing issue for this?
Environment
Current Behavior
argo task: inverted-corn-1719936000 test case name: test_inverted_locust_partition_key_dml_standalone
server:
client pod name: inverted-corn-1719936000-1616785879
Expected Behavior
No response
Steps To Reproduce
Milvus Log
No response
Anything else?
test result: