milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.36k stars 2.91k forks source link

[Bug]: [benchmark][cluster] flush 180s timeout in concurrent DML and DQL scenario #33551

Open wangting0128 opened 5 months ago

wangting0128 commented 5 months ago

Is there an existing issue for this?

Environment

- Milvus version:2.4-20240531-e68f0bf7-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-corn-1-1717336800 test case name: test_hybrid_search_locust_dml_dql_cluster

server:

NAME                                                              READY   STATUS             RESTARTS           AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-1-1717336800-4-etcd-0                           1/1     Running            0                  12h     10.104.23.227   4am-node27   <none>           <none>
multi-vector-corn-1-1717336800-4-etcd-1                           1/1     Running            0                  12h     10.104.16.38    4am-node21   <none>           <none>
multi-vector-corn-1-1717336800-4-etcd-2                           1/1     Running            0                  12h     10.104.17.250   4am-node23   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-datacoord-5d5455fd7pjhx   1/1     Running            4 (12h ago)        12h     10.104.6.243    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-datanode-7dd9555f6zmxzz   1/1     Running            5 (12h ago)        12h     10.104.20.53    4am-node22   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-indexcoord-594cfb7wkj9c   1/1     Running            0                  12h     10.104.6.240    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-indexnode-58f67b949mfhd   1/1     Running            4 (12h ago)        12h     10.104.4.5      4am-node11   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-proxy-c46ff5bdd-ds8z8     1/1     Running            5 (12h ago)        12h     10.104.4.6      4am-node11   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-querycoord-7966b6598xlz   1/1     Running            5 (12h ago)        12h     10.104.6.245    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-querynode-dbb98779kcj49   1/1     Running            4 (12h ago)        12h     10.104.14.66    4am-node18   <none>           <none>
multi-vector-corn-1-1717336800-4-milvus-rootcoord-7db5fd77jhdkw   1/1     Running            5 (12h ago)        12h     10.104.6.244    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-minio-0                          1/1     Running            0                  12h     10.104.23.231   4am-node27   <none>           <none>
multi-vector-corn-1-1717336800-4-minio-1                          1/1     Running            0                  12h     10.104.16.41    4am-node21   <none>           <none>
multi-vector-corn-1-1717336800-4-minio-2                          1/1     Running            0                  12h     10.104.21.166   4am-node24   <none>           <none>
multi-vector-corn-1-1717336800-4-minio-3                          1/1     Running            0                  12h     10.104.17.251   4am-node23   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-bookie-0                  1/1     Running            0                  12h     10.104.17.249   4am-node23   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-bookie-1                  1/1     Running            0                  12h     10.104.23.232   4am-node27   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-bookie-2                  1/1     Running            0                  12h     10.104.16.43    4am-node21   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-bookie-init-5snwn         0/1     Completed          0                  12h     10.104.6.242    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-broker-0                  1/1     Running            0                  12h     10.104.6.246    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-proxy-0                   1/1     Running            0                  12h     10.104.16.12    4am-node21   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-pulsar-init-tpqzs         0/1     Completed          0                  12h     10.104.6.241    4am-node13   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-recovery-0                1/1     Running            0                  12h     10.104.13.242   4am-node16   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-zookeeper-0               1/1     Running            0                  12h     10.104.17.246   4am-node23   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-zookeeper-1               1/1     Running            0                  12h     10.104.21.182   4am-node24   <none>           <none>
multi-vector-corn-1-1717336800-4-pulsar-zookeeper-2               1/1     Running            0                  12h     10.104.23.234   4am-node27   <none>           <none>
截屏2024-06-03 14 39 01 截屏2024-06-03 14 39 26 截屏2024-06-03 14 39 56 截屏2024-06-03 14 40 48 截屏2024-06-03 14 41 12

client pod name: multi-vector-corn-1-1717336800-3646130279 client logs: flush timeout

截屏2024-06-03 14 37 58 截屏2024-06-03 14 38 24

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DML & DQL`
            verify DML & DQL scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 1 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - insert
                - delete
                - flush
                - load
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-06-03 02:37:38,736 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-06-03 02:37:38,736 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-03 02:37:38,736 -  INFO - fouram]: grpc     delete                                                                          1680     0(0.00%) |  24047      10  187159   1300 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,736 -  INFO - fouram]: grpc     flush                                                                           1662    21(1.26%) |  72008     239  301557  67000 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,736 -  INFO - fouram]: grpc     hybrid_search                                                                   1748     0(0.00%) | 115893     393  347840 112000 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]: grpc     insert                                                                          1757     0(0.00%) |  31808     153  205137  24000 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]: grpc     load                                                                            1760     0(0.00%) |  42734       7  261663  34000 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]: grpc     query                                                                           1713     0(0.00%) | 101833     764  301754  99000 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]: grpc     search                                                                          1771     0(0.00%) | 109410   21449  259720 107000 |    0.04        0.00 (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]:          Aggregated                                                                     12091    21(0.17%) |  71289       7  347840  68000 |    0.28        0.00 (stats.py:789)
[2024-06-03 02:37:38,737 -  INFO - fouram]:  (stats.py:790)
[2024-06-03 02:37:38,744 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '64Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '33Gi'}},
                                     'replicas': 1},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '8Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '5Gi'}},
                                     'replicas': 1},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '8Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240531-e68f0bf7-amd64'}}},
            'host': 'multi-vector-corn-1-1717336800-4-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dml_dql_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {},
                                                                      'int64_1': {'index_type': 'INVERTED'},
                                                                      'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int64_1',
                                                                        'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT',
                                                  'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 1000,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 1000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1000,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 180}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 600}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 '
                                                                          '>= '
                                                                          '0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 '
                                                                                    '> '
                                                                                    '-1',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'id '
                                                                                    '> '
                                                                                    '-1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'search_list': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'varchar_1 '
                                                                                    '> '
                                                                                    '"1"',
                                                                            'top_k': 30},
                                                                           {'search_param': {'nprobe': 16},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85,
                                                                                                0.95,
                                                                                                0.51,
                                                                                                0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0,
                                                                                   100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024060269846053,
            'datetime': '2024-06-02 14:03:04.903874',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 475.2406,
                                      'float_vector_1': {'RT': 299.8403},
                                      'float_vector_2': {'RT': 211.2743},
                                      'float_vector_3': {'RT': 266.0751},
                                      'id': {'RT': 114.0471},
                                      'int64_1': {'RT': 1.0205},
                                      'varchar_1': {'RT': 1.0227}},
                            'insert': {'total_time': 166.3071,
                                       'VPS': 6012.9724,
                                       'batch_time': 1.6631,
                                       'batch': 10000},
                            'flush': {'RT': 2.5344},
                            'load': {'RT': 11.1059},
                            'Locust': {'Aggregated': {'Requests': 12091,
                                                      'Fails': 21,
                                                      'RPS': 0.28,
                                                      'fail_s': 0.0,
                                                      'RT_max': 347840.61,
                                                      'RT_avg': 71289.89,
                                                      'TP50': 68000.0,
                                                      'TP99': 219000.0},
                                       'delete': {'Requests': 1680,
                                                  'Fails': 0,
                                                  'RPS': 0.04,
                                                  'fail_s': 0.0,
                                                  'RT_max': 187159.14,
                                                  'RT_avg': 24047.11,
                                                  'TP50': 1300.0,
                                                  'TP99': 143000.0},
                                       'flush': {'Requests': 1662,
                                                 'Fails': 21,
                                                 'RPS': 0.04,
                                                 'fail_s': 0.01,
                                                 'RT_max': 301557.21,
                                                 'RT_avg': 72008.21,
                                                 'TP50': 67000.0,
                                                 'TP99': 210000.0},
                                       'hybrid_search': {'Requests': 1748,
                                                         'Fails': 0,
                                                         'RPS': 0.04,
                                                         'fail_s': 0.0,
                                                         'RT_max': 347840.61,
                                                         'RT_avg': 115893.29,
                                                         'TP50': 112000.0,
                                                         'TP99': 249000.0},
                                       'insert': {'Requests': 1757,
                                                  'Fails': 0,
                                                  'RPS': 0.04,
                                                  'fail_s': 0.0,
                                                  'RT_max': 205137.59,
                                                  'RT_avg': 31808.65,
                                                  'TP50': 24000.0,
                                                  'TP99': 140000.0},
                                       'load': {'Requests': 1760,
                                                'Fails': 0,
                                                'RPS': 0.04,
                                                'fail_s': 0.0,
                                                'RT_max': 261663.25,
                                                'RT_avg': 42734.48,
                                                'TP50': 34000.0,
                                                'TP99': 180000.0},
                                       'query': {'Requests': 1713,
                                                 'Fails': 0,
                                                 'RPS': 0.04,
                                                 'fail_s': 0.0,
                                                 'RT_max': 301754.98,
                                                 'RT_avg': 101833.75,
                                                 'TP50': 99000.0,
                                                 'TP99': 240000.0},
                                       'search': {'Requests': 1771,
                                                  'Fails': 0,
                                                  'RPS': 0.04,
                                                  'fail_s': 0.0,
                                                  'RT_max': 259720.54,
                                                  'RT_avg': 109410.58,
                                                  'TP50': 107000.0,
                                                  'TP99': 210000.0}}}}}
yanliang567 commented 5 months ago

/assign @XuanYang-cn /unassign

wangting0128 commented 5 months ago

Recurrent

argo task: multi-vector-corn-1-1717682400 test case name: test_hybrid_search_locust_dml_dql_cluster image: 2.4-20240605-443197bd-amd64

server:

NAME                                                              READY   STATUS        RESTARTS           AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-1-1717682400-4-etcd-0                           1/1     Running       0                  12h     10.104.23.78    4am-node27   <none>           <none>
multi-vector-corn-1-1717682400-4-etcd-1                           1/1     Running       0                  12h     10.104.27.92    4am-node31   <none>           <none>
multi-vector-corn-1-1717682400-4-etcd-2                           1/1     Running       0                  12h     10.104.25.222   4am-node30   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-datacoord-9bc4947dpvvgp   1/1     Running       5 (12h ago)        12h     10.104.13.183   4am-node16   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-datanode-744fc96dfpnvb2   1/1     Running       5 (12h ago)        12h     10.104.4.203    4am-node11   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-indexcoord-5f5bd45bfggc   1/1     Running       0                  12h     10.104.23.39    4am-node27   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-indexnode-cfbff6b9pqt96   1/1     Running       5 (12h ago)        12h     10.104.1.135    4am-node10   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-proxy-6f896db784-ldk47    1/1     Running       5 (12h ago)        12h     10.104.13.184   4am-node16   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-querycoord-788885ctgs44   1/1     Running       5 (12h ago)        12h     10.104.13.185   4am-node16   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-querynode-5ccd885fnwc8h   1/1     Running       5 (12h ago)        12h     10.104.18.9     4am-node25   <none>           <none>
multi-vector-corn-1-1717682400-4-milvus-rootcoord-6f6c487cf6pq2   1/1     Running       5 (12h ago)        12h     10.104.4.202    4am-node11   <none>           <none>
multi-vector-corn-1-1717682400-4-minio-0                          1/1     Running       0                  12h     10.104.25.215   4am-node30   <none>           <none>
multi-vector-corn-1-1717682400-4-minio-1                          1/1     Running       0                  12h     10.104.23.77    4am-node27   <none>           <none>
multi-vector-corn-1-1717682400-4-minio-2                          1/1     Running       0                  12h     10.104.15.43    4am-node20   <none>           <none>
multi-vector-corn-1-1717682400-4-minio-3                          1/1     Running       0                  12h     10.104.27.91    4am-node31   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-0                  1/1     Running       0                  12h     10.104.15.38    4am-node20   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-1                  1/1     Running       0                  12h     10.104.25.218   4am-node30   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-2                  1/1     Running       0                  12h     10.104.21.210   4am-node24   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-bookie-init-57gl2         0/1     Completed     0                  12h     10.104.4.200    4am-node11   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-broker-0                  1/1     Running       0                  12h     10.104.23.40    4am-node27   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-proxy-0                   1/1     Running       0                  12h     10.104.13.186   4am-node16   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-pulsar-init-8mbtk         0/1     Completed     0                  12h     10.104.4.199    4am-node11   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-recovery-0                1/1     Running       0                  12h     10.104.14.241   4am-node18   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-zookeeper-0               1/1     Running       0                  12h     10.104.27.81    4am-node31   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-zookeeper-1               1/1     Running       0                  12h     10.104.21.217   4am-node24   <none>           <none>
multi-vector-corn-1-1717682400-4-pulsar-zookeeper-2               1/1     Running       0                  12h     10.104.23.85    4am-node27   <none>           <none>

client pod name: multi-vector-corn-1-1717682400-1777966671 client logs:

截屏2024-06-07 14 22 32

test result:

[2024-06-07 02:38:11,307 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-06-07 02:38:11,308 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     delete                                                                          2152     0(0.00%) |  20745      10  254032   1900 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     flush                                                                           2201     3(0.14%) |  53477     281  284799  47000 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     hybrid_search                                                                   2253     0(0.00%) |  92728     557  291654  89000 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     insert                                                                          2236     0(0.00%) |  23130     157  145139  16000 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     load                                                                            2264     0(0.00%) |  34656       7  284603  26000 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     query                                                                           2213     0(0.00%) |  77643     210  315524  74000 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: grpc     search                                                                          2239     0(0.00%) |  84631   16585  237241  81000 |    0.05        0.00 (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-06-07 02:38:11,308 -  INFO - fouram]:          Aggregated                                                                     15558     3(0.02%) |  55454       7  315524  51000 |    0.36        0.00 (stats.py:789)
[2024-06-07 02:38:11,309 -  INFO - fouram]:  (stats.py:790)
[2024-06-07 02:38:11,312 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '64Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '33Gi'}},
                                     'replicas': 1},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '8Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '5Gi'}},
                                     'replicas': 1},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '8Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240605-443197bd-amd64'}}},
            'host': 'multi-vector-corn-1-1717682400-4-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dml_dql_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {},
                                                                      'int64_1': {'index_type': 'INVERTED'},
                                                                      'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int64_1',
                                                                        'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT',
                                                  'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 1000,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 1000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1000,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 180}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 600}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 '
                                                                          '>= '
                                                                          '0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 '
                                                                                    '> '
                                                                                    '-1',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'id '
                                                                                    '> '
                                                                                    '-1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'search_list': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'varchar_1 '
                                                                                    '> '
                                                                                    '"1"',
                                                                            'top_k': 30},
                                                                           {'search_param': {'nprobe': 16},
                                                                            'anns_field': 'float_vector_3',
                                                                            'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85,
                                                                                                0.95,
                                                                                                0.51,
                                                                                                0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0,
                                                                                   100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024060625304491,
            'datetime': '2024-06-06 14:02:10.758994',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 461.1983,
                                      'float_vector_1': {'RT': 345.9712},
                                      'float_vector_2': {'RT': 263.8506},
                                      'float_vector_3': {'RT': 241.1576},
                                      'id': {'RT': 282.8067},
                                      'int64_1': {'RT': 1.0222},
                                      'varchar_1': {'RT': 1.0199}},
                            'insert': {'total_time': 158.9733,
                                       'VPS': 6290.3645,
                                       'batch_time': 1.5897,
                                       'batch': 10000},
                            'flush': {'RT': 2.5245},
                            'load': {'RT': 7.0519},
                            'Locust': {'Aggregated': {'Requests': 15558,
                                                      'Fails': 3,
                                                      'RPS': 0.36,
                                                      'fail_s': 0.0,
                                                      'RT_max': 315524.01,
                                                      'RT_avg': 55454.58,
                                                      'TP50': 51000.0,
                                                      'TP99': 176000.0},
                                       'delete': {'Requests': 2152,
                                                  'Fails': 0,
                                                  'RPS': 0.05,
                                                  'fail_s': 0.0,
                                                  'RT_max': 254032.74,
                                                  'RT_avg': 20745.53,
                                                  'TP50': 1900.0,
                                                  'TP99': 111000.0},
                                       'flush': {'Requests': 2201,
                                                 'Fails': 3,
                                                 'RPS': 0.05,
                                                 'fail_s': 0.0,
                                                 'RT_max': 284799.27,
                                                 'RT_avg': 53477.73,
                                                 'TP50': 47000.0,
                                                 'TP99': 169000.0},
                                       'hybrid_search': {'Requests': 2253,
                                                         'Fails': 0,
                                                         'RPS': 0.05,
                                                         'fail_s': 0.0,
                                                         'RT_max': 291654.94,
                                                         'RT_avg': 92728.76,
                                                         'TP50': 89000.0,
                                                         'TP99': 208000.0},
                                       'insert': {'Requests': 2236,
                                                  'Fails': 0,
                                                  'RPS': 0.05,
                                                  'fail_s': 0.0,
                                                  'RT_max': 145139.42,
                                                  'RT_avg': 23130.14,
                                                  'TP50': 16000.0,
                                                  'TP99': 107000.0},
                                       'load': {'Requests': 2264,
                                                'Fails': 0,
                                                'RPS': 0.05,
                                                'fail_s': 0.0,
                                                'RT_max': 284603.66,
                                                'RT_avg': 34656.54,
                                                'TP50': 26000.0,
                                                'TP99': 155000.0},
                                       'query': {'Requests': 2213,
                                                 'Fails': 0,
                                                 'RPS': 0.05,
                                                 'fail_s': 0.0,
                                                 'RT_max': 315524.01,
                                                 'RT_avg': 77643.42,
                                                 'TP50': 74000.0,
                                                 'TP99': 186000.0},
                                       'search': {'Requests': 2239,
                                                  'Fails': 0,
                                                  'RPS': 0.05,
                                                  'fail_s': 0.0,
                                                  'RT_max': 237241.91,
                                                  'RT_avg': 84631.22,
                                                  'TP50': 81000.0,
                                                  'TP99': 170000.0}}}}}
XuanYang-cn commented 5 months ago

https://github.com/milvus-io/milvus/issues/30529#issuecomment-2160159587