search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.35k stars 2.91k forks source link

[Bug]: [benchmark][cluster] queryNode panic due to `locked to thread` #30358

Closed wangting0128 closed 8 months ago

wangting0128 commented 9 months ago

Is there an existing issue for this?

Environment

- Milvus version:master-20240129-74cfba02
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0rc19
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: inverted-corn-1706544000 test case name: test_inverted_locust_hnsw_diskann_dml_dql_cluster

server:

[2024-01-29 20:11:29,657 -  INFO - fouram]: [Base] Deploy initial state: 
I0129 16:08:17.892497     406 request.go:665] Waited for 1.177176765s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/rbac.authorization.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
inverted-corn-144000-8-85-6291-etcd-0                             1/1     Running            0                6m22s   10.104.18.91    4am-node25   <none>           <none>
inverted-corn-144000-8-85-6291-etcd-1                             1/1     Running            0                6m21s   10.104.19.233   4am-node28   <none>           <none>
inverted-corn-144000-8-85-6291-etcd-2                             1/1     Running            0                6m21s   10.104.27.148   4am-node31   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-datacoord-5596c5654-xdpfw   1/1     Running            0                6m23s   10.104.4.182    4am-node11   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-datanode-5cf4d7f585-qdqlb   1/1     Running            1 (112s ago)     6m22s   10.104.14.129   4am-node18   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexcoord-8d49c9577v5rl6   1/1     Running            0                6m22s   10.104.24.168   4am-node29   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8769sb   1/1     Running            0                6m23s   10.104.29.53    4am-node35   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8chngt   1/1     Running            0                6m23s   10.104.28.104   4am-node33   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8fr8mf   1/1     Running            0                6m23s   10.104.4.183    4am-node11   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8q8bpq   1/1     Running            0                6m22s   10.104.15.79    4am-node20   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-proxy-d68566657-xttbh       1/1     Running            1 (111s ago)     6m22s   10.104.23.194   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-querycoord-64cfc4485plnjt   1/1     Running            1 (112s ago)     6m23s   10.104.6.130    4am-node13   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-querynode-588dbd9b6ddvwqn   1/1     Running            0                6m23s   10.104.6.131    4am-node13   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-querynode-588dbd9b6dwl86m   1/1     Running            0                6m23s   10.104.16.120   4am-node21   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-rootcoord-6659899db6xwx6j   1/1     Running            1 (111s ago)     6m23s   10.104.23.193   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-minio-0                            1/1     Running            0                6m22s   10.104.19.231   4am-node28   <none>           <none>
inverted-corn-144000-8-85-6291-minio-1                            1/1     Running            0                6m21s   10.104.21.214   4am-node24   <none>           <none>
inverted-corn-144000-8-85-6291-minio-2                            1/1     Running            0                6m21s   10.104.34.95    4am-node37   <none>           <none>
inverted-corn-144000-8-85-6291-minio-3                            1/1     Running            0                6m20s   10.104.31.48    4am-node34   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-0                    1/1     Running            0                6m20s   10.104.21.215   4am-node24   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-1                    1/1     Running            0                6m20s   10.104.23.211   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-2                    1/1     Running            0                6m19s   10.104.34.103   4am-node37   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-init-cwwz2           0/1     Completed          0                6m23s   10.104.14.128   4am-node18   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-broker-0                    1/1     Running            0                6m21s   10.104.24.171   4am-node29   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-proxy-0                     1/1     Running            0                6m22s   10.104.27.137   4am-node31   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-pulsar-init-ds89x           0/1     Completed          0                6m23s   10.104.15.80    4am-node20   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-recovery-0                  1/1     Running            0                6m22s   10.104.23.195   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-zookeeper-0                 1/1     Running            0                6m22s   10.104.18.90    4am-node25   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-zookeeper-1                 1/1     Running            0                4m55s   10.104.21.231   4am-node24   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-zookeeper-2                 1/1     Running            0                4m      10.104.31.53    4am-node34   <none>           <none> (base.py:257)
[2024-01-29 20:11:29,657 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|inverted-corn-144000-8-85-6291-milvus|inverted-corn-144000-8-85-6291-minio|inverted-corn-144000-8-85-6291-etcd|inverted-corn-144000-8-85-6291-pulsar|inverted-corn-144000-8-85-6291-kafka|inverted-corn-144000-8-85-6291-log|inverted-corn-144000-8-85-6291-tikv'  (util_cmd.py:14)
[2024-01-29 20:11:39,489 -  INFO - fouram]: [CliClient] pod details of release(inverted-corn-144000-8-85-6291): 
 I0129 20:11:30.909844     530 request.go:665] Waited for 1.164247068s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/autoscaling/v1?timeout=32s
NAME                                                              READY   STATUS        RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
inverted-corn-144000-8-85-6291-etcd-0                             1/1     Running       0                4h9m    10.104.18.91    4am-node25   <none>           <none>
inverted-corn-144000-8-85-6291-etcd-1                             1/1     Running       0                4h9m    10.104.19.233   4am-node28   <none>           <none>
inverted-corn-144000-8-85-6291-etcd-2                             1/1     Running       0                4h9m    10.104.27.148   4am-node31   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-datacoord-5596c5654-xdpfw   1/1     Running       0                4h9m    10.104.4.182    4am-node11   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-datanode-5cf4d7f585-qdqlb   1/1     Running       1 (4h5m ago)     4h9m    10.104.14.129   4am-node18   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexcoord-8d49c9577v5rl6   1/1     Running       0                4h9m    10.104.24.168   4am-node29   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8769sb   1/1     Running       0                4h9m    10.104.29.53    4am-node35   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8chngt   1/1     Running       0                4h9m    10.104.28.104   4am-node33   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8fr8mf   1/1     Running       0                4h9m    10.104.4.183    4am-node11   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-indexnode-7df75bbff8q8bpq   1/1     Running       0                4h9m    10.104.15.79    4am-node20   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-proxy-d68566657-xttbh       1/1     Running       1 (4h5m ago)     4h9m    10.104.23.194   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-querycoord-64cfc4485plnjt   1/1     Running       1 (4h5m ago)     4h9m    10.104.6.130    4am-node13   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-querynode-588dbd9b6ddvwqn   1/1     Running       0                4h9m    10.104.6.131    4am-node13   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-querynode-588dbd9b6dwl86m   1/1     Running       1 (154m ago)     4h9m    10.104.16.120   4am-node21   <none>           <none>
inverted-corn-144000-8-85-6291-milvus-rootcoord-6659899db6xwx6j   1/1     Running       1 (4h5m ago)     4h9m    10.104.23.193   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-minio-0                            1/1     Running       0                4h9m    10.104.19.231   4am-node28   <none>           <none>
inverted-corn-144000-8-85-6291-minio-1                            1/1     Running       0                4h9m    10.104.21.214   4am-node24   <none>           <none>
inverted-corn-144000-8-85-6291-minio-2                            1/1     Running       0                4h9m    10.104.34.95    4am-node37   <none>           <none>
inverted-corn-144000-8-85-6291-minio-3                            1/1     Running       0                4h9m    10.104.31.48    4am-node34   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-0                    1/1     Running       0                4h9m    10.104.21.215   4am-node24   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-1                    1/1     Running       0                4h9m    10.104.23.211   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-2                    1/1     Running       0                4h9m    10.104.34.103   4am-node37   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-bookie-init-cwwz2           0/1     Completed     0                4h9m    10.104.14.128   4am-node18   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-broker-0                    1/1     Running       0                4h9m    10.104.24.171   4am-node29   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-proxy-0                     1/1     Running       0                4h9m    10.104.27.137   4am-node31   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-pulsar-init-ds89x           0/1     Completed     0                4h9m    10.104.15.80    4am-node20   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-recovery-0                  1/1     Running       0                4h9m    10.104.23.195   4am-node27   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-zookeeper-0                 1/1     Running       0                4h9m    10.104.18.90    4am-node25   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-zookeeper-1                 1/1     Running       0                4h8m    10.104.21.231   4am-node24   <none>           <none>
inverted-corn-144000-8-85-6291-pulsar-zookeeper-2                 1/1     Running       0                4h7m    10.104.31.53    4am-node34   <none>           <none>

image

client pod name: inverted-corn-1706544000-319352182 client log:

截屏2024-01-30 12 21 57 截屏2024-01-30 12 22 39

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_8c16m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '16.0',
                                                              'memory': '64Gi'},
                                                   'requests': {'cpu': '9.0',
                                                                'memory': '33Gi'}},
                                     'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '16Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '9Gi'}},
                                     'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '8.0',
                                                             'memory': '16Gi'},
                                                  'requests': {'cpu': '5.0',
                                                               'memory': '9Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'master-20240129-74cfba02'}}},
            'host': 'inverted-corn-144000-8-85-6291-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_inverted_locust_hnsw_diskann_dml_dql_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'int8_1': {},
                                                                      'int16_1': {},
                                                                      'int32_1': {},
                                                                      'int64_1': {},
                                                                      'double_1': {},
                                                                      'float_1': {},
                                                                      'varchar_1': {},
                                                                      'int8_2': {'index_type': 'INVERTED'},
                                                                      'int16_2': {'index_type': 'INVERTED'},
                                                                      'int32_2': {'index_type': 'INVERTED'},
                                                                      'int64_2': {'index_type': 'INVERTED'},
                                                                      'double_2': {'index_type': 'INVERTED'},
                                                                      'float_2': {'index_type': 'INVERTED'},
                                                                      'varchar_2': {'index_type': 'INVERTED'},
                                                                      'bool_2': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_2': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_3': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_2': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img',
                                                                                                           'dim': 200}},
                                                                       'float_vector_3': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img',
                                                                                                           'dim': 200}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 5000000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int8_1',
                                                                        'int16_1',
                                                                        'int32_1',
                                                                        'int64_1',
                                                                        'double_1',
                                                                        'float_1',
                                                                        'varchar_1',
                                                                        'bool_1',
                                                                        'int8_2',
                                                                        'int16_2',
                                                                        'int32_2',
                                                                        'int64_2',
                                                                        'double_2',
                                                                        'float_2',
                                                                        'varchar_2',
                                                                        'bool_2'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '3h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 10,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 5000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 9,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 180}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'ef': 64},
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& '
                                                                          'id '
                                                                          '> '
                                                                          '-1',
                                                                  'guarantee_timestamp': None,
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 180,
                                                                  'random_data': True}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'ef': 1280},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 '
                                                                                    '< '
                                                                                    '100000 '
                                                                                    '&& '
                                                                                    'float_2 '
                                                                                    '> '
                                                                                    '10.0',
                                                                            'top_k': 1000},
                                                                           {'search_param': {'search_list': 30},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'varchar_1 '
                                                                                    'like '
                                                                                    '"0%" '
                                                                                    '&& '
                                                                                    'bool_2 '
                                                                                    '== '
                                                                                    'True'},
                                                                           {'search_param': {'ef': 1024},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'int8_1 '
                                                                                    '< '
                                                                                    '64 '
                                                                                    '&& '
                                                                                    'bool_1 '
                                                                                    '== '
                                                                                    'False',
                                                                            'top_k': 1009},
                                                                           {'search_param': {'search_list': 40},
                                                                            'anns_field': 'float_vector_3',
                                                                            'expr': 'int8_2 '
                                                                                    '> '
                                                                                    '64 '
                                                                                    '|| '
                                                                                    'double_2 '
                                                                                    '> '
                                                                                    '1000000.0'}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'timeout': 60,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&&  '
                                                                          'int64_2 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'timeout': 180,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [2500000.0,
                                                                                   5000000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024012941212419,
            'datetime': '2024-01-29 16:02:01.798852',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 948.3547,
                                      'float_vector_1': {'RT': 746.5793},
                                      'float_vector_2': {'RT': 228.2082},
                                      'float_vector_3': {'RT': 93.0811},
                                      'int8_1': {'RT': 0.5193},
                                      'int16_1': {'RT': 0.5845},
                                      'int32_1': {'RT': 0.5214},
                                      'int64_1': {'RT': 0.5816},
                                      'double_1': {'RT': 0.517},
                                      'float_1': {'RT': 0.5137},
                                      'varchar_1': {'RT': 0.517},
                                      'int8_2': {'RT': 1.0522},
                                      'int16_2': {'RT': 0.5146},
                                      'int32_2': {'RT': 0.5142},
                                      'int64_2': {'RT': 0.5158},
                                      'double_2': {'RT': 0.5178},
                                      'float_2': {'RT': 0.5263},
                                      'varchar_2': {'RT': 0.518},
                                      'bool_2': {'RT': 0.5153}},
                            'insert': {'total_time': 1221.9132,
                                       'VPS': 4091.9437,
                                       'batch_time': 1.2219,
                                       'batch': 5000},
                            'flush': {'RT': 3.5295},
                            'load': {'RT': 24.8352},
                            'Locust': {'Aggregated': {'Requests': 73359,
                                                      'Fails': 28424,
                                                      'RPS': 6.8,
                                                      'fail_s': 0.39,
                                                      'RT_max': 97321.96,
                                                      'RT_avg': 2922.66,
                                                      'TP50': 250.0,
                                                      'TP99': 30000.0},
                                       'delete': {'Requests': 10518,
                                                  'Fails': 0,
                                                  'RPS': 0.97,
                                                  'fail_s': 0.0,
                                                  'RT_max': 56305.42,
                                                  'RT_avg': 178.3,
                                                  'TP50': 20,
                                                  'TP99': 1400.0},
                                       'flush': {'Requests': 10495,
                                                 'Fails': 0,
                                                 'RPS': 0.97,
                                                 'fail_s': 0.0,
                                                 'RT_max': 97321.96,
                                                 'RT_avg': 7363.53,
                                                 'TP50': 6100.0,
                                                 'TP99': 37000.0},
                                       'hybrid_search': {'Requests': 10266,
                                                         'Fails': 8476,
                                                         'RPS': 0.95,
                                                         'fail_s': 0.83,
                                                         'RT_max': 57957.0,
                                                         'RT_avg': 291.42,
                                                         'TP50': 10,
                                                         'TP99': 2300.0},
                                       'insert': {'Requests': 10433,
                                                  'Fails': 0,
                                                  'RPS': 0.97,
                                                  'fail_s': 0.0,
                                                  'RT_max': 11344.9,
                                                  'RT_avg': 633.44,
                                                  'TP50': 400.0,
                                                  'TP99': 4200.0},
                                       'load': {'Requests': 10595,
                                                'Fails': 3129,
                                                'RPS': 0.98,
                                                'fail_s': 0.3,
                                                'RT_max': 57366.15,
                                                'RT_avg': 11146.24,
                                                'TP50': 2000.0,
                                                'TP99': 30000.0},
                                       'query': {'Requests': 10527,
                                                 'Fails': 8384,
                                                 'RPS': 0.98,
                                                 'fail_s': 0.8,
                                                 'RT_max': 57604.44,
                                                 'RT_avg': 334.13,
                                                 'TP50': 52,
                                                 'TP99': 2900.0},
                                       'search': {'Requests': 10525,
                                                  'Fails': 8435,
                                                  'RPS': 0.97,
                                                  'fail_s': 0.8,
                                                  'RT_max': 57683.78,
                                                  'RT_avg': 383.4,
                                                  'TP50': 120.0,
                                                  'TP99': 3000.0}}}}}

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `vector: memory and disk index`
            verify concurrent DML & DQL scenario which has 4 float_vector fields & 16 scalar fields

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 200dim,
                'float_vector_3': 200dim,
                'int8_1', 'int16_1', 'int32_1', 'int64_1', 'double_1', 'float_1', 'varchar_1', 'bool_1',
                'int8_2', 'int16_2', 'int32_2', 'int64_2', 'double_2', 'float_2', 'varchar_2', 'bool_2'
            2. build indexes:
                HNSW: 'float_vector'
                DIAKANN_IP: 'float_vector_1'
                HNSW: 'float_vector_2'
                DIAKANN_L2: 'float_vector_3'
                scalar_default_index: 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'double_1', 'float_1', 'varchar_1'
                scalar_INVERTED_index: 'int8_2', 'int16_2', 'int32_2', 'int64_2', 'double_2', 'float_2', 'varchar_2', 'bool_2'
            3. insert 5 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - insert
                - delete
                - flush
                - load
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

No response

yanliang567 commented 9 months ago

/unassign

wangting0128 commented 8 months ago

Not reproduce

image:master-20240220-e5a16050