[Bug]: [benchmark][standalone] Milvus standalone restart: `SIGSEGV: segmentation violation` in concurrent dql & upsert scene

wangting0128 commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version: master-20240816-2736a8b8-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-concurrent-qvs7s test case name: test_bitmap_locust_dql_dml_upsert_standalone

server:

[2024-08-16 11:16:42,169 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|bitmap-dql-dml-upsert-milvus|bitmap-dql-dml-upsert-minio|bitmap-dql-dml-upsert-etcd|bitmap-dql-dml-upsert-pulsar|bitmap-dql-dml-upsert-zookeeper|bitmap-dql-dml-upsert-kafka|bitmap-dql-dml-upsert-log|bitmap-dql-dml-upsert-tikv'  (util_cmd.py:14)
[2024-08-16 11:17:02,571 -  INFO - fouram]: [CliClient] pod details of release(bitmap-dql-dml-upsert): 
 I0816 11:16:43.410546     381 request.go:665] Waited for 1.170312669s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/source.toolkit.fluxcd.io/v1?timeout=32s
I0816 11:16:53.610118     381 request.go:665] Waited for 4.197159647s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/scheduling.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
bitmap-dql-dml-upsert-etcd-0                                      1/1     Running     0               3m30s   10.104.30.143   4am-node38   <none>           <none>
bitmap-dql-dml-upsert-milvus-standalone-775f66cc44-nrxzk          1/1     Running     1 (2m9s ago)    3m30s   10.104.18.221   4am-node25   <none>           <none>
bitmap-dql-dml-upsert-minio-86f79657f7-6k7dx                      1/1     Running     0               3m30s   10.104.18.222   4am-node25   <none>           <none>

NAME                                                              READY   STATUS             RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
bitmap-dql-dml-upsert-etcd-0                                      1/1     Running            0                 2d16h   10.104.30.143   4am-node38   <none>           <none>
bitmap-dql-dml-upsert-milvus-standalone-775f66cc44-nrxzk          1/1     Running            2 (2d16h ago)     2d16h   10.104.18.221   4am-node25   <none>           <none>
bitmap-dql-dml-upsert-minio-86f79657f7-6k7dx                      1/1     Running            0                 2d16h   10.104.18.222   4am-node25   <none>           <none>

bitmap-dql-dml-upsert-milvus-standalone-775f66cc44-nrxzk_restart.log

client pod name: fouramf-concurrent-qvs7s-3328890162 client log:

[2024-08-16 11:17:03,200 -  INFO - fouram]: [Base] Collection schema: 
{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}},
            {'name': 'sparse_float_vector', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}, {'name': 'int8_1', 'description': '', 'type': <DataType.INT8: 2>},
            {'name': 'int16_1', 'description': '', 'type': <DataType.INT16: 3>}, {'name': 'int32_1', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'int64_1', 'description': '', 'type': <DataType.INT64: 5>},
            {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}}, {'name': 'bool_1', 'description': '', 'type': <DataType.BOOL: 1>},
            {'name': 'array_int8_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT8: 2>},
            {'name': 'array_int16_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT16: 3>},
            {'name': 'array_int32_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT32: 4>},
            {'name': 'array_int64_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.INT64: 5>},
            {'name': 'array_varchar_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 100, 'max_capacity': 13}, 'element_type': <DataType.VARCHAR: 21>},
            {'name': 'array_bool_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 13}, 'element_type': <DataType.BOOL: 1>}],
 'enable_dynamic_field': False} (base.py:328)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64`, shards_num=16
            1. building `BITMAP` index on all supported 12 scalar fields
            2. 2 fields of different vector types
            3. verify DQL & DML(upsert) requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=13
            2. build indexes:
                IVF_SQ8: 'float_vector'
                SPARSE_WAND: 'sparse_float_vector'
                BITMAP: all scalar fields
            3. insert 10w data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - search
                - query
                - hybrid_search
                - load
                - upsert: batch=10
                - flush: ignore RateLimiter

Milvus Log

No response

Anything else?

test config:

{"dataset_params":{"metric_type":"L2","dim":128,"max_length":100,"scalars_index":{"int8_1":{"index_type":"BITMAP"},"int16_1":{"index_type":"BITMAP"},"int32_1":{"index_type":"BITMAP"},"int64_1":{"index_type":"BITMAP"},"varchar_1":{"index_type":"BITMAP"},"bool_1":{"index_type":"BITMAP"},"array_int8_1":{"index_type":"BITMAP"},"array_int16_1":{"index_type":"BITMAP"},"array_int32_1":{"index_type":"BITMAP"},"array_int64_1":{"index_type":"BITMAP"},"array_varchar_1":{"index_type":"BITMAP"},"array_bool_1":{"index_type":"BITMAP"}},"vectors_index":{"sparse_float_vector":{"index_type":"SPARSE_INVERTED_INDEX","index_param":{"drop_ratio_build":0.2},"metric_type":"IP"}},"scalars_params":{"array_int8_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-128,128],"max_capacity":13}}},"array_int16_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-200,200],"max_capacity":13}}},"array_int32_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"specify_scope","specify_range":[-300,300],"max_capacity":13}}},"array_int64_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"fixed_value_range","specify_range":[-400,432],"batch":50,"max_capacity":13}}},"array_varchar_1":{"params":{"max_capacity":13},"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-1500,1500],"max_capacity":13}}},"array_bool_1":{"params":{"max_capacity":13}},"int8_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-128,128],"max_capacity":13}}},"int16_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-200,200],"max_capacity":13}}},"int32_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"specify_scope","specify_range":[-300,300],"max_capacity":13}}},"int64_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"fixed_value_range","specify_range":[-400,432],"batch":50,"max_capacity":13}}},"varchar_1":{"other_params":{"dataset":"random_algorithm","algorithm_params":{"algorithm_name":"random_range","specify_range":[-1500,1500],"max_capacity":13}}}},"dataset_name":"sift","dataset_size":100000,"ni_per":5000},"collection_params":{"other_fields":["sparse_float_vector","int8_1","int16_1","int32_1","int64_1","varchar_1","bool_1","array_int8_1","array_int16_1","array_int32_1","array_int64_1","array_varchar_1","array_bool_1"],"shards_num":16},"resource_groups_params":{"reset":false},"database_user_params":{"reset_rbac":false,"reset_db":false},"index_params":{"index_type":"IVF_SQ8","index_param":{"nlist":1024}},"concurrent_params":{"concurrent_number":[30],"during_time":"3h","interval":20},"concurrent_tasks":[{"type":"search","weight":1,"params":{"nq":1000,"top_k":10,"search_param":{"nprobe":16},"expr":"int8_1 == 100","guarantee_timestamp":null,"partition_names":null,"output_fields":["id","float_vector","int64_1"],"ignore_growing":false,"group_by_field":null,"timeout":null,"random_data":true,"check_task":"check_search_output","check_items":null}},{"type":"query","weight":1,"params":{"ids":null,"expr":"int64_1 > -1","output_fields":["*"],"offset":null,"limit":10,"ignore_growing":false,"partition_names":null,"timeout":null,"random_data":false,"random_count":0,"random_range":[0,1],"field_name":"id","field_type":"int64","check_task":"check_query_output","check_items":{"expect_length":10}}},{"type":"hybrid_search","weight":1,"params":{"nq":10,"top_k":10,"reqs":[{"search_param":{"nprobe":128},"anns_field":"float_vector","expr":"(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, 1)) || ((varchar_1 like \"1%\") and (bool_1 == True))","top_k":100},{"search_param":{"drop_ratio_search":0.1},"anns_field":"sparse_float_vector","expr":"not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, 1])"}],"rerank":{"RRFRanker":[]},"output_fields":["*"],"ignore_growing":false,"guarantee_timestamp":null,"partition_names":null,"timeout":null,"random_data":true,"check_task":"check_search_output","check_items":{"output_fields":["sparse_float_vector","int8_1","int16_1","int32_1","int64_1","varchar_1","bool_1","array_int8_1","array_int16_1","array_int32_1","array_int64_1","array_varchar_1","array_bool_1","id","float_vector"],"nq":10}}},{"type":"load","weight":1,"params":{"replica_number":1,"timeout":180,"check_task":"check_response","check_items":null}},{"type":"upsert","weight":1,"params":{"nb":10,"timeout":30,"random_id":true,"random_vector":true,"varchar_filled":false,"start_id":100000,"check_task":"check_response","check_items":null}},{"type":"flush","weight":1,"params":{"timeout":180,"check_task":"check_ignore_rate_limit","check_items":null}}]}

test result:

[2024-08-16 14:21:16,057 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-08-16 14:21:16,058 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     flush                                                                           4430    17(0.38%) |  66633       0  190606  22000 |    0.41        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     hybrid_search                                                                   4478     0(0.00%) |   1643      32   80459    560 |    0.41        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     load                                                                            4518     0(0.00%) |    549       4   99765    180 |    0.42        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     query                                                                           4511     0(0.00%) |    790       8  108806     99 |    0.42        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     search                                                                          4435     0(0.00%) |   2011     222  128948    740 |    0.41        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: grpc     upsert                                                                          4361     4(0.09%) |    191       7   30697     25 |    0.40        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]:          Aggregated                                                                     26733    21(0.08%) |  11908       0  190606    440 |    2.48        0.00 (stats.py:789)
[2024-08-16 14:21:16,058 -  INFO - fouram]:  (stats.py:790)
[2024-08-16 14:21:16,062 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_16c64m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '16.0', 'memory': '64Gi'}, 'requests': {'cpu': '9.0', 'memory': '33Gi'}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1, 'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone', 'metrics': {'podMonitor': {'enabled': True}}},
                       'pulsar': {'enabled': False},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20240816-2736a8b8-amd64'}}},
            'host': 'bitmap-dql-dml-upsert-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_bitmap_locust_dql_dml_upsert_standalone',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'max_length': 100,
                                                    'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
                                                                      'int16_1': {'index_type': 'BITMAP'},
                                                                      'int32_1': {'index_type': 'BITMAP'},
                                                                      'int64_1': {'index_type': 'BITMAP'},
                                                                      'varchar_1': {'index_type': 'BITMAP'},
                                                                      'bool_1': {'index_type': 'BITMAP'},
                                                                      'array_int8_1': {'index_type': 'BITMAP'},
                                                                      'array_int16_1': {'index_type': 'BITMAP'},
                                                                      'array_int32_1': {'index_type': 'BITMAP'},
                                                                      'array_int64_1': {'index_type': 'BITMAP'},
                                                                      'array_varchar_1': {'index_type': 'BITMAP'},
                                                                      'array_bool_1': {'index_type': 'BITMAP'}},
                                                    'vectors_index': {'sparse_float_vector': {'index_type': 'SPARSE_INVERTED_INDEX',
                                                                                              'index_param': {'drop_ratio_build': 0.2},
                                                                                              'metric_type': 'IP'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
                                                                                        'other_params': {'dataset': 'random_algorithm',
                                                                                                         'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                              'specify_range': [-128, 128],
                                                                                                                              'max_capacity': 13}}},
                                                                       'array_int16_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                               'specify_range': [-200, 200],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int32_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                               'specify_range': [-300, 300],
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_int64_1': {'params': {'max_capacity': 13},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                               'specify_range': [-400, 432],
                                                                                                                               'batch': 50,
                                                                                                                               'max_capacity': 13}}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 13},
                                                                                           'other_params': {'dataset': 'random_algorithm',
                                                                                                            'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                                 'specify_range': [-1500, 1500],
                                                                                                                                 'max_capacity': 13}}},
                                                                       'array_bool_1': {'params': {'max_capacity': 13}},
                                                                       'int8_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                   'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                        'specify_range': [-128, 128],
                                                                                                                        'max_capacity': 13}}},
                                                                       'int16_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                         'specify_range': [-200, 200],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int32_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                         'specify_range': [-300, 300],
                                                                                                                         'max_capacity': 13}}},
                                                                       'int64_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                         'specify_range': [-400, 432],
                                                                                                                         'batch': 50,
                                                                                                                         'max_capacity': 13}}},
                                                                       'varchar_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                      'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                           'specify_range': [-1500, 1500],
                                                                                                                           'max_capacity': 13}}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 100000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['sparse_float_vector', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
                                                                        'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
                                                                        'array_bool_1'],
                                                       'shards_num': 16},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 30, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 10,
                                                                  'search_param': {'nprobe': 16},
                                                                  'expr': 'int8_1 == 100',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['id', 'float_vector', 'int64_1'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': None,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': 10,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': None,
                                                                  'random_data': False,
                                                                  'random_count': 0,
                                                                  'random_range': [0, 1],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'check_task': 'check_query_output',
                                                                  'check_items': {'expect_length': 10}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': '(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, '
                                                                                    '1)) || ((varchar_1 like "1%") and (bool_1 == True))',
                                                                            'top_k': 100},
                                                                           {'search_param': {'drop_ratio_search': 0.1},
                                                                            'anns_field': 'sparse_float_vector',
                                                                            'expr': 'not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, '
                                                                                    '1])'}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': None,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'output_fields': ['sparse_float_vector', 'int8_1', 'int16_1', 'int32_1',
                                                                                                    'int64_1', 'varchar_1', 'bool_1', 'array_int8_1',
                                                                                                    'array_int16_1', 'array_int32_1', 'array_int64_1',
                                                                                                    'array_varchar_1', 'array_bool_1', 'id', 'float_vector'],
                                                                                  'nq': 10}}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1, 'timeout': 180, 'check_task': 'check_response', 'check_items': None}},
                                                      {'type': 'upsert',
                                                       'weight': 1,
                                                       'params': {'nb': 10,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 100000,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 180, 'check_task': 'check_ignore_rate_limit', 'check_items': None}}]},
            'run_id': 2024081668108011,
            'datetime': '2024-08-16 11:13:30.200652',
            'client_version': '2.5'},
 'result': {'test_result': {'index': {'RT': 220.5692,
                                      'sparse_float_vector': {'RT': 0.5159},
                                      'int8_1': {'RT': 0.5732},
                                      'int16_1': {'RT': 0.5125},
                                      'int32_1': {'RT': 0.513},
                                      'int64_1': {'RT': 0.5137},
                                      'varchar_1': {'RT': 0.5117},
                                      'bool_1': {'RT': 0.5127},
                                      'array_int8_1': {'RT': 0.5181},
                                      'array_int16_1': {'RT': 0.5122},
                                      'array_int32_1': {'RT': 0.5109},
                                      'array_int64_1': {'RT': 0.5102},
                                      'array_varchar_1': {'RT': 0.5122},
                                      'array_bool_1': {'RT': 0.51}},
                            'insert': {'total_time': 10.0554, 'VPS': 9944.9052, 'batch_time': 0.5028, 'batch': 5000},
                            'flush': {'RT': 3.0719},
                            'load': {'RT': 1.6284},
                            'Locust': {'Aggregated': {'Requests': 26733,
                                                      'Fails': 21,
                                                      'RPS': 2.48,
                                                      'fail_s': 0.0,
                                                      'RT_max': 190606.14,
                                                      'RT_avg': 11908.43,
                                                      'TP50': 440.0,
                                                      'TP99': 182000.0},
                                       'flush': {'Requests': 4430,
                                                 'Fails': 17,
                                                 'RPS': 0.41,
                                                 'fail_s': 0.0,
                                                 'RT_max': 190606.14,
                                                 'RT_avg': 66633.72,
                                                 'TP50': 22000.0,
                                                 'TP99': 183000.0},
                                       'hybrid_search': {'Requests': 4478,
                                                         'Fails': 0,
                                                         'RPS': 0.41,
                                                         'fail_s': 0.0,
                                                         'RT_max': 80459.57,
                                                         'RT_avg': 1643.12,
                                                         'TP50': 560.0,
                                                         'TP99': 12000.0},
                                       'load': {'Requests': 4518,
                                                'Fails': 0,
                                                'RPS': 0.42,
                                                'fail_s': 0.0,
                                                'RT_max': 99765.71,
                                                'RT_avg': 549.06,
                                                'TP50': 180.0,
                                                'TP99': 5300.0},
                                       'query': {'Requests': 4511,
                                                 'Fails': 0,
                                                 'RPS': 0.42,
                                                 'fail_s': 0.0,
                                                 'RT_max': 108806.46,
                                                 'RT_avg': 790.28,
                                                 'TP50': 99,
                                                 'TP99': 8800.0},
                                       'search': {'Requests': 4435,
                                                  'Fails': 0,
                                                  'RPS': 0.41,
                                                  'fail_s': 0.0,
                                                  'RT_max': 128948.58,
                                                  'RT_avg': 2011.3,
                                                  'TP50': 740.0,
                                                  'TP99': 13000.0},
                                       'upsert': {'Requests': 4361,
                                                  'Fails': 4,
                                                  'RPS': 0.4,
                                                  'fail_s': 0.0,
                                                  'RT_max': 30697.33,
                                                  'RT_avg': 191.92,
                                                  'TP50': 25,
                                                  'TP99': 2100.0}}}}}

yanliang567 commented 1 month ago

/assign @zhagnlu /unassign

cydrain commented 4 weeks ago

/assign

cydrain commented 4 weeks ago

runtime.gcResetMarkState
    /usr/local/go/src/runtime/mgc.go:1620 pc=0x1eb0f8f

SIGSEGV: segmentation violation
PC=0x1eb0f8f m=6931 sigcode=1

goroutine 0 [idle]:
runtime.gcResetMarkState()
    /usr/local/go/src/runtime/mgc.go:1620 +0x6f fp=0x7f10dd8bb850 sp=0x7f10dd8bb820 pc=0x1eb0f8f
runtime.systemstack()
    /usr/local/go/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f10dd8bb860 sp=0x7f10dd8bb850 pc=0x1f0548a

goroutine 2153222 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0012177e0 sp=0xc0012177d0 pc=0x1f05428
runtime.gcStart({0x1fc0?, 0x0?, 0x0?})
    /usr/local/go/src/runtime/mgc.go:669 +0x235 fp=0xc001217878 sp=0xc0012177e0 pc=0x1eaee15
runtime.GC()
    /usr/local/go/src/runtime/mgc.go:466 +0x3e fp=0xc0012178b0 sp=0xc001217878 pc=0x1eae91e
runtime/debug.freeOSMemory()
    /usr/local/go/src/runtime/mheap.go:1672 +0x13 fp=0xc0012178c8 sp=0xc0012178b0 pc=0x1f00093
runtime/debug.FreeOSMemory()
    /usr/local/go/src/runtime/debug/garbage.go:104 +0xf fp=0xc0012178d8 sp=0xc0012178c8 pc=0x23b2b6f
github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).LoadSegment(0xc002dcc500, {0x616d4a8, 0xc00ab5cf60}, {0x61f47e0?, 0xc0114ca000?}, 0xc01005b7a0)
    /workspace/source/internal/querynodev2/segments/segment_loader.go:733 +0xa67 fp=0xc001217be0 sp=0xc0012178d8 pc=0x4df2a87
github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4(0xe5cf1d2afc8417?)
    /workspace/source/internal/querynodev2/segments/segment_loader.go:302 +0x58a fp=0xc001217e00 sp=0xc001217be0 pc=0x4dec56a
github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3(0x0)
    /workspace/source/pkg/util/funcutil/parallel.go:86 +0x151 fp=0xc001217fc8 sp=0xc001217e00 pc=0x2bbf5f1
github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func4()
    /workspace/source/pkg/util/funcutil/parallel.go:104 +0x27 fp=0xc001217fe0 sp=0xc001217fc8 pc=0x2bbf467
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc001217fe8 sp=0xc001217fe0 pc=0x1f07401
created by github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel in goroutine 2153359
    /workspace/source/pkg/util/funcutil/parallel.go:73 +0x1b4

ThreadDao commented 3 weeks ago

I guess my problem is the same as this one.

image: master-20240819-14ef88f4-amd64
SIGV error log: stand_b4p85.log

pods:

zong-go-latest-etcd-0                                             1/1     Running     0                 2d19h   10.104.24.58    4am-node29   <none>           <none>
zong-go-latest-milvus-standalone-767bd55f6b-b4p85                 1/1     Running     2 (22h ago)       2d19h   10.104.24.62    4am-node29   <none>           <none>
zong-go-latest-minio-865844ffdb-w6rr8                             1/1     Running     0                 2d19h   10.104.24.59    4am-node29   <none>           <none>

xiaofan-luan commented 3 weeks ago

this seems to be related with sparse embedding index

@zhengbuqian could you please take a look at it

zhengbuqian commented 3 weeks ago

@ThreadDao this issue is about a different sigsegv, they have different stacktrace.

wangting0128 commented 3 weeks ago

The sparse vector written in this test case has no null value

zhengbuqian commented 3 weeks ago

/assign @pingliu

assigning to pingliu for now, I believe he was looking at a similar issue

sre-ci-robot commented 3 weeks ago

@zhengbuqian: GitHub didn't allow me to assign the following users: pingliu.

Note that only milvus-io members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/milvus-io/milvus/issues/35548#issuecomment-2311736793): >/assign @pingliu > >assigning to pingliu for now, I believe he was looking at a similar issue Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

pingliu commented 3 weeks ago

@zhengbuqian This instance does not enable mmap, maybe it is not the same issue.

milvus-io / milvus