search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.52k stars 2.83k forks source link

[Bug]: [benchmark][standalone] load collection raise error `collection not loaded` #35458

Closed wangting0128 closed 3 weeks ago

wangting0128 commented 1 month ago

Is there an existing issue for this?

Environment

- Milvus version: master-20240813-f3f5e158-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq   
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-b8lc8

server:

NAME                                                              READY   STATUS             RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
wt-test-etcd-0                                                    1/1     Running            0                 26d     10.104.33.149   4am-node36   <none>           <none>
wt-test-milvus-standalone-5fcfc854c5-96mq7                        1/1     Running            0                 10h     10.104.33.115   4am-node36   <none>           <none>
wt-test-minio-7b8f7b7444-cr5l2                                    1/1     Running            0                 26d     10.104.33.148   4am-node36   <none>           <none>

client pod name: fouramf-b8lc8-4126907731 client log:

截屏2024-08-13 21 23 00

schema:

{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'binary_vector', 'description': '', 'type': <DataType.BINARY_VECTOR: 100>, 'params': {'dim': 128}},
            {'name': 'float16_vector', 'description': '', 'type': <DataType.FLOAT16_VECTOR: 102>, 'params': {'dim': 128}}, {'name': 'bfloat16_vector', 'description': '', 'type': <DataType.BFLOAT16_VECTOR: 103>, 'params': {'dim': 128}},
            {'name': 'sparse_float_vector', 'description': '', 'type': <DataType.SPARSE_FLOAT_VECTOR: 104>}, {'name': 'int8_1', 'description': '', 'type': <DataType.INT8: 2>},
            {'name': 'int16_1', 'description': '', 'type': <DataType.INT16: 3>}, {'name': 'int32_1', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'int64_1', 'description': '', 'type': <DataType.INT64: 5>},
            {'name': 'double_1', 'description': '', 'type': <DataType.DOUBLE: 11>}, {'name': 'float_1', 'description': '', 'type': <DataType.FLOAT: 10>},
            {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 10}}, {'name': 'bool_1', 'description': '', 'type': <DataType.BOOL: 1>},
            {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'array_int8_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT8: 2>},
            {'name': 'array_int16_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT16: 3>},
            {'name': 'array_int32_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT32: 4>},
            {'name': 'array_int64_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT64: 5>},
            {'name': 'array_double_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.DOUBLE: 11>},
            {'name': 'array_float_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.FLOAT: 10>},
            {'name': 'array_varchar_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 10, 'max_capacity': 9}, 'element_type': <DataType.VARCHAR: 21>},
            {'name': 'array_bool_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.BOOL: 1>}, {'name': 'int8_2', 'description': '', 'type': <DataType.INT8: 2>},
            {'name': 'int16_2', 'description': '', 'type': <DataType.INT16: 3>}, {'name': 'int32_2', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'int64_2', 'description': '', 'type': <DataType.INT64: 5>},
            {'name': 'double_2', 'description': '', 'type': <DataType.DOUBLE: 11>}, {'name': 'float_2', 'description': '', 'type': <DataType.FLOAT: 10>},
            {'name': 'varchar_2', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 10}}, {'name': 'bool_2', 'description': '', 'type': <DataType.BOOL: 1>},
            {'name': 'json_2', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'array_int8_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT8: 2>},
            {'name': 'array_int16_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT16: 3>},
            {'name': 'array_int32_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT32: 4>},
            {'name': 'array_int64_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.INT64: 5>},
            {'name': 'array_double_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.DOUBLE: 11>},
            {'name': 'array_float_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.FLOAT: 10>},
            {'name': 'array_varchar_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 10, 'max_capacity': 9}, 'element_type': <DataType.VARCHAR: 21>},
            {'name': 'array_bool_2', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_capacity': 9}, 'element_type': <DataType.BOOL: 1>}],
 'enable_dynamic_field': False} (base.py:328)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64`
            1. building `BITMAP` index on INT64 primary key and all supported 12 scalar fields
            2. the other 22 scalar fields build `INVERTED`, `Trie`, `STL_SORT` indexes
            3. 4 fields of different vector types
            4. search for different expressions on BITMAP index fields

        :test steps:
            1. create collection with fields:
                'binary_vector': 128dim
                'float16_vector': 128dim
                'bfloat16_vector': 128dim
                'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector
                'id': primary key type is INT64

                all scalar fields: varchar max_length=10, array max_capacity=9
            2. build indexes:
                BIN_IVF_FLAT: 'binary_vector'
                IVF_SQ8: 'float16_vector'
                HNSW: 'bfloat16_vector'
                SPARSE_WAND: 'sparse_float_vector'
                BITMAP: 'id', '*_1' all supported field names
                INVERTED: 'array_float_1', 'array_double_1', 'float_2', 'double_2', 'bool_2', 'array_int8_2',
                          'array_int16_2', 'array_int32_2', 'array_int64_2', 'array_varchar_2', 'array_bool_2',
                          'array_float_2', 'array_double_2'
                Trie: 'varchar_2'
                STL_SORT: 'float_1', 'double_1', 'int8_2', 'int16_2', 'int32_2', 'int64_2'
            3. insert 1 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection <- raise error

Milvus Log

No response

Anything else?

test config:

{
     "dataset_params": {
          "metric_type": "JACCARD",
          "vector_field_name": "binary_vector",
          "dim": 128,
          "sparse_range": [
               1,
               100
          ],
          "max_length": 10,
          "scalars_index": {
               "id": {
                    "index_type": "BITMAP"
               },
               "int8_1": {
                    "index_type": "BITMAP"
               },
               "int16_1": {
                    "index_type": "BITMAP"
               },
               "int32_1": {
                    "index_type": "BITMAP"
               },
               "int64_1": {
                    "index_type": "BITMAP"
               },
               "varchar_1": {
                    "index_type": "BITMAP"
               },
               "bool_1": {
                    "index_type": "BITMAP"
               },
               "array_int8_1": {
                    "index_type": "BITMAP"
               },
               "array_int16_1": {
                    "index_type": "BITMAP"
               },
               "array_int32_1": {
                    "index_type": "BITMAP"
               },
               "array_int64_1": {
                    "index_type": "BITMAP"
               },
               "array_varchar_1": {
                    "index_type": "BITMAP"
               },
               "array_bool_1": {
                    "index_type": "BITMAP"
               },
               "array_float_1": {
                    "index_type": "INVERTED"
               },
               "array_double_1": {
                    "index_type": "INVERTED"
               },
               "float_2": {
                    "index_type": "INVERTED"
               },
               "double_2": {
                    "index_type": "INVERTED"
               },
               "bool_2": {
                    "index_type": "INVERTED"
               },
               "array_int8_2": {
                    "index_type": "INVERTED"
               },
               "array_int16_2": {
                    "index_type": "INVERTED"
               },
               "array_int32_2": {
                    "index_type": "INVERTED"
               },
               "array_int64_2": {
                    "index_type": "INVERTED"
               },
               "array_varchar_2": {
                    "index_type": "INVERTED"
               },
               "array_bool_2": {
                    "index_type": "INVERTED"
               },
               "array_float_2": {
                    "index_type": "INVERTED"
               },
               "array_double_2": {
                    "index_type": "INVERTED"
               },
               "varchar_2": {
                    "index_type": "Trie"
               },
               "float_1": {
                    "index_type": "STL_SORT"
               },
               "double_1": {
                    "index_type": "STL_SORT"
               },
               "int8_2": {
                    "index_type": "STL_SORT"
               },
               "int16_2": {
                    "index_type": "STL_SORT"
               },
               "int32_2": {
                    "index_type": "STL_SORT"
               },
               "int64_2": {
                    "index_type": "STL_SORT"
               }
          },
          "vectors_index": {
               "float16_vector": {
                    "index_type": "IVF_SQ8",
                    "index_param": {
                         "nlist": 1024
                    },
                    "metric_type": "L2"
               },
               "bfloat16_vector": {
                    "index_type": "HNSW",
                    "index_param": {
                         "M": 8,
                         "efConstruction": 200
                    },
                    "metric_type": "L2"
               },
               "sparse_float_vector": {
                    "index_type": "SPARSE_WAND",
                    "index_param": {
                         "drop_ratio_build": 0.2
                    },
                    "metric_type": "IP"
               }
          },
          "scalars_params": {
               "array_int8_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int16_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int32_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int64_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_double_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_float_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_varchar_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_bool_1": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int8_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int16_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int32_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_int64_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_double_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_float_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_varchar_2": {
                    "params": {
                         "max_capacity": 9
                    }
               },
               "array_bool_2": {
                    "params": {
                         "max_capacity": 9
                    }
               }
          },
          "dataset_name": "local",
          "dataset_size": "1m",
          "ni_per": 5000
     },
     "collection_params": {
          "other_fields": [
               "float16_vector",
               "bfloat16_vector",
               "sparse_float_vector",
               "int8_1",
               "int16_1",
               "int32_1",
               "int64_1",
               "double_1",
               "float_1",
               "varchar_1",
               "bool_1",
               "json_1",
               "array_int8_1",
               "array_int16_1",
               "array_int32_1",
               "array_int64_1",
               "array_double_1",
               "array_float_1",
               "array_varchar_1",
               "array_bool_1",
               "int8_2",
               "int16_2",
               "int32_2",
               "int64_2",
               "double_2",
               "float_2",
               "varchar_2",
               "bool_2",
               "json_2",
               "array_int8_2",
               "array_int16_2",
               "array_int32_2",
               "array_int64_2",
               "array_double_2",
               "array_float_2",
               "array_varchar_2",
               "array_bool_2"
          ],
          "shards_num": 2,
          "varchar_id": false
     },
     "resource_groups_params": {
          "reset": false
     },
     "database_user_params": {
          "reset_rbac": false,
          "reset_db": false
     },
     "index_params": {
          "index_type": "BIN_IVF_FLAT",
          "index_param": {
               "nlist": 2048
          }
     },
     "concurrent_params": {
          "concurrent_number": [
               20
          ],
          "during_time": 1800,
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "nq": 1000,
                    "top_k": 10,
                    "search_param": {
                         "nprobe": 16
                    },
                    "expr": "id >= 100",
                    "guarantee_timestamp": null,
                    "partition_names": null,
                    "output_fields": [
                         "*"
                    ],
                    "ignore_growing": false,
                    "group_by_field": null,
                    "timeout": 720,
                    "random_data": true
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "ids": null,
                    "expr": "id > -1 && ",
                    "output_fields": [
                         "id",
                         "binary_vector",
                         "int64_1"
                    ],
                    "offset": null,
                    "limit": null,
                    "ignore_growing": false,
                    "partition_names": null,
                    "timeout": 720,
                    "random_data": true,
                    "random_count": 10,
                    "random_range": [
                         0,
                         5000000
                    ],
                    "field_name": "id",
                    "field_type": "int64"
               }
          },
          {
               "type": "hybrid_search",
               "weight": 1,
               "params": {
                    "nq": 10,
                    "top_k": 10,
                    "reqs": [
                         {
                              "search_param": {
                                   "nprobe": 128
                              },
                              "anns_field": "binary_vector",
                              "expr": "(int64_1 % 10) == 1",
                              "top_k": 100
                         },
                         {
                              "search_param": {
                                   "nprobe": 64
                              },
                              "anns_field": "float16_vector",
                              "expr": "ARRAY_LENGTH(array_int16_1) >= 5 && array_contains_any(array_bool_1, [true])",
                              "top_k": 10
                         },
                         {
                              "search_param": {
                                   "ef": 32
                              },
                              "anns_field": "bfloat16_vector",
                              "expr": "(int32_1 % 100) <= 50",
                              "top_k": 30
                         },
                         {
                              "search_param": {
                                   "drop_ratio_search": 0.1
                              },
                              "anns_field": "sparse_float_vector",
                              "expr": "(varchar_1 like \"1%\") && (bool_1 == True)"
                         }
                    ],
                    "rerank": {
                         "RRFRanker": []
                    },
                    "output_fields": [
                         "*"
                    ],
                    "ignore_growing": false,
                    "guarantee_timestamp": null,
                    "partition_names": null,
                    "timeout": 600,
                    "random_data": true
               }
          }
     ]
}
xiaocai2333 commented 1 month ago 
[2024/08/13 14:34:08.616 +00:00] [INFO] [segments/segment_loader.go:853] ["load field binlogs done for sealed segment with index"] [traceID=2467f96cc30ab7ed953ca31a88b77d4a] [collectionID=451809678468808897] [partitionID=451809678468808898] [segmentID=451809678470037289] [rowCount=500186] [fieldID=100] [binlog="[{\"entries_num\":41399,\"timestamp_from\":451816313806651394,\"timestamp_to\":451816375332110337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037358\",\"log_size\":53719,\"logID\":451809678470037358,\"memory_size\":331192},{\"entries_num\":40599,\"timestamp_from\":451816372317192194,\"timestamp_to\":451816391912456193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037384\",\"log_size\":42206,\"logID\":451809678470037384,\"memory_size\":324792},{\"entries_num\":40699,\"timestamp_from\":451816377743572993,\"timestamp_to\":451816392961032193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037406\",\"log_size\":42326,\"logID\":451809678470037406,\"memory_size\":325592},{\"entries_num\":40499,\"timestamp_from\":451816349392699393,\"timestamp_to\":451816385267367939,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037449\",\"log_size\":42123,\"logID\":451809678470037449,\"memory_size\":323992},{\"entries_num\":40599,\"timestamp_from\":451816351739150337,\"timestamp_to\":451816371832225793,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037491\",\"log_size\":42225,\"logID\":451809678470037491,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816356811374593,\"timestamp_to\":451816366707572738,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037538\",\"log_size\":42550,\"logID\":451809678470037538,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816334699528193,\"timestamp_to\":451816363679809537,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037579\",\"log_size\":42212,\"logID\":451809678470037579,\"memory_size\":324792},{\"entries_num\":40699,\"timestamp_from\":451816330269556739,\"timestamp_to\":451816348921102337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037611\",\"log_size\":42324,\"logID\":451809678470037611,\"memory_size\":325592},{\"entries_num\":40499,\"timestamp_from\":451816325629345793,\"timestamp_to\":451816334266990593,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037678\",\"log_size\":42115,\"logID\":451809678470037678,\"memory_size\":323992},{\"entries_num\":40799,\"timestamp_from\":451816320871694337,\"timestamp_to\":451816411651899393,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037713\",\"log_size\":42414,\"logID\":451809678470037713,\"memory_size\":326392},{\"entries_num\":40599,\"timestamp_from\":451816400969531393,\"timestamp_to\":451816412635201539,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037757\",\"log_size\":42226,\"logID\":451809678470037757,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816393472475137,\"timestamp_to\":451816407746215937,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037807\",\"log_size\":42221,\"logID\":451809678470037807,\"memory_size\":324792},{\"entries_num\":11998,\"timestamp_from\":451816395962843139,\"timestamp_to\":451816397915553793,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037839\",\"log_size\":12777,\"logID\":451809678470037839,\"memory_size\":95984}]"] [current_index_version=5] [load_duration=44m58.271465815s]
yanliang567 commented 1 month ago

@xiaocai2333 so the loading is too slow with many scalar fields? @wangting0128 can we calculate the data size we need to load and the speed of loading? /assign @xiaocai2333 /unassign

xiaocai2333 commented 1 month ago

@xiaocai2333 so the loading is too slow with many scalar fields? @wangting0128 can we calculate the data size we need to load and the speed of loading? /assign @xiaocai2333 /unassign

2024-08-13 21:49:10.344 | I20240813 13:49:10.344883    86 load_index_c.cpp:236] [SERVER][AppendIndexV2][milvus] [collection=451809678468808897][segment=451809678470037289][field=100][enable_mmap=false] load index 451809678468808930 |  
-- | -- | --
  |   | 2024-08-13 21:49:10.566 | I20240813 13:49:10.566550    86 load_index_c.cpp:300] [SERVER][AppendIndexV2][milvus] [collection=451809678468808897][segment=451809678470037289][field=100][enable_mmap=false] load index 451809678468808930 done |  
  |   | 2024-08-13 21:49:12.027 | I20240813 13:49:12.027810   248 load_index_c.cpp:236] [SERVER][AppendIndexV2][milvus] [collection=451809678468808897][segment=451809678470036939][field=100][enable_mmap=false] load index 451809678468808930 |  
  |   | 2024-08-13 21:49:12.250 | I20240813 13:49:12.250844   248 load_index_c.cpp:300] [SERVER][AppendIndexV2][milvus] [collection=451809678468808897][segment=451809678470036939][field=100][enable_mmap=false] load index 451809678468808930 done |  
  |   | 2024-08-13 22:33:13.023 | [2024/08/13 14:33:13.023 +00:00] [INFO] [segments/segment.go:1207] ["updateSegmentIndex done"] [traceID=2467f96cc30ab7ed953ca31a88b77d4a] [collectionID=451809678468808897] [partitionID=451809678468808898] [segmentID=451809678470036939] [fieldID=100] |  
  |   | 2024-08-13 22:33:13.023 | [2024/08/13 14:33:13.023 +00:00] [INFO] [segments/segment_loader.go:853] ["load field binlogs done for sealed segment with index"] [traceID=2467f96cc30ab7ed953ca31a88b77d4a] [collectionID=451809678468808897] [partitionID=451809678468808898] [segmentID=451809678470036939] [rowCount=499814] [fieldID=100] [binlog="[{\"entries_num\":41299,\"timestamp_from\":451816313806651394,\"timestamp_to\":451816357309448193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470036979\",\"log_size\":48660,\"logID\":451809678470036979,\"memory_size\":330392},{\"entries_num\":40599,\"timestamp_from\":451816349392699393,\"timestamp_to\":451816361176072193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037023\",\"log_size\":42218,\"logID\":451809678470037023,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816351739150337,\"timestamp_to\":451816366707572738,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037059\",\"log_size\":42201,\"logID\":451809678470037059,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816342314811393,\"timestamp_to\":451816369433608196,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037101\",\"log_size\":42219,\"logID\":451809678470037101,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816334699528193,\"timestamp_to\":451816348921102337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037175\",\"log_size\":42313,\"logID\":451809678470037175,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816327975534593,\"timestamp_to\":451816339090440193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037210\",\"log_size\":42233,\"logID\":451809678470037210,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816328906407938,\"timestamp_to\":451816377088212995,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037248\",\"log_size\":42192,\"logID\":451809678470037248,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816377088212995,\"timestamp_to\":451816390864142337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037275\",\"log_size\":42364,\"logID\":451809678470037275,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816325629345793,\"timestamp_to\":451816392961032193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037309\",\"log_size\":42212,\"logID\":451809678470037309,\"memory_size\":324792},{\"entries_num\":40899,\"timestamp_from\":451816320871694337,\"timestamp_to\":451816409279758339,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037356\",\"log_size\":42522,\"logID\":451809678470037356,\"memory_size\":327192},{\"entries_num\":40599,\"timestamp_from\":451816393472475137,\"timestamp_to\":451816412635201539,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037412\",\"log_size\":42218,\"logID\":451809678470037412,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816395962843139,\"timestamp_to\":451816407746215937,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037444\",\"log_size\":42539,\"logID\":451809678470037444,\"memory_size\":324792},{\"entries_num\":11626,\"timestamp_from\":451816400969531393,\"timestamp_to\":451816402922766337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470036939/100/451809678470037475\",\"log_size\":12393,\"logID\":451809678470037475,\"memory_size\":93008}]"] [current_index_version=5] [load_duration=44m0.995517542s] |  
  |   | 2024-08-13 22:34:08.616 | [2024/08/13 14:34:08.616 +00:00] [INFO] [segments/segment.go:1207] ["updateSegmentIndex done"] [traceID=2467f96cc30ab7ed953ca31a88b77d4a] [collectionID=451809678468808897] [partitionID=451809678468808898] [segmentID=451809678470037289] [fieldID=100] |  
  |   | 2024-08-13 22:34:08.616 | [2024/08/13 14:34:08.616 +00:00] [INFO] [segments/segment_loader.go:853] ["load field binlogs done for sealed segment with index"] [traceID=2467f96cc30ab7ed953ca31a88b77d4a] [collectionID=451809678468808897] [partitionID=451809678468808898] [segmentID=451809678470037289] [rowCount=500186] [fieldID=100] [binlog="[{\"entries_num\":41399,\"timestamp_from\":451816313806651394,\"timestamp_to\":451816375332110337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037358\",\"log_size\":53719,\"logID\":451809678470037358,\"memory_size\":331192},{\"entries_num\":40599,\"timestamp_from\":451816372317192194,\"timestamp_to\":451816391912456193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037384\",\"log_size\":42206,\"logID\":451809678470037384,\"memory_size\":324792},{\"entries_num\":40699,\"timestamp_from\":451816377743572993,\"timestamp_to\":451816392961032193,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037406\",\"log_size\":42326,\"logID\":451809678470037406,\"memory_size\":325592},{\"entries_num\":40499,\"timestamp_from\":451816349392699393,\"timestamp_to\":451816385267367939,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037449\",\"log_size\":42123,\"logID\":451809678470037449,\"memory_size\":323992},{\"entries_num\":40599,\"timestamp_from\":451816351739150337,\"timestamp_to\":451816371832225793,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037491\",\"log_size\":42225,\"logID\":451809678470037491,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816356811374593,\"timestamp_to\":451816366707572738,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037538\",\"log_size\":42550,\"logID\":451809678470037538,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816334699528193,\"timestamp_to\":451816363679809537,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037579\",\"log_size\":42212,\"logID\":451809678470037579,\"memory_size\":324792},{\"entries_num\":40699,\"timestamp_from\":451816330269556739,\"timestamp_to\":451816348921102337,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037611\",\"log_size\":42324,\"logID\":451809678470037611,\"memory_size\":325592},{\"entries_num\":40499,\"timestamp_from\":451816325629345793,\"timestamp_to\":451816334266990593,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037678\",\"log_size\":42115,\"logID\":451809678470037678,\"memory_size\":323992},{\"entries_num\":40799,\"timestamp_from\":451816320871694337,\"timestamp_to\":451816411651899393,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037713\",\"log_size\":42414,\"logID\":451809678470037713,\"memory_size\":326392},{\"entries_num\":40599,\"timestamp_from\":451816400969531393,\"timestamp_to\":451816412635201539,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037757\",\"log_size\":42226,\"logID\":451809678470037757,\"memory_size\":324792},{\"entries_num\":40599,\"timestamp_from\":451816393472475137,\"timestamp_to\":451816407746215937,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037807\",\"log_size\":42221,\"logID\":451809678470037807,\"memory_size\":324792},{\"entries_num\":11998,\"timestamp_from\":451816395962843139,\"timestamp_to\":451816397915553793,\"log_path\":\"file/insert_log/451809678468808897/451809678468808898/451809678470037289/100/451809678470037839\",\"log_size\":12777,\"logID\":451809678470037839,\"memory_size\":95984}]"] [current_index_version=5] [load_duration=44m58.271465815s]

It seems that it got stuck in the cgo pool and didn't exit in time. PTAL @MrPresent-Han /assign @MrPresent-Han

xiaocai2333 commented 1 month ago

Reverse lookup of original data takes too long.

for (int i = 0; i < row_count; ++i) {
    insert_record_.insert_pk(int64_index->Reverse_Lookup(i),  i);
}

/assign @zhagnlu

xiaofan-luan commented 1 month ago

反向查找原始数据的时间太长。

for (int i = 0; i < row_count; ++i) {
    insert_record_.insert_pk(int64_index->Reverse_Lookup(i),  i);
}

/分配@zhagnlu

can you explian this code? why load need to reverse_lookup?

xiaofan-luan commented 1 month ago

@zhagnlu

A cache may take too much memory for the index.

Actaully, It doesn't really make sense to create bitmap index on primary key because primary key is always unique.

Let't ban the behaviour to do so.

Load with 40 minutes seems to be really one, is primary key field load the only problem?

wangting0128 commented 3 weeks ago

The PR that prohibits the primary key from building the bitmap index has been merged, and the test is being rerun after removing the primary key bitmap index

wangting0128 commented 3 weeks ago

verify that load passed

image：master-20240828-56b3490a-amd64 argo task：fouramf-bgq75-rerun

截屏2024-08-29 11 16 24