search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.45k stars 2.92k forks source link

[Bug]: [benchmark][cluster] queryNode panic `Segmentation fault` in concurrent dql & multi-partition scene #37649

Open wangting0128 opened 1 day ago

wangting0128 commented 1 day ago

Is there an existing issue for this?

Environment

- Milvus version:master-20241113-3389a6b5-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc97
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-bitmap-scenes-rg47r test case name: test_bitmap_locust_hybrid_index_cluster

server:

NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-bitmap-scenes-rg47r-10-etcd-0                             1/1     Running            0                3h9m    10.104.20.89    4am-node22   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-etcd-1                             1/1     Running            0                3h9m    10.104.27.13    4am-node31   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-etcd-2                             1/1     Running            0                3h9m    10.104.24.68    4am-node29   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-milvus-datanode-9b449c5dc-b9zbg    1/1     Running            3 (3h8m ago)     3h9m    10.104.32.110   4am-node39   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-milvus-indexnode-866977c64dc5bgf   1/1     Running            3 (3h8m ago)     3h9m    10.104.34.37    4am-node37   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-milvus-indexnode-866977c64dnmbb7   1/1     Running            4 (3h7m ago)     3h9m    10.104.15.223   4am-node20   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-milvus-mixcoord-7f4f65c487-7sd48   1/1     Running            3 (3h7m ago)     3h9m    10.104.20.68    4am-node22   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-milvus-proxy-778bc9df7-swjd6       1/1     Running            4 (3h7m ago)     3h9m    10.104.21.177   4am-node24   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-milvus-querynode-784c695d7df567g   0/1     CrashLoopBackOff   29 (3m26s ago)   3h9m    10.104.18.99    4am-node25   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-minio-0                            1/1     Running            0                3h9m    10.104.20.87    4am-node22   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-minio-1                            1/1     Running            0                3h9m    10.104.24.66    4am-node29   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-minio-2                            1/1     Running            0                3h9m    10.104.27.14    4am-node31   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-minio-3                            1/1     Running            0                3h9m    10.104.33.143   4am-node36   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-bookie-0                    1/1     Running            0                3h9m    10.104.33.140   4am-node36   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-bookie-1                    1/1     Running            0                3h9m    10.104.20.90    4am-node22   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-bookie-2                    1/1     Running            0                3h9m    10.104.27.17    4am-node31   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-bookie-init-zltw5           0/1     Completed          0                3h9m    10.104.6.215    4am-node13   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-broker-0                    1/1     Running            0                3h9m    10.104.27.8     4am-node31   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-proxy-0                     1/1     Running            0                3h9m    10.104.4.119    4am-node11   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-pulsar-init-gflmp           0/1     Completed          0                3h9m    10.104.20.67    4am-node22   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-recovery-0                  1/1     Running            0                3h9m    10.104.24.54    4am-node29   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-zookeeper-0                 1/1     Running            0                3h9m    10.104.20.86    4am-node22   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-zookeeper-1                 1/1     Running            0                3h7m    10.104.26.50    4am-node32   <none>           <none>
fouramf-bitmap-scenes-rg47r-10-pulsar-zookeeper-2                 1/1     Running            0                3h6m    10.104.27.28    4am-node31   <none>           <none>

milvus_panic.log

截屏2024-11-13 17 40 35

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64`
            1. building default index on all supported 16 scalar fields
            2. load and search partial partitions & DQL requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=9
            2. build indexes:
                IVF_SQ8: 'float_vector'

                default scalar index: all scalar fields
            3. insert 6 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - scene_test_partition
                    (partition: create->insert->flush->index again->load->search->release->search failed->drop)
                - scene_test_partition_hybrid_search
                    (partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop)
                - search
                - query

Milvus Log

No response

Anything else?

server config:

{
     "queryNode": {
          "resources": {
               "limits": {
                    "cpu": "16.0",
                    "memory": "32Gi"
               },
               "requests": {
                    "cpu": "9.0",
                    "memory": "17Gi"
               }
          },
          "replicas": 1
     },
     "indexNode": {
          "resources": {
               "limits": {
                    "cpu": "8.0",
                    "memory": "8Gi"
               },
               "requests": {
                    "cpu": "5.0",
                    "memory": "5Gi"
               }
          },
          "replicas": 2
     },
     "dataNode": {
          "resources": {
               "limits": {
                    "cpu": "8.0",
                    "memory": "16Gi"
               },
               "requests": {
                    "cpu": "5.0",
                    "memory": "9Gi"
               }
          }
     },
     "cluster": {
          "enabled": true
     },
     "pulsar": {},
     "kafka": {},
     "minio": {
          "metrics": {
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "etcd": {
          "metrics": {
               "enabled": true,
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "metrics": {
          "serviceMonitor": {
               "enabled": true
          }
     },
     "log": {
          "level": "debug"
     },
     "image": {
          "all": {
               "repository": "harbor.milvus.io/milvus/milvus",
               "tag": "master-20241113-3389a6b5-amd64"
          }
     }
}

client config:

{
     "dataset_params": {
          "metric_type": "L2",
          "dim": 128,
          "max_length": 100,
          "scalars_index": {
               "id": {},
               "int8_1": {},
               "int16_1": {},
               "int32_1": {},
               "int64_1": {},
               "double_1": {},
               "float_1": {},
               "varchar_1": {},
               "bool_1": {},
               "array_int8_1": {},
               "array_int16_1": {},
               "array_int32_1": {},
               "array_int64_1": {},
               "array_double_1": {},
               "array_float_1": {},
               "array_varchar_1": {},
               "array_bool_1": {}
          },
          "scalars_params": {
               "array_int8_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_int16_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_int32_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_int64_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_double_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_float_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_varchar_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "array_bool_1": {
                    "params": {
                         "max_capacity": 9
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "int8_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "int16_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "int32_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "int64_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "double_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "float_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "varchar_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               },
               "bool_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope_custom_size",
                              "specify_range": [
                                   0,
                                   5000
                              ],
                              "base_size": 1000,
                              "custom_size": {
                                   "101000": [
                                        0,
                                        1,
                                        2,
                                        3,
                                        4,
                                        5,
                                        6,
                                        7,
                                        8,
                                        9
                                   ]
                              },
                              "max_capacity": 9
                         }
                    }
               }
          },
          "dataset_name": "sift",
          "dataset_size": 6000000,
          "ni_per": 5000
     },
     "collection_params": {
          "other_fields": [
               "int8_1",
               "int16_1",
               "int32_1",
               "int64_1",
               "double_1",
               "float_1",
               "varchar_1",
               "bool_1",
               "array_int8_1",
               "array_int16_1",
               "array_int32_1",
               "array_int64_1",
               "array_double_1",
               "array_float_1",
               "array_varchar_1",
               "array_bool_1"
          ],
          "shards_num": 3
     },
     "resource_groups_params": {
          "reset": false
     },
     "database_user_params": {
          "reset_rbac": false,
          "reset_db": false
     },
     "index_params": {
          "index_type": "IVF_SQ8",
          "index_param": {
               "nlist": 1024
          }
     },
     "concurrent_params": {
          "concurrent_number": [
               20
          ],
          "during_time": "3h",
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "scene_test_partition",
               "weight": 1,
               "params": {
                    "data_size": 3000,
                    "ni": 3000,
                    "nq": 1,
                    "search_param": {
                         "nprobe": 64
                    },
                    "limit": 1,
                    "expr": null,
                    "output_fields": [
                         "*"
                    ],
                    "guarantee_timestamp": null,
                    "timeout": 600,
                    "search_counts": 1
               }
          },
          {
               "type": "scene_test_partition_hybrid_search",
               "weight": 1,
               "params": {
                    "nq": 1,
                    "top_k": 1,
                    "reqs": [
                         {
                              "search_param": {
                                   "nprobe": 128
                              },
                              "anns_field": "float_vector",
                              "top_k": 100
                         },
                         {
                              "search_param": {
                                   "nprobe": 64
                              },
                              "anns_field": "float_vector",
                              "top_k": 10
                         }
                    ],
                    "rerank": {
                         "RRFRanker": []
                    },
                    "output_fields": [
                         "*"
                    ],
                    "ignore_growing": false,
                    "guarantee_timestamp": null,
                    "timeout": 600,
                    "random_data": true,
                    "hybrid_search_counts": 1,
                    "data_size": 3000,
                    "ni": 3000
               }
          },
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "nq": 1000,
                    "top_k": 10,
                    "search_param": {
                         "nprobe": 16
                    },
                    "expr": "int8_1 == 100",
                    "guarantee_timestamp": null,
                    "partition_names": [
                         "_default"
                    ],
                    "output_fields": [
                         "id",
                         "float_vector",
                         "int64_1"
                    ],
                    "ignore_growing": false,
                    "group_by_field": null,
                    "timeout": 600,
                    "random_data": true,
                    "check_task": "check_search_output",
                    "check_items": {
                         "nq": 1000
                    }
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "ids": null,
                    "expr": "int64_1 > -1",
                    "output_fields": [
                         "*"
                    ],
                    "offset": null,
                    "limit": 10,
                    "ignore_growing": false,
                    "partition_names": [
                         "_default"
                    ],
                    "timeout": 600,
                    "consistency_level": null,
                    "random_data": false,
                    "random_count": 0,
                    "random_range": [
                         0,
                         1
                    ],
                    "field_name": "id",
                    "field_type": "int64",
                    "check_task": "check_query_output",
                    "check_items": {
                         "expect_length": 10
                    }
               }
          },
          {
               "type": "hybrid_search",
               "weight": 1,
               "params": {
                    "nq": 10,
                    "top_k": 10,
                    "reqs": [
                         {
                              "search_param": {
                                   "nprobe": 128
                              },
                              "anns_field": "float_vector",
                              "expr": "(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, 1)) || ((varchar_1 like \"1%\") and (bool_1 == True))",
                              "top_k": 100
                         },
                         {
                              "search_param": {
                                   "nprobe": 64
                              },
                              "anns_field": "float_vector",
                              "expr": "not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, 1])"
                         }
                    ],
                    "rerank": {
                         "RRFRanker": []
                    },
                    "output_fields": [
                         "*"
                    ],
                    "ignore_growing": false,
                    "guarantee_timestamp": null,
                    "partition_names": [
                         "_default"
                    ],
                    "timeout": 600,
                    "random_data": true,
                    "check_task": "check_search_output",
                    "check_items": {
                         "output_fields": [
                              "int8_1",
                              "int16_1",
                              "int32_1",
                              "int64_1",
                              "double_1",
                              "float_1",
                              "varchar_1",
                              "bool_1",
                              "array_int8_1",
                              "array_int16_1",
                              "array_int32_1",
                              "array_int64_1",
                              "array_double_1",
                              "array_float_1",
                              "array_varchar_1",
                              "array_bool_1",
                              "id",
                              "float_vector"
                         ],
                         "nq": 10
                    }
               }
          }
     ]
}
sunby commented 1 day ago

/assign

sunby commented 2 hours ago

@sunby There's a bug when calculating next batch size when two fields has inverted index. I will fix it and add more unit tests