Open NicoYuan1986 opened 2 weeks ago
- Milvus version: 02bd916 - Deployment mode(standalone or cluster): cluster - MQ type(rocksmq, pulsar or kafka): kafka - SDK version(e.g. pymilvus v2.0.0rc2): - OS(Ubuntu or CentOS): - CPU/Memory: - GPU: - Others:
Sealed segment is unexpected after load balance.
[pytest : test] _________________ TestUtilityAdvanced.test_load_balance_normal _________________ [pytest : test] [gw4] linux -- Python 3.8.17 /usr/local/bin/python3 [pytest : test] [pytest : test] self = <test_utility.TestUtilityAdvanced object at 0x7fca1fb55a00> [pytest : test] [pytest : test] @pytest.mark.tags(CaseLabel.L2) [pytest : test] def test_load_balance_normal(self): [pytest : test] """ [pytest : test] target: test load balance of collection [pytest : test] method: init a collection and load balance [pytest : test] expected: sealed_segment_ids is subset of des_sealed_segment_ids [pytest : test] """ [pytest : test] # init a collection [pytest : test] self._connect() [pytest : test] querynode_num = len(MilvusSys().query_nodes) [pytest : test] if querynode_num < 2: [pytest : test] pytest.skip("skip load balance testcase when querynode number less than 2") [pytest : test] c_name = cf.gen_unique_str(prefix) [pytest : test] collection_w = self.init_collection_wrap(name=c_name) [pytest : test] collection_w.create_index(default_field_name, default_index_params) [pytest : test] ms = MilvusSys() [pytest : test] nb = 3000 [pytest : test] df = cf.gen_default_dataframe_data(nb) [pytest : test] collection_w.insert(df) [pytest : test] # get sealed segments [pytest : test] collection_w.num_entities [pytest : test] # get growing segments [pytest : test] collection_w.insert(df) [pytest : test] collection_w.load() [pytest : test] # prepare load balance params [pytest : test] time.sleep(0.5) [pytest : test] res, _ = self.utility_wrap.get_query_segment_info(c_name) [pytest : test] segment_distribution = cf.get_segment_distribution(res) [pytest : test] all_querynodes = [node["identifier"] for node in ms.query_nodes] [pytest : test] assert len(all_querynodes) > 1 [pytest : test] all_querynodes = sorted(all_querynodes, [pytest : test] key=lambda x: len(segment_distribution[x]["sealed"]) [pytest : test] if x in segment_distribution else 0, reverse=True) [pytest : test] src_node_id = all_querynodes[0] [pytest : test] des_node_ids = all_querynodes[1:] [pytest : test] sealed_segment_ids = segment_distribution[src_node_id]["sealed"] [pytest : test] # load balance [pytest : test] self.utility_wrap.load_balance(collection_w.name, src_node_id, des_node_ids, sealed_segment_ids) [pytest : test] # get segments distribution after load balance [pytest : test] time.sleep(0.5) [pytest : test] res, _ = self.utility_wrap.get_query_segment_info(c_name) [pytest : test] segment_distribution = cf.get_segment_distribution(res) [pytest : test] sealed_segment_ids_after_load_banalce = segment_distribution[src_node_id]["sealed"] [pytest : test] # assert src node has no sealed segments [pytest : test] > assert sealed_segment_ids_after_load_banalce == [] [pytest : test] E assert [453117048979248010] == [] [pytest : test] E Left contains one more item: 453117048979248010 [pytest : test] E Full diff: [pytest : test] E - [] [pytest : test] E + [453117048979248010] [pytest : test]
pass
No response
/assign @weiliu1031 /unassign
Is there an existing issue for this?
Environment
Current Behavior
Sealed segment is unexpected after load balance.
Expected Behavior
pass
Steps To Reproduce
No response
Milvus Log
Anything else?
No response