Closed wangting0128 closed 2 months ago
wangting0128
commented
2 months ago After receiving the request to drop the partition, milvus performs GetIndexInfos, GetIndexInfos will load the released partition again, which will cause an error when dropping the partition: partition cannot be dropped, partition is loaded
{pod=~"bitmap-dql-dml-partitions-milvus-standalone-697bbd944-gmhm6"} |~ "5042adfc26aeaba15778f4cddc08012e|scene_test_partition_LNBcQVjY"
yanliang567
commented
2 months ago /assign @longjiquan /unassign
longjiquan
commented
2 months ago Concurrent load/release/drop caused this issue. /unassign /assign @wangting0128
longjiquan
commented
2 months ago 
xiaofan-luan
commented
2 months ago shall this be in a queue to execute? @longjiquan
wangting0128
commented
2 months ago 
load collection and verification passedtest result:
[2024-08-22 12:18:41,247 - INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-08-22 12:18:41,247 - INFO - fouram]: Type Name # reqs # fails | Avg Min Max Med | req/s failures/s (stats.py:789)
[2024-08-22 12:18:41,247 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: grpc hybrid_search 214 0(0.00%) | 3490 617 32145 2000 | 0.02 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: grpc query 210 0(0.00%) | 1632 13 9759 650 | 0.02 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: grpc scene_insert_partition 237 0(0.00%) | 59806 3670 153865 56000 | 0.02 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: grpc scene_test_partition 228 0(0.00%) | 637299 71412 1642622 616000 | 0.02 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: grpc scene_test_partition_hybrid_search 222 0(0.00%) | 697161 57983 1664454 665000 | 0.02 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: grpc search 244 0(0.00%) | 3895 927 32528 2400 | 0.02 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: Aggregated 1355 0(0.00%) | 233423 13 1664454 11000 | 0.13 0.00 (stats.py:789)
[2024-08-22 12:18:41,248 - INFO - fouram]: (stats.py:790)
[2024-08-22 12:18:41,252 - INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'standalone',
'config_name': 'standalone_8c16m',
'config': {'standalone': {'resources': {'limits': {'cpu': '16', 'memory': '64Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}},
'cluster': {'enabled': False},
'etcd': {'replicaCount': 1, 'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
'minio': {'mode': 'standalone', 'metrics': {'podMonitor': {'enabled': True}}},
'pulsar': {'enabled': False},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'indexNode': {'replicas': 2},
'queryNode': {'replicas': 2},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20240822-3107701f-amd64'}}},
'host': 'bitmap-dql-dml-partitions-rerun-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_bitmap_locust_dql_dml_partitions_cluster',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'max_length': 100,
'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
'int16_1': {'index_type': 'BITMAP'},
'int32_1': {'index_type': 'BITMAP'},
'int64_1': {'index_type': 'BITMAP'},
'varchar_1': {'index_type': 'BITMAP'},
'bool_1': {'index_type': 'BITMAP'},
'array_int8_1': {'index_type': 'BITMAP'},
'array_int16_1': {'index_type': 'BITMAP'},
'array_int32_1': {'index_type': 'BITMAP'},
'array_int64_1': {'index_type': 'BITMAP'},
'array_varchar_1': {'index_type': 'BITMAP'},
'array_bool_1': {'index_type': 'BITMAP'}},
'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
'index_param': {'M': 8, 'efConstruction': 200},
'metric_type': 'L2'}},
'scalars_params': {'array_int8_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-128, 128],
'max_capacity': 13}}},
'array_int16_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-200, 200],
'max_capacity': 13}}},
'array_int32_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'specify_scope',
'specify_range': [-300, 300],
'max_capacity': 13}}},
'array_int64_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'fixed_value_range',
'specify_range': [-400, 432],
'batch': 50,
'max_capacity': 13}}},
'array_varchar_1': {'params': {'max_capacity': 13},
'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-1500, 1500],
'max_capacity': 13}}},
'array_bool_1': {'params': {'max_capacity': 13}},
'int8_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-128, 128],
'max_capacity': 13}}},
'int16_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-200, 200],
'max_capacity': 13}}},
'int32_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'specify_scope',
'specify_range': [-300, 300],
'max_capacity': 13}}},
'int64_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'fixed_value_range',
'specify_range': [-400, 432],
'batch': 50,
'max_capacity': 13}}},
'varchar_1': {'other_params': {'dataset': 'random_algorithm',
'algorithm_params': {'algorithm_name': 'random_range',
'specify_range': [-1500, 1500],
'max_capacity': 13}}}},
'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'data_repeated': False},
'dataset_name': 'sift',
'dataset_size': 5000000,
'ni_per': 5000},
'collection_params': {'other_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1', 'bool_1',
'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1', 'array_varchar_1',
'array_bool_1'],
'shards_num': 16},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False, 'reset_db': False},
'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 30, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
'concurrent_tasks': [{'type': 'scene_insert_partition',
'weight': 1,
'params': {'data_size': 3000, 'ni': 1000, 'with_flush': True, 'timeout': 600}},
{'type': 'scene_test_partition',
'weight': 1,
'params': {'data_size': 3000,
'ni': 3000,
'nq': 1,
'search_param': {'nprobe': 64},
'limit': 1,
'expr': None,
'output_fields': ['*'],
'guarantee_timestamp': None,
'timeout': 600}},
{'type': 'scene_test_partition_hybrid_search',
'weight': 1,
'params': {'nq': 1,
'top_k': 1,
'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
{'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'timeout': 600,
'random_data': True,
'data_size': 3000,
'ni': 3000}},
{'type': 'search',
'weight': 1,
'params': {'nq': 1000,
'top_k': 10,
'search_param': {'nprobe': 16},
'expr': 'int8_1 == 100',
'guarantee_timestamp': None,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'output_fields': ['id', 'float_vector', 'int64_1'],
'ignore_growing': False,
'group_by_field': None,
'timeout': None,
'random_data': True,
'check_task': 'check_search_output',
'check_items': None}},
{'type': 'query',
'weight': 1,
'params': {'ids': None,
'expr': 'int64_1 > -1',
'output_fields': ['*'],
'offset': None,
'limit': 10,
'ignore_growing': False,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'timeout': None,
'random_data': False,
'random_count': 0,
'random_range': [0, 1],
'field_name': 'id',
'field_type': 'int64',
'check_task': 'check_query_output',
'check_items': {'expect_length': 10}}},
{'type': 'hybrid_search',
'weight': 1,
'params': {'nq': 10,
'top_k': 10,
'reqs': [{'search_param': {'nprobe': 128},
'anns_field': 'float_vector',
'expr': '(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, '
'1)) || ((varchar_1 like "1%") and (bool_1 == True))',
'top_k': 100},
{'search_param': {'ef': 64},
'anns_field': 'float_vector_1',
'expr': 'not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, '
'1])'}],
'rerank': {'RRFRanker': []},
'output_fields': ['*'],
'ignore_growing': False,
'guarantee_timestamp': None,
'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
'partition_5', 'partition_6', 'partition_7', 'partition_8',
'partition_9'],
'timeout': None,
'random_data': True,
'check_task': 'check_search_output',
'check_items': {'output_fields': ['float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1',
'varchar_1', 'bool_1', 'array_int8_1', 'array_int16_1',
'array_int32_1', 'array_int64_1', 'array_varchar_1',
'array_bool_1', 'id', 'float_vector'],
'nq': 10}}}]},
'run_id': 2024082234653923,
'datetime': '2024-08-22 07:57:45.054286',
'client_version': '2.2'},
'result': {'test_result': {'index': {'RT': 3827.4489,
'float_vector_1': {'RT': 7.1541},
'int8_1': {'RT': 0.5183},
'int16_1': {'RT': 0.5222},
'int32_1': {'RT': 2.0356},
'int64_1': {'RT': 0.8006},
'varchar_1': {'RT': 0.5181},
'bool_1': {'RT': 0.5166},
'array_int8_1': {'RT': 0.5271},
'array_int16_1': {'RT': 0.5215},
'array_int32_1': {'RT': 0.5197},
'array_int64_1': {'RT': 0.5219},
'array_varchar_1': {'RT': 0.5362},
'array_bool_1': {'RT': 0.5193}},
'insert': {'total_time': 638.9451, 'VPS': 8142.8233, 'batch_time': 0.639, 'batch': 5000.0},
'flush': {'RT': 6.0304},
'load': {'RT': 8.7945},
'Locust': {'Aggregated': {'Requests': 1355,
'Fails': 0,
'RPS': 0.13,
'fail_s': 0.0,
'RT_max': 1664454.35,
'RT_avg': 233423.53,
'TP50': 11000.0,
'TP99': 1600000.0},
'hybrid_search': {'Requests': 214,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 32145.93,
'RT_avg': 3490.77,
'TP50': 2000.0,
'TP99': 31000.0},
'query': {'Requests': 210,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 9759.9,
'RT_avg': 1632.76,
'TP50': 650.0,
'TP99': 8900.0},
'scene_insert_partition': {'Requests': 237,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 153865.29,
'RT_avg': 59806.9,
'TP50': 56000.0,
'TP99': 141000.0},
'scene_test_partition': {'Requests': 228,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 1642622.74,
'RT_avg': 637299.37,
'TP50': 617000.0,
'TP99': 1582000.0},
'scene_test_partition_hybrid_search': {'Requests': 222,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 1664454.35,
'RT_avg': 697161.62,
'TP50': 665000.0,
'TP99': 1659000.0},
'search': {'Requests': 244,
'Fails': 0,
'RPS': 0.02,
'fail_s': 0.0,
'RT_max': 32528.96,
'RT_avg': 3895.91,
'TP50': 2400.0,
'TP99': 14000.0}}}}}
shall this be in a queue to execute? @longjiquan
yes, they are scheduled to be executed in sequence from queue, but even we(client) do release_partition, load_collection, drop_partition in sequence, this issue still exists.
longjiquan
commented
2 months ago The load_collection will reload the released partitions, and then they are not allowed to be dropped.
Is there an existing issue for this?
Environment
Current Behavior
argo task: fouramf-concurrent-wkxjs test case name: test_bitmap_locust_dql_dml_partitions_cluster
server:
{pod=~"bitmap-dql-dml-partitions-milvus-standalone-697bbd944-gmhm6"} |~ "scene_test_partition_LNBcQVjY"client pod name: fouramf-concurrent-wkxjs-2635105422 client log:
hybrid_search error
test result:
Expected Behavior
birdwtacher:
Steps To Reproduce
Milvus Log
No response
Anything else?
No response