Open ThreadDao opened 6 days ago
/unassign
search failed fixed, but deletegator and 2 querynodes oom
compact-opt-100m-2-milvus-querynode-1-68b66fcf88-22v7w 1/1 Running 2 (3h54m ago) 26h 10.104.23.52 4am-node27 <none> <none>
compact-opt-100m-2-milvus-querynode-1-68b66fcf88-g9rdb 1/1 Running 1 (3h53m ago) 26h 10.104.34.82 4am-node37 <none> <none>
compact-opt-100m-2-milvus-querynode-1-68b66fcf88-vxdnw 1/1 Running 2 (3h34m ago) 26h 10.104.30.132 4am-node38 <none> <none>
compact-opt-100m-2-milvus-querynode-1-68b66fcf88-zbqhq 1/1 Running 3 (3h52m ago) 26h 10.104.24.50 4am-node29 <none> <none>
[2024-10-18 14:39:08,390 - INFO - ci_test]: start to delete [4880000, ..., 4939999] with length 60000 (tmp.py:40)
...
[2024-10-18 14:44:47,257 - INFO - ci_test]: delete cost 0.43274760246276855 with res (insert count: 0, delete count: 60000, upsert count: 0, timestamp: 0, success count: 0, err count: 0 (tmp.py:44)
[2024-10-18 14:44:47,830 - INFO - ci_test]: start to delete [25100000, ..., 25159999] with length 60000 (tmp.py:40)
RPC error: [delete], <MilvusException: (code=9, message=quota exceeded[reason=memory quota exceeded, please allocate more resources])>, <Time:{'RPC start': '2024-10-18 14:44:47.846284', 'RPC error': '2024-10-18 14:44:47.913232'}>
Traceback (most recent call last):
seems that it hits quota limitation and shouldn't be OOMed?
The OOM happened after "memory quota exceeded" log?
Is there an existing issue for this?
Environment
Current Behavior
milvus cluster
deploy a milvus with config
test steps
client delete log:
Expected Behavior
No response
Steps To Reproduce
compact-opt-100m-2-milvus-datanode-74b5c7854b-xxcdl 1/1 Running 0 3h53m 10.104.14.7 4am-node18
compact-opt-100m-2-milvus-indexnode-6cd9b49f5-9xtfj 1/1 Running 0 3h52m 10.104.4.36 4am-node11
compact-opt-100m-2-milvus-indexnode-6cd9b49f5-qb26s 1/1 Running 0 3h53m 10.104.17.2 4am-node23
compact-opt-100m-2-milvus-indexnode-6cd9b49f5-zp5bj 1/1 Running 0 3h51m 10.104.1.234 4am-node10
compact-opt-100m-2-milvus-mixcoord-8f9875d6d-khsb4 1/1 Running 0 3h53m 10.104.4.33 4am-node11
compact-opt-100m-2-milvus-proxy-5bd9875bb4-tkrzw 1/1 Running 0 3h53m 10.104.9.107 4am-node14
compact-opt-100m-2-milvus-querynode-0-7488f76b9b-8dz69 1/1 Running 0 3h52m 10.104.20.48 4am-node22
compact-opt-100m-2-milvus-querynode-0-7488f76b9b-dqwzf 1/1 Running 0 3h49m 10.104.23.93 4am-node27
compact-opt-100m-2-milvus-querynode-0-7488f76b9b-hs6hn 1/1 Running 0 3h53m 10.104.24.147 4am-node29
compact-opt-100m-2-milvus-querynode-0-7488f76b9b-p5dcv 1/1 Running 0 3h50m 10.104.30.192 4am-node38