[Bug]: No L0 compaction tasks completed

ThreadDao commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version: 2.4-20240705-261b61e8
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

create collection with 2 shards, enable partition_key with 32 num_partitions
create index
insert 30m-128d data and flush
index and load
concurrent requests: search + upsert + flush
No L0 compaction tasks completed from the grafana monitor metrics of level-zero-par-key-op-2-7900

By the way, import metrics no data !!

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/8c535d6e-68ea-4dc3-ac49-13772107181f?nodeId=level-zero-stable-1720105200-try-2331900307

Milvus Log

pods:

level-zero-par-key-op-2-7900-etcd-0                               1/1     Running            0              12h     10.104.17.197   4am-node23   <none>           <none>
level-zero-par-key-op-2-7900-etcd-1                               1/1     Running            0              12h     10.104.20.18    4am-node22   <none>           <none>
level-zero-par-key-op-2-7900-etcd-2                               1/1     Running            0              12h     10.104.16.224   4am-node21   <none>           <none>
level-zero-par-key-op-2-7900-milvus-datanode-57d586fcf4-ggscp     1/1     Running            0              12h     10.104.20.45    4am-node22   <none>           <none>
level-zero-par-key-op-2-7900-milvus-datanode-57d586fcf4-j8hth     1/1     Running            0              12h     10.104.32.112   4am-node39   <none>           <none>
level-zero-par-key-op-2-7900-milvus-indexnode-b64979c56-qtgn4     1/1     Running            0              12h     10.104.5.45     4am-node12   <none>           <none>
level-zero-par-key-op-2-7900-milvus-indexnode-b64979c56-sqchw     1/1     Running            0              12h     10.104.6.74     4am-node13   <none>           <none>
level-zero-par-key-op-2-7900-milvus-mixcoord-6795788c48-wpj48     1/1     Running            0              12h     10.104.17.223   4am-node23   <none>           <none>
level-zero-par-key-op-2-7900-milvus-proxy-7fd59c66d5-g4zg9        1/1     Running            0              12h     10.104.26.210   4am-node32   <none>           <none>
level-zero-par-key-op-2-7900-milvus-querynode-0-8479f96668b9bdn   1/1     Running            0              12h     10.104.15.75    4am-node20   <none>           <none>
level-zero-par-key-op-2-7900-milvus-querynode-0-8479f96668hl8j7   1/1     Running            0              12h     10.104.21.140   4am-node24   <none>           <none>
level-zero-par-key-op-2-7900-milvus-querynode-0-8479f96668rl89n   1/1     Running            0              12h     10.104.26.211   4am-node32   <none>           <none>
level-zero-par-key-op-2-7900-milvus-querynode-0-8479f96668xvb8x   1/1     Running            0              12h     10.104.19.10    4am-node28   <none>           <none>
level-zero-par-key-op-2-7900-minio-0                              1/1     Running            0              12h     10.104.20.20    4am-node22   <none>           <none>
level-zero-par-key-op-2-7900-minio-1                              1/1     Running            0              12h     10.104.17.196   4am-node23   <none>           <none>
level-zero-par-key-op-2-7900-minio-2                              1/1     Running            0              12h     10.104.16.225   4am-node21   <none>           <none>
level-zero-par-key-op-2-7900-minio-3                              1/1     Running            0              12h     10.104.18.252   4am-node25   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-bookie-0                      1/1     Running            0              12h     10.104.20.21    4am-node22   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-bookie-1                      1/1     Running            0              12h     10.104.17.198   4am-node23   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-bookie-2                      1/1     Running            0              12h     10.104.16.229   4am-node21   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-bookie-init-mpzp2             0/1     Completed          0              12h     10.104.1.237    4am-node10   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-broker-0                      1/1     Running            0              12h     10.104.1.238    4am-node10   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-proxy-0                       1/1     Running            0              12h     10.104.17.192   4am-node23   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-pulsar-init-lfsfm             0/1     Completed          0              12h     10.104.20.12    4am-node22   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-recovery-0                    1/1     Running            0              12h     10.104.14.216   4am-node18   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-zookeeper-0                   1/1     Running            0              12h     10.104.20.19    4am-node22   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-zookeeper-1                   1/1     Running            0              12h     10.104.17.200   4am-node23   <none>           <none>
level-zero-par-key-op-2-7900-pulsar-zookeeper-2                   1/1     Running            0              12h     10.104.16.235   4am-node21   <none>           <none>

Anything else?

No response

ThreadDao commented 1 month ago

Other issue instance: level-zero-insert-op-40-4759

pods:

level-zero-insert-op-40-4759-milvus-datanode-84c8455dc5-nk5bz     1/1     Running            0              12h     10.104.13.192   4am-node16   <none>           <none>
level-zero-insert-op-40-4759-milvus-datanode-84c8455dc5-rpjvj     1/1     Running            0              12h     10.104.32.113   4am-node39   <none>           <none>
level-zero-insert-op-40-4759-milvus-indexnode-7875d97cf7-6zwg6    1/1     Running            0              12h     10.104.13.194   4am-node16   <none>           <none>
level-zero-insert-op-40-4759-milvus-indexnode-7875d97cf7-qmfwg    1/1     Running            0              12h     10.104.14.221   4am-node18   <none>           <none>
level-zero-insert-op-40-4759-milvus-mixcoord-c9f4cffd5-bswwg      1/1     Running            0              12h     10.104.13.193   4am-node16   <none>           <none>
level-zero-insert-op-40-4759-milvus-proxy-6f547dcfc4-tf68j        1/1     Running            0              12h     10.104.13.191   4am-node16   <none>           <none>
level-zero-insert-op-40-4759-milvus-querynode-0-6f46cf6dbfc2rbv   1/1     Running            0              12h     10.104.17.229   4am-node23   <none>           <none>
level-zero-insert-op-40-4759-milvus-querynode-0-6f46cf6dbffqkfb   1/1     Running            0              12h     10.104.6.75     4am-node13   <none>           <none>
level-zero-insert-op-40-4759-milvus-querynode-0-6f46cf6dbfkrxfv   1/1     Running            0              12h     10.104.18.7     4am-node25   <none>           <none>
level-zero-insert-op-40-4759-milvus-querynode-0-6f46cf6dbfmds8v   1/1     Running            0              12h     10.104.1.242    4am-node10   <none>           <none>

yanliang567 commented 1 month ago

/assign @XuanYang-cn /unassign

XuanYang-cn commented 1 month ago

/assign @ThreadDao /unassign Please help verify

XuanYang-cn commented 1 month ago

/assign

ThreadDao commented 1 month ago

fixed 2.4-20240711-1d2062a6-amd64

ThreadDao commented 1 month ago

@XuanYang-cn It seems that there is still a problem with the delete in the upsert scenario image: 2.4-20240711-86b57b78-amd64 metrics of level-zero-upsert-op-53-4572

pods:

level-zero-upsert-op-53-4572-milvus-datanode-8468b55c8-tc45l      1/1     Running            0                8h      10.104.32.50    4am-node39   <none>           <none>
level-zero-upsert-op-53-4572-milvus-datanode-8468b55c8-vc8d8      1/1     Running            0                8h      10.104.19.168   4am-node28   <none>           <none>
level-zero-upsert-op-53-4572-milvus-indexnode-744dd9b9b-5jmp5     1/1     Running            0                8h      10.104.33.185   4am-node36   <none>           <none>
level-zero-upsert-op-53-4572-milvus-indexnode-744dd9b9b-smmrx     1/1     Running            0                8h      10.104.19.169   4am-node28   <none>           <none>
level-zero-upsert-op-53-4572-milvus-mixcoord-6f84c9c544-v9gvg     1/1     Running            0                8h      10.104.6.81     4am-node13   <none>           <none>
level-zero-upsert-op-53-4572-milvus-proxy-744dd89f5c-cwgdd        1/1     Running            0                8h      10.104.14.65    4am-node18   <none>           <none>
level-zero-upsert-op-53-4572-milvus-querynode-0-6fdf4b78d8kvm6d   1/1     Running            0                8h      10.104.30.99    4am-node38   <none>           <none>
level-zero-upsert-op-53-4572-milvus-querynode-0-6fdf4b78d8mln7g   1/1     Running            0                8h      10.104.14.66    4am-node18   <none>           <none>
level-zero-upsert-op-53-4572-milvus-querynode-0-6fdf4b78d8sgvnr   1/1     Running            0                8h      10.104.16.158   4am-node21   <none>           <none>
level-zero-upsert-op-53-4572-milvus-querynode-0-6fdf4b78d8xjnk7   1/1     Running            0                8h      10.104.13.150   4am-node16   <none>           <none>

czs007 commented 1 month ago

@ThreadDao please verify use latest 2.4 branch

ThreadDao commented 1 month ago

fixed 2.4-20240716-dfb41582-amd64

milvus-io / milvus