[Bug]: After deleting about 300 million data: failed to query: segment lacks[segment=xxx]: channel not available

[ThreadDao]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: 2.4-20240911-134422ee-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

• dn: 1*4c16g
• qn: 14*8c32g

  config:
  dataCoord:
    enableActiveStandby: true
    compaction:
      **enableAutoCompaction: false**
    segment:
      expansionRate: 1.15
      maxSize: 512 
      sealProportion: 0.12
  indexCoord:
    enableActiveStandby: true
  log:
    level: debug
  minio:
    accessKeyID: miniozong
    bucketName: bucket-zong
    rootPath: compact
    secretAccessKey: miniozong
  queryCoord:
    enableActiveStandby: true
  quotaAndLimits:
    dml:
      deleteRate:
      enabled: false
    growingSegmentsSizeProtection:
      enabled: false
    limitWriting:
      memProtection:
        dataNodeMemoryHighWaterLevel: 0.9
        dataNodeMemoryLowWaterLevel: 0.85
        queryNodeMemoryHighWaterLevel: 0.9
        queryNodeMemoryLowWaterLevel: 0.85
    limits:
      complexDeleteLimitEnable: true
  rootCoord:
    enableActiveStandby: true
  trace:
    exporter: jaeger
    jaeger:
      url: http://tempo-distributor.tempo:14268/api/traces"
    sampleFraction: 1

  test steps

  1. Collection fouram_MGRdheUl has 1 shard. Loaded 458,050,000 128dim entities with queryNode 14*8c32g
  2. Delete with expr id >= 0 and gets error failed to search/query delegator 12 for channel compact-opt-rate-100m-1-rootcoord-dml_0_452472962528840167v0: quota exceeded[reason=memory quota exceeded, please allocate more resources])
  3. Delegator OOM
  4. scale-out queryNode to 17*8c32g, and enable autoCompaction by dynamic refresh config
  5. The number of delegator L0 segments decreased as the L0 compaction task started. However, the segment handoff did not seem to be successful. After the 3-hour compaction finished, the query still reported an error.

     pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=failed to query: segment lacks[segment=452472962529121144]: channel not available[channel=compact-opt-rate-100m-1-rootcoord-dml_0_452472962528840167v0])>

metrics

https://grafana-4am.zilliz.cc/d/uLf5cJ3Ga/milvus2-0?orgId=1&var-datasource=P1809F7CD0C75ACF3&var-namespace=qa-milvus&var-instance=compact-opt-rate-100m-1&var-collection=All&var-app_name=milvus&from=1726112083863&to=1726116529531

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @XuanYang-cn /unassign