search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.91k stars 2.95k forks source link

[Bug]: [benchmark][cluster] queryNode panic `called Result::unwrap() on an Err value: OpenReadError(FileDoesNotExist("*/*.idx"))` in continuous load-release collection scenario #32154

Closed wangting0128 closed 7 months ago

wangting0128 commented 7 months ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4-20240410-beb3b552-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-scene-mix-vdnjj test case name: test_hybrid_search_locust_load_release_replica_cluster

server:

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-scene-mix-vdnjj-2-etcd-0                             1/1     Running                           0               6m43s   10.104.17.78    4am-node23   <none>           <none>
multi-vector-scene-mix-vdnjj-2-etcd-1                             1/1     Running                           0               6m42s   10.104.18.137   4am-node25   <none>           <none>
multi-vector-scene-mix-vdnjj-2-etcd-2                             1/1     Running                           0               6m42s   10.104.27.99    4am-node31   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-datacoord-565475cb842qrwl   1/1     Running                           0               6m45s   10.104.21.137   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-datanode-7f7c49679-lhj4d    1/1     Running                           1 (2m13s ago)   6m45s   10.104.21.139   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-indexcoord-5bb9f559fxrbxf   1/1     Running                           0               6m45s   10.104.21.136   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-indexnode-5cd5bf84588zdgm   1/1     Running                           0               6m45s   10.104.33.251   4am-node36   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-proxy-7486c5c7cf-kz64s      1/1     Running                           1 (2m14s ago)   6m46s   10.104.21.135   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-querycoord-84fb78b84dlfr6   1/1     Running                           1 (2m14s ago)   6m45s   10.104.21.138   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-querynode-676674c76bnhdd8   1/1     Running                           0               6m44s   10.104.26.97    4am-node32   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-querynode-676674c76btr5zz   1/1     Running                           0               6m45s   10.104.9.173    4am-node14   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-rootcoord-55b86cb5f6gmkbl   1/1     Running                           1 (2m15s ago)   6m46s   10.104.21.134   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-0                            1/1     Running                           0               6m43s   10.104.17.76    4am-node23   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-1                            1/1     Running                           0               6m43s   10.104.25.171   4am-node30   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-2                            1/1     Running                           0               6m42s   10.104.18.138   4am-node25   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-3                            1/1     Running                           0               6m42s   10.104.27.98    4am-node31   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-0                    1/1     Running                           0               6m42s   10.104.25.170   4am-node30   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-1                    1/1     Running                           0               6m42s   10.104.17.80    4am-node23   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-2                    1/1     Running                           0               6m41s   10.104.23.47    4am-node27   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-init-qr56z           0/1     Completed                         0               6m46s   10.104.4.31     4am-node11   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-broker-0                    1/1     Running                           0               6m44s   10.104.4.33     4am-node11   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-proxy-0                     1/1     Running                           0               6m44s   10.104.14.189   4am-node18   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-pulsar-init-78gvh           0/1     Completed                         0               6m46s   10.104.4.30     4am-node11   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-recovery-0                  1/1     Running                           0               6m44s   10.104.1.86     4am-node10   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-zookeeper-0                 1/1     Running                           0               6m44s   10.104.24.128   4am-node29   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-zookeeper-1                 1/1     Running                           0               5m32s   10.104.34.89    4am-node37   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-zookeeper-2                 1/1     Running                           0               2m55s   10.104.15.173   4am-node20   <none>           <none> (base.py:257)
[2024-04-10 22:16:01,227 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|multi-vector-scene-mix-vdnjj-2-milvus|multi-vector-scene-mix-vdnjj-2-minio|multi-vector-scene-mix-vdnjj-2-etcd|multi-vector-scene-mix-vdnjj-2-pulsar|multi-vector-scene-mix-vdnjj-2-zookeeper|multi-vector-scene-mix-vdnjj-2-kafka|multi-vector-scene-mix-vdnjj-2-log|multi-vector-scene-mix-vdnjj-2-tikv'  (util_cmd.py:14)
[2024-04-10 22:16:11,082 -  INFO - fouram]: [CliClient] pod details of release(multi-vector-scene-mix-vdnjj-2): 
 I0410 22:16:02.470573     510 request.go:665] Waited for 1.173785693s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/storage.k8s.io/v1beta1?timeout=32s
NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-scene-mix-vdnjj-2-etcd-0                             1/1     Running                           0               15h     10.104.17.78    4am-node23   <none>           <none>
multi-vector-scene-mix-vdnjj-2-etcd-1                             1/1     Running                           0               15h     10.104.18.137   4am-node25   <none>           <none>
multi-vector-scene-mix-vdnjj-2-etcd-2                             1/1     Running                           0               15h     10.104.27.99    4am-node31   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-datacoord-565475cb842qrwl   1/1     Running                           0               15h     10.104.21.137   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-datanode-7f7c49679-lhj4d    1/1     Running                           1 (15h ago)     15h     10.104.21.139   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-indexcoord-5bb9f559fxrbxf   1/1     Running                           0               15h     10.104.21.136   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-indexnode-5cd5bf84588zdgm   1/1     Running                           0               15h     10.104.33.251   4am-node36   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-proxy-7486c5c7cf-kz64s      1/1     Running                           1 (15h ago)     15h     10.104.21.135   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-querycoord-84fb78b84dlfr6   1/1     Running                           1 (15h ago)     15h     10.104.21.138   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-querynode-676674c76bnhdd8   1/1     Running                           2 (8h ago)      15h     10.104.26.97    4am-node32   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-querynode-676674c76btr5zz   1/1     Running                           12 (3m5s ago)   15h     10.104.9.173    4am-node14   <none>           <none>
multi-vector-scene-mix-vdnjj-2-milvus-rootcoord-55b86cb5f6gmkbl   1/1     Running                           1 (15h ago)     15h     10.104.21.134   4am-node24   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-0                            1/1     Running                           0               15h     10.104.17.76    4am-node23   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-1                            1/1     Running                           0               15h     10.104.25.171   4am-node30   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-2                            1/1     Running                           0               15h     10.104.18.138   4am-node25   <none>           <none>
multi-vector-scene-mix-vdnjj-2-minio-3                            1/1     Running                           0               15h     10.104.27.98    4am-node31   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-0                    1/1     Running                           0               15h     10.104.25.170   4am-node30   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-1                    1/1     Running                           0               15h     10.104.17.80    4am-node23   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-2                    1/1     Running                           0               15h     10.104.23.47    4am-node27   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-bookie-init-qr56z           0/1     Completed                         0               15h     10.104.4.31     4am-node11   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-broker-0                    1/1     Running                           0               15h     10.104.4.33     4am-node11   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-proxy-0                     1/1     Running                           0               15h     10.104.14.189   4am-node18   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-pulsar-init-78gvh           0/1     Completed                         0               15h     10.104.4.30     4am-node11   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-recovery-0                  1/1     Running                           0               15h     10.104.1.86     4am-node10   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-zookeeper-0                 1/1     Running                           0               15h     10.104.24.128   4am-node29   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-zookeeper-1                 1/1     Running                           0               15h     10.104.34.89    4am-node37   <none>           <none>
multi-vector-scene-mix-vdnjj-2-pulsar-zookeeper-2                 1/1     Running                           0               15h     10.104.15.173   4am-node20   <none>           <none>
截屏2024-04-11 12 01 48 截屏2024-04-11 12 02 52 截屏2024-04-11 12 03 18

querynode-676674c76btr5zz_panic.log

2024-04-10 10:56:38.678 
thread '<unnamed>' panicked at src/index_reader.rs:25:14:

2024-04-10 10:56:38.678 
called `Result::unwrap()` on an `Err` value: OpenReadError(FileDoesNotExist("/var/lib/milvus/data/querynode/index_files/448982217358165391/1/16bdf3519aba4637b49b573fe33ddda7.term"))
2024-04-10 10:56:38.678 
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2024-04-10 10:56:38.678 
fatal runtime error: failed to initiate panic, error 5
2024-04-10 10:56:38.680 
SIGABRT: abort
2024-04-10 10:56:38.680 
PC=0x7f2cdc00003b m=2816 sigcode=18446744073709551610
2024-04-10 10:56:38.680 
signal arrived during cgo execution
2024-04-10 10:56:38.680 

2024-04-10 10:56:38.680 
goroutine 192094 [syscall, locked to thread]:
2024-04-10 10:56:39.165 
thread '<unnamed>' panicked at src/index_reader.rs:25:14:
2024-04-10 10:56:39.165 
called `Result::unwrap()` on an `Err` value: OpenReadError(FileDoesNotExist("/var/lib/milvus/data/querynode/index_files/448982217358177804/1/3415bc07c7aa4f29b292162dc27e645c.idx"))

2024-04-10 10:56:39.165 
fatal runtime error: failed to initiate panic, error 5
2024-04-10 10:56:52.188 
non-Go function
2024-04-10 10:56:52.188 
    pc=0x7f2cdc00003b
2024-04-10 10:56:52.188 
non-Go function
2024-04-10 10:56:52.188 
    pc=0x7f2cdbfdf858
2024-04-10 10:56:52.211 
_ZN3std3sys4unix14abort_internal17hb82186f9b9b64ef6E
2024-04-10 10:56:52.211 
    library/std/src/sys/unix/mod.rs:365 pc=0x7f2cdb04b786
2024-04-10 10:56:52.211 
rust_panic
2024-04-10 10:56:52.211 
    library/std/src/panicking.rs:758 pc=0x7f2cdb03fb73
2024-04-10 10:56:52.211 
_ZN3std9panicking20rust_panic_with_hook17h57e78470c47c84deE
2024-04-10 10:56:52.211 
    library/std/src/panicking.rs:729 pc=0x7f2cdb03f9d1
2024-04-10 10:56:52.211 
_ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h3dfd2453cf356ecbE
2024-04-10 10:56:52.211 
    library/std/src/panicking.rs:599 pc=0x7f2cdb03f716
2024-04-10 10:56:52.211 
_ZN3std10sys_common9backtrace26__rust_end_short_backtrace17hdb177d43678e4d7eE
2024-04-10 10:56:52.211 
    library/std/src/sys_common/backtrace.rs:170 pc=0x7f2cdb03ca05
2024-04-10 10:56:52.211 
rust_begin_unwind
2024-04-10 10:56:52.211 
    library/std/src/panicking.rs:595 pc=0x7f2cdb03f461
2024-04-10 10:56:52.218 
_ZN4core9panicking9panic_fmt17hd1e971d8d7c78e0eE
2024-04-10 10:56:52.218 
    library/core/src/panicking.rs:67 pc=0x7f2cdaa75222
2024-04-10 10:56:52.218 
runtime.cgocall(0x490bf00, 0xc002056e58)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc002056e30 sp=0xc002056df8 pc=0x1b4f8bc
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/internal/querynodev2/segments._Cfunc_AppendIndexV2({0xc0029a5780, 0xc0029a5790, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, ...)
2024-04-10 10:56:52.218 
    _cgo_gotypes.go:230 +0x55 fp=0xc002056e58 sp=0xc002056e30 pc=0x46da175
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/internal/querynodev2/segments.(*LoadIndexInfo).appendIndexData.func1.2({0xc0029a5780, 0xc0029a5790, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}, ...)
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/load_index_info.go:226 +0x67 fp=0xc002056eb8 sp=0xc002056e58 pc=0x46e42c7
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/internal/querynodev2/segments.(*LoadIndexInfo).appendIndexData.func1()
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/load_index_info.go:226 +0xb6 fp=0xc002056f28 sp=0xc002056eb8 pc=0x46e4216
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1()
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81 +0xbc fp=0xc002056f88 sp=0xc002056f28 pc=0x46c7b3c
2024-04-10 10:56:52.218 
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
2024-04-10 10:56:52.218 
    /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67 +0x97 fp=0xc002056fe0 sp=0xc002056f88 pc=0x35d4137
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002056fe8 sp=0xc002056fe0 pc=0x1bc0c21
2024-04-10 10:56:52.218 
created by github.com/panjf2000/ants/v2.(*goWorker).run
2024-04-10 10:56:52.218 
    /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:48 +0x65
2024-04-10 10:56:52.218 

2024-04-10 10:56:52.218 
goroutine 1 [chan receive, 258 minutes]:
2024-04-10 10:56:52.218 
runtime.gopark(0x795f110?, 0xc0021a3770?, 0xa?, 0x9b?, 0xc0005b23c0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0025ed700 sp=0xc0025ed6e0 pc=0x1b8a356
2024-04-10 10:56:52.218 
runtime.chanrecv(0xc0004164e0, 0x0, 0x1)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0025ed790 sp=0xc0025ed700 pc=0x1b526fd
2024-04-10 10:56:52.218 
runtime.chanrecv1(0x7947e08?, 0x52738c0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0025ed7b8 sp=0xc0025ed790 pc=0x1b521f8
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0005eb360)
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:399 +0xafa fp=0xc0025edc78 sp=0xc0025ed7b8 pc=0x480fe3a
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x0?, {0xc000052180?, 0x3, 0x3}, 0xc0004e0780)
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:47 +0x2e5 fp=0xc0025edd48 sp=0xc0025edc78 pc=0x481ad85
2024-04-10 10:56:52.218 
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000052180?, 0x3, 0x3})
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x20e fp=0xc0025eddc0 sp=0xc0025edd48 pc=0x481aa0e
2024-04-10 10:56:52.218 
main.main()
2024-04-10 10:56:52.218 
    /go/src/github.com/milvus-io/milvus/cmd/main.go:95 +0x3e5 fp=0xc0025edf80 sp=0xc0025eddc0 pc=0x481f745
2024-04-10 10:56:52.218 
runtime.main()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc0025edfe0 sp=0xc0025edf80 pc=0x1b89f27
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0025edfe8 sp=0xc0025edfe0 pc=0x1bc0c21
2024-04-10 10:56:52.218 

2024-04-10 10:56:52.218 
goroutine 2 [force gc (idle), 45 minutes]:
2024-04-10 10:56:52.218 
runtime.gopark(0x2824424b849869?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000140fb0 sp=0xc000140f90 pc=0x1b8a356
2024-04-10 10:56:52.218 
runtime.goparkunlock(...)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:387
2024-04-10 10:56:52.218 
runtime.forcegchelper()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000140fe0 sp=0xc000140fb0 pc=0x1b8a190
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000140fe8 sp=0xc000140fe0 pc=0x1bc0c21
2024-04-10 10:56:52.218 
created by runtime.init.6
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:293 +0x25
2024-04-10 10:56:52.218 

2024-04-10 10:56:52.218 
goroutine 3 [GC sweep wait]:
2024-04-10 10:56:52.218 
runtime.gopark(0x7909901?, 0x79099e0?, 0xc?, 0x14?, 0x1?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000141780 sp=0xc000141760 pc=0x1b8a356
2024-04-10 10:56:52.218 
runtime.goparkunlock(...)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:387
2024-04-10 10:56:52.218 
runtime.bgsweep(0x0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0001417c8 sp=0xc000141780 pc=0x1b72a1e
2024-04-10 10:56:52.218 
runtime.gcenable.func1()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0001417e0 sp=0xc0001417c8 pc=0x1b67ae6
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001417e8 sp=0xc0001417e0 pc=0x1bc0c21
2024-04-10 10:56:52.218 
created by runtime.gcenable
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgc.go:178 +0x6b
2024-04-10 10:56:52.218 

2024-04-10 10:56:52.218 
goroutine 4 [GC scavenge wait]:
2024-04-10 10:56:52.218 
runtime.gopark(0x28268c535c4985?, 0x6a506a?, 0x0?, 0x0?, 0x0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000141f70 sp=0xc000141f50 pc=0x1b8a356
2024-04-10 10:56:52.218 
runtime.goparkunlock(...)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:387
2024-04-10 10:56:52.218 
runtime.(*scavengerState).park(0x79123c0)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000141fa0 sp=0xc000141f70 pc=0x1b70853
2024-04-10 10:56:52.218 
runtime.bgscavenge(0x0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000141fc8 sp=0xc000141fa0 pc=0x1b70e65
2024-04-10 10:56:52.218 
runtime.gcenable.func2()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000141fe0 sp=0xc000141fc8 pc=0x1b67a86
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x1bc0c21
2024-04-10 10:56:52.218 
created by runtime.gcenable
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgc.go:179 +0xaa
2024-04-10 10:56:52.218 

2024-04-10 10:56:52.218 
goroutine 5 [finalizer wait]:
2024-04-10 10:56:52.218 
runtime.gopark(0x0?, 0x548dc70?, 0xc0?, 0xe3?, 0x1000000010?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001d5be28 sp=0xc001d5be08 pc=0x1b8a356
2024-04-10 10:56:52.218 
runtime.runfinq()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc001d5bfe0 sp=0xc001d5be28 pc=0x1b66aa7
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001d5bfe8 sp=0xc001d5bfe0 pc=0x1bc0c21
2024-04-10 10:56:52.218 
created by runtime.createfing
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mfinal.go:163 +0x45
2024-04-10 10:56:52.218 

2024-04-10 10:56:52.218 
goroutine 6 [GC worker (idle), 260 minutes]:
2024-04-10 10:56:52.218 
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000142750 sp=0xc000142730 pc=0x1b8a356
2024-04-10 10:56:52.218 
runtime.gcBgMarkWorker()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001427e0 sp=0xc000142750 pc=0x1b69851
2024-04-10 10:56:52.218 
runtime.goexit()
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001427e8 sp=0xc0001427e0 pc=0x1bc0c21
2024-04-10 10:56:52.218 
created by runtime.gcBgMarkStartWorkers
2024-04-10 10:56:52.218 
    /usr/local/go/src/runtime/mgc.go:1199 +0x25

client pod name: multi-vector-scene-mix-vdnjj-1627126718

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `load -> release collection, replica=2`
            verify load -> release collection scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 5 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 2
            7. concurrent request: (concurrent_number=1)
                - load_release

Milvus Log

No response

Anything else?

test result:

[2024-04-10 22:15:42,701 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-04-10 22:15:42,702 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-04-10 22:15:42,702 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-10 22:15:42,702 -  INFO - fouram]: grpc     load_release                                                                     935     0(0.00%) |  46175    1829  180624  31000 |    0.02        0.00 (stats.py:789)
[2024-04-10 22:15:42,702 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-10 22:15:42,702 -  INFO - fouram]:          Aggregated                                                                       935     0(0.00%) |  46175    1829  180624  31000 |    0.02        0.00 (stats.py:789)
[2024-04-10 22:15:42,702 -  INFO - fouram]:  (stats.py:790)
[2024-04-10 22:15:42,704 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '32Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '17Gi'}},
                                     'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '8Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '5Gi'}},
                                     'replicas': 1},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '8Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240410-beb3b552-amd64'}}},
            'host': 'multi-vector-scene-mix-vdnjj-2-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_load_release_replica_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {},
                                                                      'int64_1': {'index_type': 'INVERTED'},
                                                                      'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 5000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int64_1',
                                                                        'varchar_1'],
                                                       'shards_num': 2},
                                 'load_params': {'replica_number': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT',
                                                  'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 1,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'load_release',
                                                       'weight': 1,
                                                       'params': {'replica_number': 2,
                                                                  'timeout': 600}}]},
            'run_id': 2024041009621551,
            'datetime': '2024-04-10 06:36:02.519054',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 4448.5947,
                                      'float_vector_1': {'RT': 4171.7617},
                                      'float_vector_2': {'RT': 2378.3071},
                                      'float_vector_3': {'RT': 958.6572},
                                      'id': {'RT': 2.0276},
                                      'int64_1': {'RT': 0.5159},
                                      'varchar_1': {'RT': 0.5143}},
                            'insert': {'total_time': 536.7463,
                                       'VPS': 9315.3879,
                                       'batch_time': 1.0735,
                                       'batch': 10000},
                            'flush': {'RT': 3.0257},
                            'load': {'RT': 46.3867},
                            'Locust': {'Aggregated': {'Requests': 935,
                                                      'Fails': 0,
                                                      'RPS': 0.02,
                                                      'fail_s': 0.0,
                                                      'RT_max': 180624.0,
                                                      'RT_avg': 46175.64,
                                                      'TP50': 31000.0,
                                                      'TP99': 131000.0},
                                       'load_release': {'Requests': 935,
                                                        'Fails': 0,
                                                        'RPS': 0.02,
                                                        'fail_s': 0.0,
                                                        'RT_max': 180624.0,
                                                        'RT_avg': 46175.64,
                                                        'TP50': 31000.0,
                                                        'TP99': 131000.0}}}}}
czs007 commented 7 months ago

@longjiquan image

image

what 's the purpose of /tmp/ prefix?

wangting0128 commented 7 months ago

verification passed

image：2.4-20240418-238f9a4a-amd64