Closed Nik96i closed 9 months ago
yanliang567
commented
1 year ago @Nik96i if it works fine after 15mins, i think its okay. According to the logs, the proxy is waiting for rootcoord and the other pods ready during the 10-15mins.
/assign @Nik96i
Nik96i
commented
1 year ago I run it multiple times and it doesn't work anymore. "error="err: failed to connect 10.0.0.62:53100, reason: context deadline exceeded" what is the difference between docker swarm and docker compose ? can u please help me fix this issue. thaks.
yanliang567
commented
1 year ago i am not an expert in docker, but usually rootcoord does not turn to ready because it cannot connect to etcd or minio. Could you please double check if they are working well?
Nik96i
commented
1 year ago I change the docker-compose file to this: https://file.io/OZQMvmCKXWSd export minio ports and set console-address and address in commands section. it's accessible from outside and it works fine but the standalone logs changed and cannot connect to minio. here is the full log: https://file.io/7hsR3YiUtAxx
the minio container is accessible from the standalone container, I test it with curl inside the standalone like this: HTTP://minio:9000 but in the logs the standalone doesn't see minio.
yanliang567
commented
1 year ago /assign @LoveEachDay any ideas?
Nik96i
commented
1 year ago Hi again. I created an overlay docker network with the "attachable" option set to true.
docker network create --driver=overlay --attachable=true public-network
now when I run Milvus with docker-compose and use the external network I made before it works fine, but when I run Milvus with the docker stack it doesn't work. Milvus doesn't work on swarm mode. the containers see each other but don't work. it returns a "proxy not healthy" error.
Full log: https://file.io/ywbEikqYcj8Z
stale[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
EeyoreLee
commented
1 year ago I run it multiple times and it doesn't work anymore. "error="err: failed to connect 10.0.0.62:53100, reason: context deadline exceeded" what is the difference between docker swarm and docker compose ? can u please help me fix this issue. thaks.
same situation. maybe cause each service uses different local ip in swarm mode but milvus has it's own service discovery.
farizap
commented
1 year ago Hi @Nik96i, I also found this issue when running milvus standalone using docker swam. I fixed it by changing the address of indexCoord, queryCoord, rootCoord, and dataCoord from localhost to 0.0.0.0 in the milvus.yaml. You can see my milvus.yaml file here
https://gist.github.com/farizap/31896c8a96e9c41dac376cda20c23086
EeyoreLee
commented
1 year ago @farizap Hi, thanks for your reply, but it still not work by following your solution. Could you please drop out a docker-compose.yaml using swarm mode.
ieugen
commented
1 year ago I think this is related to: https://github.com/milvus-io/milvus/issues/17191 (auto-closed, not fixed).
The issue IMO is the fact that proxy component (maybe others) does not allow you to specify an address for proxy .
It is very important to allow people to customize the addresses the app listens on. An app can listen to multiple interfaces / ip addresses at once.
For convenience, most use IPV4 "0.0.0.0" which binds to all network interface.
We also have an issue with this. We can't run milvus in Docker swarm because it randomly picks up an ip address to bind to and in swarm, you can have multiple interfaces. See issue https://github.com/milvus-io/milvus/issues/17191 for code references.
xiaofan-luan
commented
1 year ago I don't think we ever tried on docker swarm. But if there is a quickfix the maintainer team can help on that.
@LoveEachDay could you help on investigating this issue?
xiaofan-luan
commented
1 year ago anyone else want to fix this issue is also highly welcomed
farizap
commented
1 year ago Hi @EeyoreLee, sorry for the late reply. Here my docker-compose
version: '3.5'
services:
etcd:
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ./volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
minio:
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
volumes:
- ./volumes/minio:/minio_data
command: minio server /minio_data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.2.11
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ./volumes/milvus:/var/lib/milvus
- ./milvus.yaml:/milvus/configs/milvus.yaml
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
networks:
default:
external:
name: 2p5I start the service by using this command
docker network create --driver overlay 2p5
docker stack deploy -c docker-compose.yml milvusHello,
I found some time to look at this again and I believe there is a bug here that affects milvus 2.3.0 as well and manifests I deployed milvus as part of a stack with a configuration like the one @farizap has published. I updated to milvus 2.3.0.
Milvus failed to start and it prints out TONS of logs. The warning logs should be errors perhaps since the app fails to start.
I included also the network interfaces from inside the container and the listening ports.
As you can see from those logs, the app starts and binds to IPV6 on all interfaces to the port 53100 . When the client in the app tries to connect to 10.0.0.235:53100 (ip of the host and the port) it fails.
Could it be that milvus does not work when IPV6 networks are involved ?
[11:00:01.205] [DEBUG] [client/client.go:101] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[11:00:01.206] [WARN] [grpcclient/client.go:147] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[11:00:01.206] [WARN] [client/client.go:130] ["ClientBase ReCall grpc second call get error"] [role=datacoord] [address=] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:387 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:425 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:216 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit\n"]
[11:00:01.206] [DEBUG] [client/client.go:95] ["QueryCoordClient msess key not existed"] [key=querycoord]
[11:00:01.206] [WARN] [grpcclient/client.go:147] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[11:00:01.206] [WARN] [client/client.go:122] ["ClientBase ReCall grpc second call get error"] [role=querycoord] [address=] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:387 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:122 github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:294 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:191 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit\n"]
[11:00:01.206] [WARN] [rootcoord/quota_center.go:146] ["quotaCenter sync metrics failed"] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:387 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:425 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:216 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit\n"]
[11:00:03.178] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.235:53100]
[11:00:03.178] [DEBUG] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.235:53100] [serverID=44]
[11:00:03.379] [WARN] [client/client.go:133] ["ClientBase ReCall grpc first call get error "] [role=rootcoord] [address=] [error="err: failed to connect 10.0.0.235:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:378 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:133 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:147 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"]
[11:00:03.379] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.235:53100]
[11:00:03.379] [DEBUG] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.235:53100] [serverID=44]
[11:00:03.579] [WARN] [client/client.go:133] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [address=] [error="err: failed to connect 10.0.0.235:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:387 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:133 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:147 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"]
[11:00:03.607] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.235:53100]
[11:00:03.607] [DEBUG] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.235:53100] [serverID=44]
[11:00:03.645] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.235:53100]
[11:00:03.645] [DEBUG] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.235:53100] [serverID=44]
[11:00:03.808] [WARN] [client/client.go:133] ["ClientBase ReCall grpc first call get error "] [role=rootcoord] [address=] [error="err: failed to connect 10.0.0.235:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:378 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:133 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:147 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"]
[11:00:03.809] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.235:53100]
[11:00:03.809] [DEBUG] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.235:53100] [serverID=44]To include network information use apt update ; apt install iproute2 net-tools while exec'ed inside container .
root@96f48e7eeff2:/milvus# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
1382: eth0@if1383: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:00:00:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.0.0.235/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
1384: eth2@if1385: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:0f brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet 172.18.0.15/16 brd 172.18.255.255 scope global eth2
valid_lft forever preferred_lft forever
1386: eth1@if1387: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:0a:00:07:0e brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 10.0.7.14/24 brd 10.0.7.255 scope global eth1
valid_lft forever preferred_lft forever
root@96f48e7eeff2:/milvus# netstat -tlupn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:44549 0.0.0.0:* LISTEN -
tcp6 0 0 :::53100 :::* LISTEN 8/milvus
tcp6 46 0 :::19530 :::* LISTEN 8/milvus
tcp6 0 0 :::19531 :::* LISTEN 8/milvus
tcp6 0 0 :::19529 :::* LISTEN 8/milvus
tcp6 0 0 :::21124 :::* LISTEN 8/milvus
tcp6 0 0 :::21123 :::* LISTEN 8/milvus
tcp6 0 0 :::21121 :::* LISTEN 8/milvus
tcp6 0 0 :::9091 :::* LISTEN 8/milvus
tcp6 0 0 :::13333 :::* LISTEN 8/milvus
udp 0 0 127.0.0.11:44416 0.0.0.0:* - After some time if finaly failed with error:
[2023/09/20 11:17:07.948 +00:00] [DEBUG] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.235:53100] [serverID=44]
[2023/09/20 11:17:08.091 +00:00] [ERROR] [datacoord/handler.go:407] ["datacoord ServerHandler HasCollection finally failed"] [collectionID=443936088141151033] [stack="github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:407\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:492\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:349\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:315\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2023/09/20 11:17:08.091 +00:00] [PANIC] [datacoord/handler.go:381] ["datacoord ServerHandler HasCollection finally failed"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:381\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:407\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:492\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:349\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:315\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
panic: datacoord ServerHandler HasCollection finally failed
goroutine 378 [running]:
panic({0x3a15220, 0xc00337fd30})
/usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc00417ea10 sp=0xc00417e950 pc=0x153fbdb
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000e4e300, {0x0, 0x0, 0x0})
/go/pkg/mod/go.uber.org/zap@v1.17.0/zapcore/entry.go:234 +0x49b fp=0xc00417ebd0 sp=0xc00417ea10 pc=0x1729b1b
go.uber.org/zap.(*Logger).Panic(0xc0036b9800?, {0x4085b27?, 0xc00417ec40?}, {0x0, 0x0, 0x0})
/go/pkg/mod/go.uber.org/zap@v1.17.0/logger.go:227 +0x59 fp=0xc00417ec00 sp=0xc00417ebd0 pc=0x19836f9
github.com/milvus-io/milvus/pkg/log.Panic({0x4085b27?, 0x4085b27?}, {0x0?, 0xc0020cfa00?, 0x1?})
/go/src/github.com/milvus-io/milvus/pkg/log/global.go:58 +0x8b fp=0xc00417ec50 sp=0xc00417ec00 pc=0x1b1f8ab
github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection(0xc000e90d50, {0x44b3c98?, 0xc000e8a050?}, 0x6292d8524a13b39)
/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:381 +0x25f fp=0xc00417ed58 sp=0xc00417ec50 pc=0x2e3cfff
github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel(0xc000e90d50, {0xc001a30060, 0x2b}, 0x1b26e45?)
/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:407 +0xa5 fp=0xc00417eeb0 sp=0xc00417ed58 pc=0x2e3d625
github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels(0xc0015221a0)
/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:254 +0xf9 fp=0xc00417f0b8 sp=0xc00417eeb0 pc=0x2e16d79
github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup(0xc0015221a0, {0x44b3c98?, 0xc000e8a050}, {0x5d3dae0?, 0x0, 0x0})
/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:164 +0x773 fp=0xc00417f5d8 sp=0xc00417f0b8 pc=0x2e15db3
github.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup(0xc000dc1c20, {0x44b3c98, 0xc000e8a050}, {0x5d3dae0, 0x0, 0x1?})
/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57 +0x11a fp=0xc00417f638 sp=0xc00417f5d8 pc=0x2e2241a
github.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery(0xc0011f4000)
/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:492 +0x4bc fp=0xc00417fa80 sp=0xc00417f638 pc=0x2e7c93c
github.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord(0xc0011f4000)
/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:349 +0x39b fp=0xc00417fb98 sp=0xc00417fa80 pc=0x2e7b0fb
github.com/milvus-io/milvus/internal/datacoord.(*Server).Init(0xc0011f4000)
/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:315 +0xf8 fp=0xc00417fbd0 sp=0xc00417fb98 pc=0x2e7ab38
github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init(0xc0011e4060)
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:108 +0x4cc fp=0xc00417fe48 sp=0xc00417fbd0 pc=0x2ea6dac
github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run(0xc000344120?)
/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:229 +0x25 fp=0xc00417fe88 sp=0xc00417fe48 pc=0x2ea84e5
github.com/milvus-io/milvus/cmd/components.(*DataCoord).Run(0xc0010af9c0?)
/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49 +0x2f fp=0xc00417ff58 sp=0xc00417fe88 pc=0x374f7ef
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112 +0x159 fp=0xc00417ffe0 sp=0xc00417ff58 pc=0x37551f9
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00417ffe8 sp=0xc00417ffe0 pc=0x1579301
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...]
/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:99 +0x15c
goroutine 1 [semacquire, 28 minutes]:
runtime.gopark(0x0?, 0xc0004657b8?, 0x20?, 0x41?, 0xc0010502a0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0011c3778 sp=0xc0011c3758 pc=0x1543096
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc000afad38, 0x30?, 0x1, 0x0, 0x1?)
/usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc0011c37e0 sp=0xc0011c3778 pc=0x1554e6f
sync.runtime_Semacquire(0x153e47f?)
/usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc0011c3818 sp=0xc0011c37e0 pc=0x1574d07
sync.(*WaitGroup).Wait(0xc000d9e4c0?)
/usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc0011c3840 sp=0xc0011c3818 pc=0x1588f4b
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0011c3e48, 0x1, {0xc000d96090?, 0xe?})
/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:340 +0x8fa fp=0xc0011c3df8 sp=0xc0011c3840 pc=0x3754a3a
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xc0003fc0c0, {0xc000052090?, 0x3, 0x3}, 0xc000d902a0)
/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117 +0x68e fp=0xc0011c3ee0 sp=0xc0011c3df8 pc=0x3760f2e
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000052090?, 0x3, 0x3})
/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x21e fp=0xc0011c3f58 sp=0xc0011c3ee0 pc=0x376079e
main.main()
/go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e fp=0xc0011c3f80 sp=0xc0011c3f58 pc=0x376376e
runtime.main()
/usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc0011c3fe0 sp=0xc0011c3f80 pc=0x1542c67
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0011c3fe8 sp=0xc0011c3fe0 pc=0x1579301
goroutine 2 [force gc (idle), 28 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000084fb0 sp=0xc000084f90 pc=0x1543096
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000084fe0 sp=0xc000084fb0 pc=0x1542ed0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x1579301
created by runtime.init.6
/usr/local/go/src/runtime/proc.go:293 +0x25
goroutine 3 [GC sweep wait]:
runtime.gopark(0x5ced501?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000085780 sp=0xc000085760 pc=0x1543096
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
/usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000857c8 sp=0xc000085780 pc=0x152b81e
runtime.gcenable.func1()
/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x15208e6
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x1579301
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:178 +0x6b
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000aa000?, 0x447c178?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000085f70 sp=0xc000085f50 pc=0x1543096
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x5cf5c40)
/usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000085fa0 sp=0xc000085f70 pc=0x1529653
runtime.bgscavenge(0x0?)
/usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000085fc8 sp=0xc000085fa0 pc=0x1529c65
runtime.gcenable.func2()
/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x1520886
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x1579301
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:179 +0xaa
goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0x4104b08?, 0x0?, 0x80?, 0x1000000010?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc002fd7e28 sp=0xc002fd7e08 pc=0x1543096
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc002fd7fe0 sp=0xc002fd7e28 pc=0x151f927
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002fd7fe8 sp=0xc002fd7fe0 pc=0x1579301
created by runtime.createfing
/usr/local/go/src/runtime/mfinal.go:163 +0x45
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x544da09defb40?, 0x3?, 0x6b?, 0x10?, 0xc0000867a8?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000086750 sp=0xc000086730 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000867e0 sp=0xc000086750 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x544da09d5d5f1?, 0x3?, 0xc4?, 0xc5?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000086f50 sp=0xc000086f30 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000086fe0 sp=0xc000086f50 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 34 [GC worker (idle)]:
runtime.gopark(0x544da09dfdb6c?, 0x3?, 0x84?, 0x55?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000080750 sp=0xc000080730 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000807e0 sp=0xc000080750 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 19 [GC worker (idle)]:
runtime.gopark(0x544da09dfa085?, 0x3?, 0xfc?, 0x1?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000132750 sp=0xc000132730 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001327e0 sp=0xc000132750 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001327e8 sp=0xc0001327e0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x544da09d67f39?, 0x1?, 0xeb?, 0x4e?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000132f50 sp=0xc000132f30 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000132fe0 sp=0xc000132f50 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000132fe8 sp=0xc000132fe0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x544da09d702af?, 0x1?, 0x1?, 0x3b?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000133750 sp=0xc000133730 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001337e0 sp=0xc000133750 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001337e8 sp=0xc0001337e0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x544da09df0a90?, 0x1?, 0x65?, 0x2f?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000133f50 sp=0xc000133f30 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000133fe0 sp=0xc000133f50 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000133fe8 sp=0xc000133fe0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x5d40440?, 0x3?, 0xd8?, 0x28?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000087750 sp=0xc000087730 pc=0x1543096
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000877e0 sp=0xc000087750 pc=0x1522651
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x1579301
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1199 +0x25
goroutine 139 [select, 28 minutes]:
runtime.gopark(0xc000d37f98?, 0x2?, 0x2?, 0x0?, 0xc000d37f7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00155de00 sp=0xc00155dde0 pc=0x1543096
runtime.selectgo(0xc00155df98, 0xc000d37f78, 0xc000186fc0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00155df40 sp=0xc00155de00 pc=0x1553bbe
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000e62e00)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:115 +0x73 fp=0xc00155dfc8 sp=0xc00155df40 pc=0x1c46bd3
google.golang.org/grpc.newCCBalancerWrapper.func1()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x26 fp=0xc00155dfe0 sp=0xc00155dfc8 pc=0x1c46b26
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00155dfe8 sp=0xc00155dfe0 pc=0x1579301
created by google.golang.org/grpc.newCCBalancerWrapper
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x22a
goroutine 31 [select]:
runtime.gopark(0xc000135f80?, 0x2?, 0xe8?, 0x2a?, 0xc000135f54?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000135dc0 sp=0xc000135da0 pc=0x1543096
runtime.selectgo(0xc000135f80, 0xc000135f50, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000135f00 sp=0xc000135dc0 pc=0x1553bbe
github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc000176780, {0x44b3c98, 0xc0000b7220})
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:83 +0x10a fp=0xc000135fb8 sp=0xc000135f00 pc=0x28c9f4a
github.com/panjf2000/ants/v2.(*Pool).goPurge.func1()
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:147 +0x2e fp=0xc000135fe0 sp=0xc000135fb8 pc=0x28ca54e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000135fe8 sp=0xc000135fe0 pc=0x1579301
created by github.com/panjf2000/ants/v2.(*Pool).goPurge
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:147 +0xe5
goroutine 32 [select]:
runtime.gopark(0xc000080f80?, 0x2?, 0xb?, 0xc3?, 0xc000080f44?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000080dc8 sp=0xc000080da8 pc=0x1543096
runtime.selectgo(0xc000080f80, 0xc000080f40, 0xc00230ea20?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000080f08 sp=0xc000080dc8 pc=0x1553bbe
github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc000176780, {0x44b3c98, 0xc0000b7270})
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:125 +0x145 fp=0xc000080fb8 sp=0xc000080f08 pc=0x28ca2e5
github.com/panjf2000/ants/v2.(*Pool).goTicktock.func1()
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:154 +0x2e fp=0xc000080fe0 sp=0xc000080fb8 pc=0x28ca6ee
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x1579301
created by github.com/panjf2000/ants/v2.(*Pool).goTicktock
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:154 +0x115
goroutine 138 [select]:
runtime.gopark(0xc004371eb8?, 0x2?, 0xa4?, 0x1?, 0xc004371e7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc004371d00 sp=0xc004371ce0 pc=0x1543096
runtime.selectgo(0xc004371eb8, 0xc004371e78, 0xc000d2e0c0?, 0x0, 0x1519a45?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc004371e40 sp=0xc004371d00 pc=0x1553bbe
github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0xc000d90540, {0x3fdf54f, 0xa})
/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:68 +0x1f4 fp=0xc004371fb8 sp=0xc004371e40 pc=0x21ed354
github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.1()
/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0x2e fp=0xc004371fe0 sp=0xc004371fb8 pc=0x21ed04e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc004371fe8 sp=0xc004371fe0 pc=0x1579301
created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1
/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0xae
goroutine 195 [select, 28 minutes]:
runtime.gopark(0xc00214ff98?, 0x2?, 0x2?, 0x0?, 0xc00214ff7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001e1fe00 sp=0xc001e1fde0 pc=0x1543096
runtime.selectgo(0xc001e1ff98, 0xc00214ff78, 0xc000187260?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001e1ff40 sp=0xc001e1fe00 pc=0x1553bbe
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000e63400)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:115 +0x73 fp=0xc001e1ffc8 sp=0xc001e1ff40 pc=0x1c46bd3
google.golang.org/grpc.newCCBalancerWrapper.func1()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x26 fp=0xc001e1ffe0 sp=0xc001e1ffc8 pc=0x1c46b26
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001e1ffe8 sp=0xc001e1ffe0 pc=0x1579301
created by google.golang.org/grpc.newCCBalancerWrapper
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x22a
goroutine 159 [select]:
runtime.gopark(0xc000092f80?, 0x2?, 0x9?, 0x18?, 0xc000092f54?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000092dc0 sp=0xc000092da0 pc=0x1543096
runtime.selectgo(0xc000092f80, 0xc000092f50, 0x5435277560648?, 0x0, 0xc000092fd0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000092f00 sp=0xc000092dc0 pc=0x1553bbe
github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc00026e140, {0x44b3c98, 0xc000ed6d20})
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:83 +0x10a fp=0xc000092fb8 sp=0xc000092f00 pc=0x28c9f4a
github.com/panjf2000/ants/v2.(*Pool).goPurge.func1()
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:147 +0x2e fp=0xc000092fe0 sp=0xc000092fb8 pc=0x28ca54e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x1579301
created by github.com/panjf2000/ants/v2.(*Pool).goPurge
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:147 +0xe5
goroutine 194 [select]:
runtime.gopark(0xc00436beb8?, 0x2?, 0xa4?, 0x1?, 0xc00436be7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00436bd00 sp=0xc00436bce0 pc=0x1543096
runtime.selectgo(0xc00436beb8, 0xc00436be78, 0xc0010d2c00?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00436be40 sp=0xc00436bd00 pc=0x1553bbe
github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0xc000d90c60, {0x3fdf54f, 0xa})
/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:68 +0x1f4 fp=0xc00436bfb8 sp=0xc00436be40 pc=0x21ed354
github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.1()
/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0x2e fp=0xc00436bfe0 sp=0xc00436bfb8 pc=0x21ed04e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00436bfe8 sp=0xc00436bfe0 pc=0x1579301
created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1
/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:50 +0xae
goroutine 260 [IO wait, 28 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000ef7980 sp=0xc000ef7960 pc=0x1543096
runtime.netpollblock(0xc000ef79f0?, 0x150772f?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000ef79b8 sp=0xc000ef7980 pc=0x153b2b7
internal/poll.runtime_pollWait(0x7f6d5167c778, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000ef79d8 sp=0xc000ef79b8 pc=0x1572589
internal/poll.(*pollDesc).wait(0xc001209600?, 0xc000cc8260?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000ef7a00 sp=0xc000ef79d8 pc=0x15bb2d2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc001209600)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000ef7aa8 sp=0xc000ef7a00 pc=0x15c0d5d
net.(*netFD).accept(0xc001209600)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000ef7b60 sp=0xc000ef7aa8 pc=0x181c495
net.(*TCPListener).accept(0xc00050e468)
/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000ef7b88 sp=0xc000ef7b60 pc=0x183a8a5
net.(*TCPListener).Accept(0xc00050e468)
/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000ef7bb8 sp=0xc000ef7b88 pc=0x183983d
google.golang.org/grpc.(*Server).Serve(0xc000f7e3c0, {0x44b1450?, 0xc00050e468})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:824 +0x475 fp=0xc000ef7d00 sp=0xc000ef7bb8 pc=0x1c5be15
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop(0xc001054300, 0x5283)
/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:213 +0x9bf fp=0xc000ef7fc0 sp=0xc000ef7d00 pc=0x35caebf
github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init.func1()
/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:117 +0x2a fp=0xc000ef7fe0 sp=0xc000ef7fc0 pc=0x35ca1aa
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000ef7fe8 sp=0xc000ef7fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init
/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:117 +0x6c5
goroutine 160 [select]:
runtime.gopark(0xc000ec9f80?, 0x2?, 0x2?, 0x0?, 0xc000ec9f44?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000ec9dc8 sp=0xc000ec9da8 pc=0x1543096
runtime.selectgo(0xc000ec9f80, 0xc000ec9f40, 0xc003728408?, 0x0, 0xc0001e7f80?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000ec9f08 sp=0xc000ec9dc8 pc=0x1553bbe
github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc00026e140, {0x44b3c98, 0xc000ed6d70})
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:125 +0x145 fp=0xc000ec9fb8 sp=0xc000ec9f08 pc=0x28ca2e5
github.com/panjf2000/ants/v2.(*Pool).goTicktock.func1()
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:154 +0x2e fp=0xc000ec9fe0 sp=0xc000ec9fb8 pc=0x28ca6ee
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000ec9fe8 sp=0xc000ec9fe0 pc=0x1579301
created by github.com/panjf2000/ants/v2.(*Pool).goTicktock
/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/pool.go:154 +0x115
goroutine 253 [IO wait, 28 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d3b768 sp=0xc000d3b748 pc=0x1543096
runtime.netpollblock(0xc000d3b7d8?, 0x150772f?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000d3b7a0 sp=0xc000d3b768 pc=0x153b2b7
internal/poll.runtime_pollWait(0x7f6d5167ca48, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000d3b7c0 sp=0xc000d3b7a0 pc=0x1572589
internal/poll.(*pollDesc).wait(0xc00019ee80?, 0xc000510f40?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000d3b7e8 sp=0xc000d3b7c0 pc=0x15bb2d2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00019ee80)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000d3b890 sp=0xc000d3b7e8 pc=0x15c0d5d
net.(*netFD).accept(0xc00019ee80)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000d3b948 sp=0xc000d3b890 pc=0x181c495
net.(*TCPListener).accept(0xc000012a98)
/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000d3b970 sp=0xc000d3b948 pc=0x183a8a5
net.(*TCPListener).Accept(0xc000012a98)
/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000d3b9a0 sp=0xc000d3b970 pc=0x183983d
google.golang.org/grpc.(*Server).Serve(0xc000d5c000, {0x44b1450?, 0xc000012a98})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:824 +0x475 fp=0xc000d3bae8 sp=0xc000d3b9a0 pc=0x1c5be15
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startInternalGrpc(0xc0000012c0, 0x4c49, 0x0?)
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:344 +0xd59 fp=0xc000d3bfb8 sp=0xc000d3bae8 pc=0x34a8979
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startInternalRPCServer.func1()
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:185 +0x2e fp=0xc000d3bfe0 sp=0xc000d3bfb8 pc=0x34a5dee
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d3bfe8 sp=0xc000d3bfe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startInternalRPCServer
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:185 +0xa5
goroutine 254 [select, 28 minutes]:
runtime.gopark(0xc000d7d5b0?, 0x2?, 0x10?, 0x0?, 0xc000d7d59c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000d7d428 sp=0xc000d7d408 pc=0x1543096
runtime.selectgo(0xc000d7d5b0, 0xc000d7d598, 0x1b65a69?, 0x0, 0xc00021ef40?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000d7d568 sp=0xc000d7d428 pc=0x1553bbe
github.com/soheilhy/cmux.muxListener.Accept(...)
/go/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:262
github.com/soheilhy/cmux.(*muxListener).Accept(0x44e5ec0?)
<autogenerated>:1 +0x7e fp=0xc000d7d5e0 sp=0xc000d7d568 pc=0x20e5d5e
google.golang.org/grpc.(*Server).Serve(0xc000d5c1e0, {0x44b20e0?, 0xc000510f60})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:824 +0x475 fp=0xc000d7d728 sp=0xc000d7d5e0 pc=0x1c5be15
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc(0xc0000012c0, 0x4c4a, 0x0?)
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:280 +0x17d7 fp=0xc000d7dfb8 sp=0xc000d7d728 pc=0x34a7737
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalRPCServer.func1()
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:190 +0x2e fp=0xc000d7dfe0 sp=0xc000d7dfb8 pc=0x34a5f2e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000d7dfe8 sp=0xc000d7dfe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalRPCServer
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:190 +0xa5
goroutine 248 [select]:
runtime.gopark(0xc00045b128?, 0x2?, 0xf0?, 0xaf?, 0xc00045b11c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00045afa8 sp=0xc00045af88 pc=0x1543096
runtime.selectgo(0xc00045b128, 0xc00045b118, 0x0?, 0x0, 0x1c4c2b7?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00045b0e8 sp=0xc00045afa8 pc=0x1553bbe
google.golang.org/grpc.(*ClientConn).WaitForStateChange(0xc0021e8a80, {0x44b3d08, 0xc0025086c0}, 0x1)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:525 +0xb7 fp=0xc00045b158 sp=0xc00045b0e8 pc=0x1c4c597
google.golang.org/grpc.DialContext({0x44b3d08, 0xc0025086c0}, {0xc003edcca0, 0x10}, {0xc00045c048, 0xb, 0x7f6d239dab18?})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:331 +0x18e9 fp=0xc00045bc58 sp=0xc00045b158 pc=0x1c4aec9
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect(0x44ff360, {0x44b3c98, 0xc000d925a0})
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:241 +0x175e fp=0xc00045c108 sp=0xc00045bc58 pc=0x2e056fe
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient(0x44ff360, {0x44b3c98, 0xc000d925a0})
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:147 +0x165 fp=0xc00045c198 sp=0xc00045c108 pc=0x2e05d05
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce(0x44ff360, {0x44b3c98, 0xc000d925a0}, 0xc00206d6b0)
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:298 +0x1a9 fp=0xc00045c5b8 sp=0xc00045c198 pc=0x2e03449
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall(0x44ff360, {0x44b3c98, 0xc000d925a0}, 0xc000442001)
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:372 +0x96 fp=0xc00045c7f8 sp=0xc00045c5b8 pc=0x2e026b6
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall({0x44b3c98?, 0xc000d925a0?}, 0x3c173c0?)
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:367 +0x3c fp=0xc00045c830 sp=0xc00045c7f8 pc=0x2e0615c
github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]({0x44b3c98, 0xc000d925a0?}, 0xc001a6e960, 0xc002a7cbb8)
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:133 +0xc2 fp=0xc00045c868 sp=0xc00045c830 pc=0x2e08442
github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates(0xc00102e1a0?, {0x44b3c98?, 0xc000d925a0})
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:147 +0x85 fp=0xc00045c8b0 sp=0xc00045c868 pc=0x2df9025
github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1()
/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 +0x52 fp=0xc00045c938 sp=0xc00045c8b0 pc=0x2ea9dd2
github.com/milvus-io/milvus/pkg/util/retry.Do({0x44b3c98, 0xc000d925a0}, 0xc00045cb58, {0xc000d42b48, 0x2, 0x0?})
/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 +0x108 fp=0xc00045caf8 sp=0xc00045c938 pc=0x23c5188
github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates({0x44b3c98?, 0xc000d925a0?}, {0x7f6d239c52d8?, 0xc001a6e960?}, {0x3fdd3c8?, 0x1?}, {0xc000d42c54?, 0x18?, 0xc000b6ae60?}, 0xf4240, ...)
/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 +0x12f fp=0xc00045cbb8 sp=0xc00045caf8 pc=0x2ea9caf
github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy(...)
/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init(0xc0000012c0)
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531 +0x28e5 fp=0xc00045dd20 sp=0xc00045cbb8 pc=0x34abda5
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run(0xc0000012c0)
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355 +0x56 fp=0xc00045de88 sp=0xc00045dd20 pc=0x34a8e56
github.com/milvus-io/milvus/cmd/components.(*Proxy).Run(0xc000b60000?)
/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 +0x2e fp=0xc00045df58 sp=0xc00045de88 pc=0x375042e
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112 +0x159 fp=0xc00045dfe0 sp=0xc00045df58 pc=0x37551f9
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00045dfe8 sp=0xc00045dfe0 pc=0x1579301
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...]
/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:99 +0x15c
goroutine 249 [select]:
runtime.gopark(0xc000ef5f40?, 0x2?, 0x68?, 0xaa?, 0xc000ef5ed4?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000ef5d50 sp=0xc000ef5d30 pc=0x1543096
runtime.selectgo(0xc000ef5f40, 0xc000ef5ed0, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000ef5e90 sp=0xc000ef5d50 pc=0x1553bbe
github.com/milvus-io/milvus/internal/proxy.(*LookAsideBalancer).checkQueryNodeHealthLoop(0xc0001e73e0, {0x44b3c98?, 0xc000d925a0?})
/go/src/github.com/milvus-io/milvus/internal/proxy/look_aside_balancer.go:202 +0x2aa fp=0xc000ef5fb8 sp=0xc000ef5e90 pc=0x33bc82a
github.com/milvus-io/milvus/internal/proxy.(*LookAsideBalancer).Start.func1()
/go/src/github.com/milvus-io/milvus/internal/proxy/look_aside_balancer.go:77 +0x2e fp=0xc000ef5fe0 sp=0xc000ef5fb8 pc=0x33bb7ee
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000ef5fe8 sp=0xc000ef5fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/proxy.(*LookAsideBalancer).Start
/go/src/github.com/milvus-io/milvus/internal/proxy/look_aside_balancer.go:77 +0xaa
goroutine 250 [select, 28 minutes]:
runtime.gopark(0xc0017d3f98?, 0x2?, 0x2?, 0x0?, 0xc0017d3f7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001d50e00 sp=0xc001d50de0 pc=0x1543096
runtime.selectgo(0xc001d50f98, 0xc0017d3f78, 0xc000012a80?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001d50f40 sp=0xc001d50e00 pc=0x1553bbe
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0004b0500)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:115 +0x73 fp=0xc001d50fc8 sp=0xc001d50f40 pc=0x1c46bd3
google.golang.org/grpc.newCCBalancerWrapper.func1()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x26 fp=0xc001d50fe0 sp=0xc001d50fc8 pc=0x1c46b26
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001d50fe8 sp=0xc001d50fe0 pc=0x1579301
created by google.golang.org/grpc.newCCBalancerWrapper
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x22a
goroutine 205 [IO wait, 28 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000ef9b18 sp=0xc000ef9af8 pc=0x1543096
runtime.netpollblock(0x7f6d8e0759a8?, 0x150772f?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000ef9b50 sp=0xc000ef9b18 pc=0x153b2b7
internal/poll.runtime_pollWait(0x7f6d5167cd18, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000ef9b70 sp=0xc000ef9b50 pc=0x1572589
internal/poll.(*pollDesc).wait(0xc000f73400?, 0xc000ef9c58?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000ef9b98 sp=0xc000ef9b70 pc=0x15bb2d2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000f73400)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000ef9c40 sp=0xc000ef9b98 pc=0x15c0d5d
net.(*netFD).accept(0xc000f73400)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000ef9cf8 sp=0xc000ef9c40 pc=0x181c495
net.(*TCPListener).accept(0xc000186300)
/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000ef9d20 sp=0xc000ef9cf8 pc=0x183a8a5
net.(*TCPListener).Accept(0xc000186300)
/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000ef9d50 sp=0xc000ef9d20 pc=0x183983d
net/http.(*onceCloseListener).Accept(0x44b3cd0?)
<autogenerated>:1 +0x2a fp=0xc000ef9d68 sp=0xc000ef9d50 pc=0x19776aa
net/http.(*Server).Serve(0xc00054c5a0, {0x44b1450, 0xc000186300})
/usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000ef9e98 sp=0xc000ef9d68 pc=0x194bf25
net/http.(*Server).ListenAndServe(0xc00054c5a0)
/usr/local/go/src/net/http/server.go:2988 +0x7d fp=0xc000ef9ec8 sp=0xc000ef9e98 pc=0x194ba9d
github.com/milvus-io/milvus/internal/http.ServeHTTP.func1()
/go/src/github.com/milvus-io/milvus/internal/http/server.go:79 +0x15a fp=0xc000ef9fe0 sp=0xc000ef9ec8 pc=0x30f6dfa
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000ef9fe8 sp=0xc000ef9fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/http.ServeHTTP
/go/src/github.com/milvus-io/milvus/internal/http/server.go:75 +0x25
goroutine 40450 [IO wait]:
runtime.gopark(0x38?, 0x3dab020?, 0x60?, 0x0?, 0x7f6d239c89e0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00436c5a0 sp=0xc00436c580 pc=0x1543096
runtime.netpollblock(0x157712e?, 0x154b651?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc00436c5d8 sp=0xc00436c5a0 pc=0x153b2b7
internal/poll.runtime_pollWait(0x7f6d239c7028, 0x77)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc00436c5f8 sp=0xc00436c5d8 pc=0x1572589
internal/poll.(*pollDesc).wait(0xc000dd3900?, 0x0?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00436c620 sp=0xc00436c5f8 pc=0x15bb2d2
internal/poll.(*pollDesc).waitWrite(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:93
internal/poll.(*FD).WaitWrite(...)
/usr/local/go/src/internal/poll/fd_unix.go:741
net.(*netFD).connect(0xc000dd3900, {0x44b3d40?, 0xc002508c30}, {0x7f6d239c7028?, 0xc00436c800?}, {0x4497ce0?, 0xc00250a100?})
/usr/local/go/src/net/fd_unix.go:141 +0x799 fp=0xc00436c7b8 sp=0xc00436c620 pc=0x181bd39
net.(*netFD).dial(0xc000dd3900, {0x44b3d40, 0xc002508c30}, {0x44bfbb0?, 0x0?}, {0x44bfbb0?, 0xc002508c90}, 0xc00436c900?)
/usr/local/go/src/net/sock_posix.go:151 +0x3c5 fp=0xc00436c888 sp=0xc00436c7b8 pc=0x1836945
net.socket({0x44b3d40, 0xc002508c30}, {0x3fcd0eb, 0x3}, 0x2, 0x1, 0x151b650?, 0x0?, {0x44bfbb0, 0x0}, ...)
/usr/local/go/src/net/sock_posix.go:70 +0x2d2 fp=0xc00436c938 sp=0xc00436c888 pc=0x18362d2
net.internetSocket({0x44b3d40, 0xc002508c30}, {0x3fcd0eb, 0x3}, {0x44bfbb0, 0x0}, {0x44bfbb0, 0xc002508c90}, 0xc00436ca80?, 0x0, ...)
/usr/local/go/src/net/ipsock_posix.go:142 +0xf8 fp=0xc00436c9b8 sp=0xc00436c938 pc=0x1828838
net.(*sysDialer).doDialTCP(0xc00250c000, {0x44b3d40, 0xc002508c30}, 0x0, 0x5d3dae0?)
/usr/local/go/src/net/tcpsock_posix.go:74 +0xef fp=0xc00436ca70 sp=0xc00436c9b8 pc=0x183a3ef
net.(*sysDialer).dialTCP(0x1744546?, {0x44b3d40?, 0xc002508c30?}, 0x17444c5?, 0xc004083cd8?)
/usr/local/go/src/net/tcpsock_posix.go:64 +0x69 fp=0xc00436cab0 sp=0xc00436ca70 pc=0x183a289
net.(*sysDialer).dialSingle(0xc00250c000, {0x44b3d40, 0xc002508c30}, {0x44a1088?, 0xc002508c90})
/usr/local/go/src/net/dial.go:580 +0x1f2 fp=0xc00436cb80 sp=0xc00436cab0 pc=0x18100b2
net.(*sysDialer).dialSerial(0xc00250c000, {0x44b3d40, 0xc002508c30}, {0xc001d5a8e0?, 0x1, 0x151b650?})
/usr/local/go/src/net/dial.go:548 +0x245 fp=0xc00436cc88 sp=0xc00436cb80 pc=0x180fa05
net.(*sysDialer).dialParallel(0xc001d5a8d0?, {0x44b3d40?, 0xc002508c30?}, {0xc001d5a8e0?, 0xc002508c30?, 0x3fce264?}, {0x0?, 0x3fcd0eb?, 0x722f646574756216?})
/usr/local/go/src/net/dial.go:449 +0x413 fp=0xc00436cee0 sp=0xc00436cc88 pc=0x180f073
net.(*Dialer).DialContext(0xc00436d068, {0x44b3d40, 0xc002508c30}, {0x3fcd0eb, 0x3}, {0xc00250a08f, 0x10})
/usr/local/go/src/net/dial.go:440 +0x70c fp=0xc00436d018 sp=0xc00436cee0 pc=0x180ea8c
google.golang.org/grpc/internal/transport.proxyDial({0x44b3d40, 0xc002508c30}, {0xc00250a08f, 0x10}, {0x3fea31b, 0xe})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/proxy.go:125 +0xe5 fp=0xc00436d0e0 sp=0xc00436d018 pc=0x1be3a85
google.golang.org/grpc/internal/transport.dial({0x44b3d40, 0xc002508c30}, 0x0, {{0xc00250a08f, 0x10}, {0xc00250a08f, 0x10}, 0x0, 0x0, 0x0, ...}, ...)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/http2_client.go:175 +0x296 fp=0xc00436d1e8 sp=0xc00436d0e0 pc=0x1bc7616
google.golang.org/grpc/internal/transport.newHTTP2Client({_, _}, {_, _}, {{0xc00250a08f, 0x10}, {0xc00250a08f, 0x10}, 0x0, 0x0, ...}, ...)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/http2_client.go:214 +0x194 fp=0xc00436d8c0 sp=0xc00436d1e8 pc=0x1bc7994
google.golang.org/grpc/internal/transport.NewClientTransport(...)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/transport.go:606
google.golang.org/grpc.(*addrConn).createTransport(_, {{0xc00250a08f, 0x10}, {0xc00250a08f, 0x10}, 0x0, 0x0, 0x0, {0x0, 0x0}}, ...)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:1285 +0x305 fp=0xc00436dc00 sp=0xc00436d8c0 pc=0x1c50d45
google.golang.org/grpc.(*addrConn).tryAllAddrs(0xc0021f6840, {0xc004083b80?, 0x1, 0x0?}, {0xc002508ba0?, 0xc003408ee0?, 0x5cf3f40?})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:1233 +0x4c5 fp=0xc00436de80 sp=0xc00436dc00 pc=0x1c50965
google.golang.org/grpc.(*addrConn).resetTransport(0xc0021f6840)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:1168 +0x1c5 fp=0xc00436df78 sp=0xc00436de80 pc=0x1c50105
google.golang.org/grpc.(*addrConn).connect(0xc0021f6840)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:819 +0x185 fp=0xc00436dfc8 sp=0xc00436df78 pc=0x1c4e365
google.golang.org/grpc.(*acBalancerWrapper).Connect.func2()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:413 +0x26 fp=0xc00436dfe0 sp=0xc00436dfc8 pc=0x1c48746
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00436dfe8 sp=0xc00436dfe0 pc=0x1579301
created by google.golang.org/grpc.(*acBalancerWrapper).Connect
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:413 +0xbc
goroutine 386 [select, 28 minutes]:
runtime.gopark(0xc000d7bf98?, 0x2?, 0x2?, 0x0?, 0xc000d7bf7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001e20e00 sp=0xc001e20de0 pc=0x1543096
runtime.selectgo(0xc001e20f98, 0xc000d7bf78, 0xc000278330?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001e20f40 sp=0xc001e20e00 pc=0x1553bbe
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000554d40)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:115 +0x73 fp=0xc001e20fc8 sp=0xc001e20f40 pc=0x1c46bd3
google.golang.org/grpc.newCCBalancerWrapper.func1()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x26 fp=0xc001e20fe0 sp=0xc001e20fc8 pc=0x1c46b26
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001e20fe8 sp=0xc001e20fe0 pc=0x1579301
created by google.golang.org/grpc.newCCBalancerWrapper
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x22a
goroutine 938 [select]:
runtime.gopark(0xc001096f38?, 0x2?, 0x5a?, 0xcd?, 0xc001096f1c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001096d80 sp=0xc001096d60 pc=0x1543096
runtime.selectgo(0xc001096f38, 0xc001096f18, 0x5cf3f40?, 0x0, 0x1913b323aab?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc001096ec0 sp=0xc001096d80 pc=0x1553bbe
go.etcd.io/etcd/client/v3.(*lessor).deadlineLoop(0xc0004ecf00)
/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:548 +0x85 fp=0xc001096fc8 sp=0xc001096ec0 pc=0x1d1d405
go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1.2()
/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:300 +0x26 fp=0xc001096fe0 sp=0xc001096fc8 pc=0x1d1b146
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001096fe8 sp=0xc001096fe0 pc=0x1579301
created by go.etcd.io/etcd/client/v3.(*lessor).KeepAlive.func1
/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:300 +0xa5
goroutine 1418 [select, 27 minutes]:
runtime.gopark(0xc0010bbfb0?, 0x2?, 0x0?, 0x0?, 0xc0010bbf94?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0010bbe20 sp=0xc0010bbe00 pc=0x1543096
runtime.selectgo(0xc0010bbfb0, 0xc0010bbf90, 0x1c4e365?, 0x0, 0x1?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0010bbf60 sp=0xc0010bbe20 pc=0x1553bbe
github.com/milvus-io/milvus/pkg/util/timerecord.(*LongTermChecker).Start.func1()
/go/src/github.com/milvus-io/milvus/pkg/util/timerecord/time_recorder.go:129 +0x85 fp=0xc0010bbfe0 sp=0xc0010bbf60 pc=0x2161de5
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0010bbfe8 sp=0xc0010bbfe0 pc=0x1579301
created by github.com/milvus-io/milvus/pkg/util/timerecord.(*LongTermChecker).Start
/go/src/github.com/milvus-io/milvus/pkg/util/timerecord/time_recorder.go:127 +0x8a
goroutine 207 [select, 28 minutes]:
runtime.gopark(0xc000f83f98?, 0x2?, 0x2?, 0x0?, 0xc000f83f7c?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00155ee00 sp=0xc00155ede0 pc=0x1543096
runtime.selectgo(0xc00155ef98, 0xc000f83f78, 0xc00050e2e8?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00155ef40 sp=0xc00155ee00 pc=0x1553bbe
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0005d1c80)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:115 +0x73 fp=0xc00155efc8 sp=0xc00155ef40 pc=0x1c46bd3
google.golang.org/grpc.newCCBalancerWrapper.func1()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x26 fp=0xc00155efe0 sp=0xc00155efc8 pc=0x1c46b26
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00155efe8 sp=0xc00155efe0 pc=0x1579301
created by google.golang.org/grpc.newCCBalancerWrapper
/go/pkg/mod/google.golang.org/grpc@v1.54.0/balancer_conn_wrappers.go:76 +0x22a
goroutine 258 [IO wait, 28 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0011c59c0 sp=0xc0011c59a0 pc=0x1543096
runtime.netpollblock(0xc0011c5a30?, 0x150772f?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0011c59f8 sp=0xc0011c59c0 pc=0x153b2b7
internal/poll.runtime_pollWait(0x7f6d5167ce08, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0011c5a18 sp=0xc0011c59f8 pc=0x1572589
internal/poll.(*pollDesc).wait(0xc001209500?, 0xc0004affe0?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0011c5a40 sp=0xc0011c5a18 pc=0x15bb2d2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc001209500)
/usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc0011c5ae8 sp=0xc0011c5a40 pc=0x15c0d5d
net.(*netFD).accept(0xc001209500)
/usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0011c5ba0 sp=0xc0011c5ae8 pc=0x181c495
net.(*TCPListener).accept(0xc00050e318)
/usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0011c5bc8 sp=0xc0011c5ba0 pc=0x183a8a5
net.(*TCPListener).Accept(0xc00050e318)
/usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc0011c5bf8 sp=0xc0011c5bc8 pc=0x183983d
google.golang.org/grpc.(*Server).Serve(0xc000f7e1e0, {0x44b1450?, 0xc00050e318})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:824 +0x475 fp=0xc0011c5d40 sp=0xc0011c5bf8 pc=0x1c5be15
github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).startGrpcLoop(0xc000054990, 0xcf6c)
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:282 +0x9ab fp=0xc0011c5fc0 sp=0xc0011c5d40 pc=0x374d46b
github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).startGrpc.func1()
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:222 +0x2a fp=0xc0011c5fe0 sp=0xc0011c5fc0 pc=0x374ca8a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0011c5fe8 sp=0xc0011c5fe0 pc=0x1579301
created by github.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).startGrpc
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:222 +0x7f
goroutine 813 [IO wait]:
runtime.gopark(0xc001bddd98?, 0xb?, 0x0?, 0x0?, 0x21?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000e03ad8 sp=0xc000e03ab8 pc=0x1543096
runtime.netpollblock(0x1592c25?, 0x150772f?, 0x0?)
/usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000e03b10 sp=0xc000e03ad8 pc=0x153b2b7
internal/poll.runtime_pollWait(0x7f6d5167c2c8, 0x72)
/usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000e03b30 sp=0xc000e03b10 pc=0x1572589
internal/poll.(*pollDesc).wait(0xc001a70800?, 0xc001d04000?, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000e03b58 sp=0xc000e03b30 pc=0x15bb2d2
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001a70800, {0xc001d04000, 0x8000, 0x8000})
/usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000e03bf0 sp=0xc000e03b58 pc=0x15bc839
net.(*netFD).Read(0xc001a70800, {0xc001d04000?, 0x1060100000000?, 0x8?})
/usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000e03c38 sp=0xc000e03bf0 pc=0x181a2a9
net.(*conn).Read(0xc0010b1790, {0xc001d04000?, 0xc002086928?, 0xc000f74a00?})
/usr/local/go/src/net/net.go:183 +0x45 fp=0xc000e03c80 sp=0xc000e03c38 pc=0x182f285
net.(*TCPConn).Read(0x801010601?, {0xc001d04000?, 0xc000e03d28?, 0x801010601?})
<autogenerated>:1 +0x29 fp=0xc000e03cb0 sp=0xc000e03c80 pc=0x1845de9
bufio.(*Reader).Read(0xc001a4ad20, {0xc000eefc40, 0x9, 0x7f6d8e06a108?})
/usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000e03ce8 sp=0xc000e03cb0 pc=0x16a2bbb
io.ReadAtLeast({0x4491c60, 0xc001a4ad20}, {0xc000eefc40, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:332 +0x9a fp=0xc000e03d30 sp=0xc000e03ce8 pc=0x15b3f3a
io.ReadFull(...)
/usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc000eefc40?, 0x9?, 0x1915f1a62cb?}, {0x4491c60?, 0xc001a4ad20?})
/go/pkg/mod/golang.org/x/net@v0.10.0/http2/frame.go:237 +0x6e fp=0xc000e03d80 sp=0xc000e03d30 pc=0x1b9740e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000eefc00)
/go/pkg/mod/golang.org/x/net@v0.10.0/http2/frame.go:498 +0x95 fp=0xc000e03e30 sp=0xc000e03d80 pc=0x1b981b5
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001a41b00, 0xc001a40d80?)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/http2_client.go:1592 +0x257 fp=0xc000e03fc0 sp=0xc000e03e30 pc=0x1bd4777
google.golang.org/grpc/internal/transport.newHTTP2Client.func11()
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/http2_client.go:394 +0x2a fp=0xc000e03fe0 sp=0xc000e03fc0 pc=0x1bca02a
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000e03fe8 sp=0xc000e03fe0 pc=0x1579301
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/go/pkg/mod/google.golang.org/grpc@v1.54.0/internal/transport/http2_client.go:394 +0x1e2a
goroutine 317 [select]:
runtime.gopark(0xc0002b9dd0?, 0x2?, 0x98?, 0x9c?, 0xc0002b9dc4?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0002b9c50 sp=0xc0002b9c30 pc=0x1543096
runtime.selectgo(0xc0002b9dd0, 0xc0002b9dc0, 0x0?, 0x0, 0x1c4c2b7?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0002b9d90 sp=0xc0002b9c50 pc=0x1553bbe
google.golang.org/grpc.(*ClientConn).WaitForStateChange(0xc0020dea80, {0x44b3d08, 0xc0020fe780}, 0x1)
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:525 +0xb7 fp=0xc0002b9e00 sp=0xc0002b9d90 pc=0x1c4c597
google.golang.org/grpc.DialContext({0x44b3d08, 0xc0020fe780}, {0xc003f48a30, 0x10}, {0xc0002bacf0, 0xb, 0xc0002ba968?})
/go/pkg/mod/google.golang.org/grpc@v1.54.0/clientconn.go:331 +0x18e9 fp=0xc0002ba900 sp=0xc0002b9e00 pc=0x1c4aec9
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect(0x44ff360, {0x44b3c98, 0xc000ed7860})
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:241 +0x175e fp=0xc0002badb0 sp=0xc0002ba900 pc=0x2e056fe
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient(0x44ff360, {0x44b3c98, 0xc000ed7860})
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:147 +0x165 fp=0xc0002bae40 sp=0xc0002badb0 pc=0x2e05d05
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).callOnce(0x44ff360, {0x44b3c98, 0xc000ed7860}, 0xc0020fe330)
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:298 +0x1a9 fp=0xc0002bb260 sp=0xc0002bae40 pc=0x2e03449
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall(0x44ff360, {0x44b3c98, 0xc000ed7860}, 0xc00007cc01)
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:372 +0x96 fp=0xc0002bb4a0 sp=0xc0002bb260 pc=0x2e026b6
github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall({0x44b3c98?, 0xc000ed7860?}, 0x3c173c0?)
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:367 +0x3c fp=0xc0002bb4d8 sp=0xc0002bb4a0 pc=0x2e0615c
github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]({0x44b3c98, 0xc000ed7860?}, 0xc001b3ac48, 0xc0037287f8)
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:133 +0xc2 fp=0xc0002bb510 sp=0xc0002bb4d8 pc=0x2e08442
github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates(0xc000468340?, {0x44b3c98?, 0xc000ed7860})
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:147 +0x85 fp=0xc0002bb558 sp=0xc0002bb510 pc=0x2df9025
github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1()
/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 +0x52 fp=0xc0002bb5e0 sp=0xc0002bb558 pc=0x2ea9dd2
github.com/milvus-io/milvus/pkg/util/retry.Do({0x44b3c98, 0xc000ed7860}, 0xc0002bb800, {0xc000d3f7f0, 0x2, 0x0?})
/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 +0x108 fp=0xc0002bb7a0 sp=0xc0002bb5e0 pc=0x23c5188
github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates({0x44b3c98?, 0xc000ed7860?}, {0x7f6d239c52d8?, 0xc001b3ac48?}, {0x3fdd3c8?, 0x1?}, {0xc000d3f8c4?, 0x18?, 0xc000b6ae60?}, 0xf4240, ...)
/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 +0x12f fp=0xc0002bb860 sp=0xc0002bb7a0 pc=0x2ea9caf
github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy(...)
/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73
github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init(0xc000de2480)
/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156 +0x6f8 fp=0xc0002bbe48 sp=0xc0002bb860 pc=0x354a298
github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run(0xc000344120?)
/go/src/github.com/milvus-io/milvus/iI tested version 2.3.1 and it works fine. Does anybody still have this issue?
ieugen
commented
1 year ago @Nik96i : Can you share your deployment template ? can you share what IP addresses you have in the container? (Instruction above) Are you using ipv4 / ipv6 / both?
Nik96i
commented
1 year ago Hi @ieugen . Sure:
First create a swarm overlay network:
sudo docker network create --scope=swarm --attachable --opt encrypted=true --driver=overlay mil-netversion: "3.9"
services:
milvus-etcd:
image: quay.io/coreos/etcd:v3.5.5
networks:
- mil-net
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ./vols/milvus/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3
deploy:
restart_policy:
condition: any
placement:
constraints:
- "node.role==manager"
milvus-minio:
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
networks:
- mil-net
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
volumes:
- ./vols/milvus/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
deploy:
restart_policy:
condition: any
placement:
constraints:
- "node.role==manager"
milvus-standalone:
image: milvusdb/milvus:v2.3.1
networks:
- mil-net
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: milvus-etcd:2379
MINIO_ADDRESS: milvus-minio:9000
volumes:
- ./vols/milvus/standalone:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "milvus-etcd"
- "milvus-minio"
deploy:
restart_policy:
condition: any
placement:
constraints:
- "node.role==manager"
networks:
mil-net:
external: trueIP Addr:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
4152: eth0@if4153: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1424 qdisc noqueue state UP group default
link/ether 02:42:0a:00:01:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.0.1.197/24 brd 10.0.1.255 scope global eth0
valid_lft forever preferred_lft forever
4154: eth1@if4155: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:06 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet 172.18.0.6/16 brd 172.18.255.255 scope global eth1
valid_lft forever preferred_lft foreverNo. I don't use IPv6, I guess. ?
ieugen
commented
1 year ago Perhaps this is the cause ? milvus does not work well with dual networking?
That would be curios since binding to tcp6 will bind to tcp4 address as well.
Not sure I can disable ipv6 on that server. I will see if I can find a solution to test this out so we can confirm. What I do know is that milvus codebase does explicitly use an IPV4 .
I do think the code that selects the IP address is not the best:
The code seems to take into account only ipv4 addresses (See to4 bellow and https://pkg.go.dev/net#IP.To4 ). Which is not a big deal but it also takes the first IP, without any option to filter it by CIDR or device name.
func GetIP(ip string) string {
if len(ip) == 0 {
return GetLocalIP()
}
return ip
}
// GetLocalIP return the local ip address
func GetLocalIP() string {
addrs, err := net.InterfaceAddrs()
if err == nil {
for _, addr := range addrs {
ipaddr, ok := addr.(*net.IPNet)
if ok && ipaddr.IP.IsGlobalUnicast() && ipaddr.IP.To4() != nil {
return ipaddr.IP.String()
}
}
}
return "127.0.0.1"
}@ieugen I don't understand. Milvus is in a swarm service, completely separate from the server. why does it matter if you have IP6 or not? Don't expose Milvus ports (do not bind at all) and use a reverse proxy like Traefik or Nginx and expose necessary ports. Does it work?
ieugen
commented
1 year ago ~I do hava ipv6 inside docker container as well.~
It seems that milvus binds to ipv6 stack inside container but I don't have any ipv6 addresses inside container.
I will try to enable ipv6tables option to see if it makes a difference: https://docs.docker.com/config/daemon/ipv6/
I might also try to disable ipv6 inside container, maybe this will fix milvus.
@Nik96i The issue is that milvus is a unch of servers that need to talk to each other and for some reason they fail to do that in docker swarm networks, in my case. I don't have issues with other services (except ElasticSearch which has it's discovery protocol - but even that works with tweaks). For the record, I do manage ~ 5 swarm cluster across 140+ servers with a lot of running containers and have been doing for several years now. Milvus is the only service we have that does not work as it should.
ieugen
commented
12 months ago I did make some progress on this issue and I would like to confirm it.
I added healthchecks which I was missing. This makes the service run eventually after a random number of restarts. I have tweaked the interval and retries to be as low as possible (I think).
@Nik96i (or anyone else) : In your swarm deployment, does milvus start right away or it fails for a few number of times and then it starts?
In my experience, milvus restarts a couple of times until it gets the IP address that works:
docker service ps REDACTED_standalone
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
hsjiudmhftzp REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Running Running about a minute ago
yez7irhok1vt \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Failed 2 minutes ago "task: non-zero exit (137): do…"
v3p2cagmte71 \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Failed 4 minutes ago "task: non-zero exit (137): do…"
dp7bo1cdfv7a \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Shutdown 7 minutes ago
vdvtgnl1yqnm \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Failed 25 minutes ago "task: non-zero exit (137): do…" The tasks are ~2 minutes apart - the exact time it takes to go through all healthchecks and re-create the container. Eventually this leads to a running container, but it is very unreliable IMO and not safe to use in any serious capacity.
EeyoreLee
commented
11 months ago I did make some progress on this issue and I would like to confirm it.
I added healthchecks which I was missing. This makes the service run eventually after a random number of restarts. I have tweaked the interval and retries to be as low as possible (I think).
@Nik96i (or anyone else) : In your swarm deployment, does milvus start right away or it fails for a few number of times and then it starts?
In my experience, milvus restarts a couple of times until it gets the IP address that works:
docker service ps REDACTED_standalone ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS hsjiudmhftzp REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Running Running about a minute ago yez7irhok1vt \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Failed 2 minutes ago "task: non-zero exit (137): do…" v3p2cagmte71 \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Failed 4 minutes ago "task: non-zero exit (137): do…" dp7bo1cdfv7a \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Shutdown 7 minutes ago vdvtgnl1yqnm \_ REDACTED_standalone.1 milvusdb/milvus:v2.3.1 prod2 Shutdown Failed 25 minutes ago "task: non-zero exit (137): do…"The tasks are ~2 minutes apart - the exact time it takes to go through all healthchecks and re-create the container. Eventually this leads to a running container, but it is very unreliable IMO and not safe to use in any serious capacity.
Aha, so the reason why I fail to run milvus on swarm mode is only I limited the retry time to 5 ?
stale[bot]
commented
10 months ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
ieugen
commented
7 months ago This is not done. Milvus can't handle IPV6 and that is what is causing the issues. If it has an IPV6 interface ... well good luck with that.
Is there an existing issue for this?
Environment
Current Behavior
When I run this docker-compose with
docker compose up -dit works fine. but when I run it withdocker stack deploy -c docker-compose.yml milvusit's not working. first it gives "Proxy not healthy" then after about 10-15 minutes it works fine.` version: "3.9"
services:
etcd: image: quay.io/coreos/etcd:v3.5.5 networks:
type: bind source: /home/frs/milvus/etcd target: /etcd command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd deploy: placement: constraints:
minio: image: minio/minio:RELEASE.2023-03-20T20-16-18Z networks:
type: bind source: /home/frs/milvus/minio target: /minio_data environment: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin command: minio server /minio_data healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 deploy: placement: constraints:
standalone: image: milvusdb/milvus:v2.2.10 networks:
networks: frs-public: `
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
Complete log: https://file.io/vxEe999zcntN
[2023/06/20 09:10:12.537 +00:00] [WARN] [rootcoord/proxy_client_manager.go:239] ["proxy client is empty, GetMetrics will not send to any client"] [2023/06/20 09:10:12.538 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0] [2023/06/20 09:10:12.538 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:12.538 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [2023/06/20 09:10:12.538 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord] [2023/06/20 09:10:12.538 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:12.539 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [2023/06/20 09:10:12.539 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0] [2023/06/20 09:10:12.539 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:12.540 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord] [2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:12.540 +00:00] [WARN] [rootcoord/quota_center.go:143] ["quotaCenter sync metrics failed"] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [2023/06/20 09:10:13.083 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.084 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:13.093 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.093 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:13.284 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #753 SubChannel #754] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:13.284 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148 github.com/milvus-io/milvus/internal/distributed/indexcoord.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81 github.com/milvus-io/milvus/internal/distributed/indexcoord.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(IndexCoord).Run\n"] [2023/06/20 09:10:13.285 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.285 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:13.294 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974 github.com/milvus-io/milvus/internal/datacoord.(Server).hasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361 github.com/milvus-io/milvus/internal/datacoord.(ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360 github.com/milvus-io/milvus/internal/datacoord.(ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396 github.com/milvus-io/milvus/internal/datacoord.(ServerHandler).CheckShouldDropChannel\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246 github.com/milvus-io/milvus/internal/datacoord.(ChannelManager).unwatchDroppedChannels\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163 github.com/milvus-io/milvus/internal/datacoord.(ChannelManager).Startup\n"] [2023/06/20 09:10:13.294 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #755 SubChannel #756] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled\""] [2023/06/20 09:10:13.295 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.295 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:13.486 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #757 SubChannel #758] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:13.486 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148 github.com/milvus-io/milvus/internal/distributed/indexcoord.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81 github.com/milvus-io/milvus/internal/distributed/indexcoord.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(IndexCoord).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/indexcoord.(Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148\ngithub.com/milvus-io/milvus/internal/distributed/indexcoord.(Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81\ngithub.com/milvus-io/milvus/cmd/components.(IndexCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"] [2023/06/20 09:10:13.496 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #759 SubChannel #760] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:13.496 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974 github.com/milvus-io/milvus/internal/datacoord.(Server).hasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361 github.com/milvus-io/milvus/internal/datacoord.(ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360 github.com/milvus-io/milvus/internal/datacoord.(ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396 github.com/milvus-io/milvus/internal/datacoord.(ServerHandler).CheckShouldDropChannel\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246 github.com/milvus-io/milvus/internal/datacoord.(ChannelManager).unwatchDroppedChannels\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163 github.com/milvus-io/milvus/internal/datacoord.(ChannelManager).Startup\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234\ngithub.com/milvus-io/milvus/internal/datacoord.(Server).hasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974\ngithub.com/milvus-io/milvus/internal/datacoord.(ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/datacoord.(ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360\ngithub.com/milvus-io/milvus/internal/datacoord.(ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396\ngithub.com/milvus-io/milvus/internal/datacoord.(ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246\ngithub.com/milvus-io/milvus/internal/datacoord.(ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163\ngithub.com/milvus-io/milvus/internal/datacoord.(Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:441\ngithub.com/milvus-io/milvus/internal/datacoord.(Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:310\ngithub.com/milvus-io/milvus/internal/datacoord.(Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:283\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:241\ngithub.com/milvus-io/milvus/cmd/components.(DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"] [2023/06/20 09:10:14.070 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.070 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:14.164 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.164 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:14.172 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.172 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:14.271 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #761 SubChannel #762] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:14.271 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387 github.com/milvus-io/milvus/internal/distributed/proxy.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295 github.com/milvus-io/milvus/internal/distributed/proxy.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 github.com/milvus-io/milvus/cmd/components.(Proxy).Run\n"] [2023/06/20 09:10:14.272 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.272 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:14.365 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #763 SubChannel #764] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:14.366 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164 github.com/milvus-io/milvus/internal/distributed/querycoord.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98 github.com/milvus-io/milvus/internal/distributed/querycoord.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(QueryCoord).Run\n"] [2023/06/20 09:10:14.367 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.367 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:14.372 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275 github.com/milvus-io/milvus/internal/distributed/datanode.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172 github.com/milvus-io/milvus/internal/distributed/datanode.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52 github.com/milvus-io/milvus/cmd/components.(DataNode).Run\n"] [2023/06/20 09:10:14.372 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #765 SubChannel #766] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled\""] [2023/06/20 09:10:14.373 +00:00] [INFO] [sessionutil/session_util.go:477] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.373 +00:00] [INFO] [client/client.go:109] ["RootCoordClient GetSessions success"] [address=10.0.0.61:53100] [2023/06/20 09:10:14.472 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #767 SubChannel #768] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled\""] [2023/06/20 09:10:14.472 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387 github.com/milvus-io/milvus/internal/distributed/proxy.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295 github.com/milvus-io/milvus/internal/distributed/proxy.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 github.com/milvus-io/milvus/cmd/components.(Proxy).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295\ngithub.com/milvus-io/milvus/cmd/components.(Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"] [2023/06/20 09:10:14.568 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #769 SubChannel #770] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:14.568 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164 github.com/milvus-io/milvus/internal/distributed/querycoord.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98 github.com/milvus-io/milvus/internal/distributed/querycoord.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(QueryCoord).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98\ngithub.com/milvus-io/milvus/cmd/components.(QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"] [2023/06/20 09:10:14.574 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #771 SubChannel #772] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"10.0.0.61:53100\",\n \"ServerName\": \"10.0.0.61:53100\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout\""] [2023/06/20 09:10:14.574 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=rootcoord] [error="err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275 github.com/milvus-io/milvus/internal/distributed/datanode.(Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172 github.com/milvus-io/milvus/internal/distributed/datanode.(Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52 github.com/milvus-io/milvus/cmd/components.(DataNode).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172\ngithub.com/milvus-io/milvus/cmd/components.(DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120"] [2023/06/20 09:10:15.537 +00:00] [WARN] [rootcoord/proxy_client_manager.go:239] ["proxy client is empty, GetMetrics will not send to any client"] [2023/06/20 09:10:15.539 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord] [2023/06/20 09:10:15.539 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0] [2023/06/20 09:10:15.539 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:15.539 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:15.539 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [2023/06/20 09:10:15.539 +00:00] [WARN] [grpcclient/client.go:318] ["ClientBase ReCall grpc first call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [2023/06/20 09:10:15.540 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0] [2023/06/20 09:10:15.540 +00:00] [WARN] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available datacoord, check datacoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:158] ["failed to get client address"] [error="find no available querycoord, check querycoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:330] ["ClientBase ReCall grpc second call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"] [2023/06/20 09:10:15.540 +00:00] [WARN] [rootcoord/quota_center.go:143] ["quotaCenter sync metrics failed"] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
Anything else?
No response