search

langgenius / dify

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.
https://dify.ai
Other
53.13k stars 7.75k forks source link

[Milvus Bug]: find no available rootcoord, check rootcoord state + Your previous request to create the named bucket succeeded and you already own it. + there is no value on key = by-dev/meta/root-coord/credential/grantee-privileges/public/Global/*.* find no available querycoord, check querycoord state #11018

Open taowang1993 opened 5 days ago

taowang1993 commented 5 days ago

Self Checks

Dify version

0.11.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When I run docker compose up -d, I ran into the following errors:

error="find no available rootcoord, check rootcoord state
error="Your previous request to create the named bucket succeeded and you already own it.
error="there is no value on key = by-dev/meta/root-coord/credential/grantee-privileges/public/Global/.
error="find no available querycoord, check querycoord state

Steps to Reproduce

git clone https://github.com/langgenius/dify.git cd dify/docker cp .env.example .env set VECTOR_STORE=milvus docker compose up -d

logs

[2024/11/23 09:36:34.427 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.427 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.428 +00:00] [WARN] [grpcclient/client.go:483] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.428 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.430 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.430 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.430 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.430 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:531\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:355\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/11/23 09:36:34.495 +00:00] [DEBUG] [rootcoord/service.go:181] ["grpc init done ..."]
[2024/11/23 09:36:34.495 +00:00] [DEBUG] [rootcoord/service.go:184] ["RootCoord start to create DataCoord client"]
[2024/11/23 09:36:34.495 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.496 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.496 +00:00] [DEBUG] [rootcoord/service.go:201] ["RootCoord start to create QueryCoord client"]
[2024/11/23 09:36:34.496 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.497 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.500 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/23 09:36:34.500 +00:00] [DEBUG] [server/global_rmq.go:39] ["initializing global rmq"] [path=/var/lib/milvus/rdb_data]
[2024/11/23 09:36:34.501 +00:00] [DEBUG] [server/rocksmq_impl.go:179] ["Start rocksmq "] ["max proc"=3] [parallism=1] ["lru cache"=536870912]
[2024/11/23 09:36:34.511 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/23 09:36:34.512 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.513 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.514 +00:00] [DEBUG] [querycoord/service.go:155] ["QueryCoord try to wait for RootCoord ready"]
[2024/11/23 09:36:34.514 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.514 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.514 +00:00] [WARN] [grpcclient/client.go:483] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.514 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.515 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.515 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.515 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.515 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:156\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:95\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/11/23 09:36:34.518 +00:00] [INFO] [datanode/service.go:271] ["initializing RootCoord client for DataNode"]
[2024/11/23 09:36:34.519 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.519 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
[2024/11/23 09:36:34.520 +00:00] [INFO] [indexnode/indexnode.go:197] ["IndexNode init"] [state=Initializing]
[2024/11/23 09:36:34.520 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.521 +00:00] [DEBUG] [querynode/service.go:125] [QueryNode] [State=Initializing]
[2024/11/23 09:36:34.522 +00:00] [INFO] [querynodev2/server.go:239] ["QueryNode session info"] [metaPath=by-dev/meta]
[2024/11/23 09:36:34.522 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.523 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.524 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.528 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.528 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.528 +00:00] [WARN] [grpcclient/client.go:483] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.528 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.529 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:34.529 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.530 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.530 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.530 +00:00] [ERROR] [grpcclient/client.go:405] ["retry func failed"] ["retry time"=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [stack="github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:405\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:483\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:499\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:134\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:285\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:189\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/11/23 09:36:34.532 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.535 +00:00] [DEBUG] [sessionutil/session_util.go:343] ["Session get serverID success"] [key=id] [ServerId=1]
[2024/11/23 09:36:34.535 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=querynode] [address=172.20.0.6:21123] [id=1]
[2024/11/23 09:36:34.535 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:34.536 +00:00] [INFO] [querynodev2/server.go:154] ["QueryNode init session"] [nodeID=1] ["node address"=172.20.0.6:21123]
[2024/11/23 09:36:34.536 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/23 09:36:34.535 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=indexnode] [address=172.20.0.6:21121] [id=1]
[2024/11/23 09:36:34.536 +00:00] [INFO] [indexnode/indexnode.go:204] ["IndexNode init session successful"] [serverID=1]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20241123 09:36:34.536718    22 knowhere_config.cc:93] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384
I20241123 09:36:34.536818    22 knowhere_config.cc:104] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0
I20241123 09:36:34.536841    22 knowhere_config.cc:52] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.2.1
[2024/11/23 09:36:34.541 +00:00] [INFO] [indexnode/indexnode.go:209] ["Init IndexNode finished"] []
[2024/11/23 09:36:34.542 +00:00] [DEBUG] [indexnode/service.go:72] ["IndexNode init done ..."]
[2024/11/23 09:36:34.542 +00:00] [INFO] [indexnode/indexnode.go:221] [IndexNode] [State=Healthy]
[2024/11/23 09:36:34.542 +00:00] [INFO] [indexnode/indexnode.go:224] ["IndexNode start finished"] []
[2024/11/23 09:36:34.542 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexnode] [ServerID=1]
[2024/11/23 09:36:34.542 +00:00] [DEBUG] [indexnode/task_scheduler.go:242] ["IndexNode TaskScheduler start build loop ..."]
[2024/11/23 09:36:34.549 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexnode-1] [value="{\"ServerID\":1,\"ServerName\":\"indexnode\",\"Address\":\"172.20.0.6:21121\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120086}"]
[2024/11/23 09:36:34.550 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexnode] [serverID=1]
[2024/11/23 09:36:34.558 +00:00] [DEBUG] [indexnode/service.go:204] ["IndexNode Register etcd success"]
[2024/11/23 09:36:34.558 +00:00] [DEBUG] [indexnode/service.go:76] ["IndexNode start done ..."]
[2024/11/23 09:36:34.558 +00:00] [DEBUG] [components/index_node.go:55] ["IndexNode successfully started"]
[2024/11/23 09:36:34.607 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/23 09:36:37.605 +00:00] [next=2024/11/23 09:36:34.605 +00:00]
[2024/11/23 09:36:34.609 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/23 09:36:34.609 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.609 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/23 09:36:34.610 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.609 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/23 09:36:34.609 +00:00] [DEBUG] [server/rocksmq_retention.go:80] ["Rocksmq retention goroutine start!"]
[2024/11/23 09:36:34.610 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.612 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.613 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:34.613 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=rootcoord] [address=172.20.0.6:53100] [id=1]
[2024/11/23 09:36:34.613 +00:00] [INFO] [rootcoord/root_coord.go:159] ["update rootcoord state"] [state=Initializing]
[2024/11/23 09:36:34.614 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=indexnode] [key=by-dev/meta/session/indexnode-1]
[2024/11/23 09:36:34.614 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:34.615 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=indexcoord] [address=172.20.0.6:13333] [id=1]
[2024/11/23 09:36:34.615 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.616 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.618 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/23 09:36:37.616 +00:00] [next=2024/11/23 09:36:34.616 +00:00]
[2024/11/23 09:36:34.621 +00:00] [INFO] [rootcoord/root_coord.go:366] ["id allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/23 09:36:34.621 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:34.621 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=datacoord] [address=172.20.0.6:13333] [id=1]
[2024/11/23 09:36:34.621 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:34.622 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:34.624 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/23 09:36:37.622 +00:00] [next=2024/11/23 09:36:34.622 +00:00]
[2024/11/23 09:36:34.625 +00:00] [INFO] [rootcoord/root_coord.go:382] ["tso allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/23 09:36:34.626 +00:00] [INFO] [storage/minio_chunk_manager.go:132] ["blob bucket not exist, create bucket."] ["bucket name"=a-bucket]
[2024/11/23 09:36:34.626 +00:00] [INFO] [storage/minio_chunk_manager.go:132] ["blob bucket not exist, create bucket."] ["bucket name"=a-bucket]
[2024/11/23 09:36:34.632 +00:00] [DEBUG] [rootcoord/suffix_snapshot.go:557] ["suffix snapshot GC goroutine start!"]
[2024/11/23 09:36:34.632 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.635 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/11/23 09:36:34.640 +00:00] [WARN] [client/client.go:93] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/23 09:36:34.641 +00:00] [WARN] [grpcclient/client.go:176] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.641 +00:00] [WARN] [grpcclient/client.go:409] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.641 +00:00] [WARN] [storage/minio_chunk_manager.go:135] ["failed to create blob bucket"] [bucket=a-bucket] [error="Your previous request to create the named bucket succeeded and you already own it."]
[2024/11/23 09:36:34.641 +00:00] [ERROR] [storage/minio_chunk_manager.go:144] ["retry func failed"] ["retry time"=0] [error="Your previous request to create the named bucket succeeded and you already own it."] [stack="github.com/milvus-io/milvus/internal/storage.newMinioChunkManagerWithConfig\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_chunk_manager.go:144\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).newChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:51\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:58\ngithub.com/milvus-io/milvus/internal/util/dependency.(*DefaultFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/util/dependency/factory.go:149\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:268\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:237\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:126\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:223\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
[2024/11/23 09:36:34.643 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=0]
[2024/11/23 09:36:34.648 +00:00] [INFO] [rootcoord/meta_table.go:249] ["create database"] [db=default] [ts=454126362852589569]
[2024/11/23 09:36:34.650 +00:00] [INFO] [rootcoord/meta_table.go:188] ["recover collections from db"] [collection_num=0] [partition_num=0]
[2024/11/23 09:36:34.651 +00:00] [INFO] [datacoord/meta.go:170] ["DataCoord meta reloadFromKV done"] [duration=15.994846ms]
[2024/11/23 09:36:34.653 +00:00] [INFO] [rootcoord/meta_table.go:204] ["RootCoord meta table reload done"] [duration=26.718846ms]
[2024/11/23 09:36:34.654 +00:00] [INFO] [datacoord/channel_store.go:171] ["channel store reload done"] [duration=518.169µs]
[2024/11/23 09:36:34.656 +00:00] [INFO] [datacoord/server.go:479] ["DataCoord success to get DataNode sessions"] [sessions={}]
[2024/11/23 09:36:34.657 +00:00] [INFO] [datacoord/channel_manager.go:171] ["starting etcd states checker"]
[2024/11/23 09:36:34.657 +00:00] [INFO] [datacoord/channel_manager.go:176] ["starting background balance checker"]
[2024/11/23 09:36:34.657 +00:00] [INFO] [datacoord/channel_manager.go:179] ["cluster start up"] [nodes="[]"] [oNodes="[]"] ["old onlines"="[]"] ["new onlines"="[]"] [offLines="[]"]
[2024/11/23 09:36:34.659 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_0] [elapsed=3]
[2024/11/23 09:36:34.659 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_1] [elapsed=0]
[2024/11/23 09:36:34.659 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_2] [elapsed=0]
[2024/11/23 09:36:34.659 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_3] [elapsed=0]
[2024/11/23 09:36:34.659 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_4] [elapsed=0]
[2024/11/23 09:36:34.659 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_5] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_6] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_7] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_8] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_9] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_10] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_11] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_12] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_13] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_14] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [server/rocksmq_impl.go:426] ["Rocksmq create topic successfully "] [topic=by-dev-rootcoord-dml_15] [elapsed=0]
[2024/11/23 09:36:34.660 +00:00] [INFO] [rootcoord/dml_channels.go:216] ["init dml channels"] [prefix=by-dev-rootcoord-dml] [num=16]
[2024/11/23 09:36:34.660 +00:00] [DEBUG] [rootcoord/root_coord.go:451] ["RootCoord init QuotaCenter done"]
[2024/11/23 09:36:34.662 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-1] [address=172.20.0.6:21121]
[2024/11/23 09:36:34.662 +00:00] [DEBUG] [datacoord/indexnode_manager.go:83] ["add IndexNode"] [nodeID=1] ["node address"=172.20.0.6:21121]
[2024/11/23 09:36:34.662 +00:00] [DEBUG] [datacoord/indexnode_manager.go:56] ["set IndexNode client"] [nodeID=1]
[2024/11/23 09:36:34.662 +00:00] [DEBUG] [datacoord/indexnode_manager.go:61] ["IndexNode IndexNodeManager setClient success"] [nodeID=1] ["IndexNode num"=1]
[2024/11/23 09:36:34.662 +00:00] [INFO] [datacoord/garbage_collector.go:71] ["GC with option"] [enabled=true] [interval=1h0m0s] [missingTolerance=1h0m0s] [dropTolerance=3h0m0s]
[2024/11/23 09:36:34.662 +00:00] [DEBUG] [datacoord/service.go:232] ["DataCoord init done ..."]
[2024/11/23 09:36:34.662 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=indexcoord] [ServerID=1]
[2024/11/23 09:36:34.663 +00:00] [DEBUG] [rootcoord/kv_catalog.go:386] ["not found the user"] [key=root-coord/credential/users/root]
[2024/11/23 09:36:34.664 +00:00] [DEBUG] [rootcoord/root_coord.go:511] ["RootCoord init user root"]
[2024/11/23 09:36:34.670 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/indexcoord] [value="{\"ServerID\":1,\"ServerName\":\"indexcoord\",\"Address\":\"172.20.0.6:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120123}"]
[2024/11/23 09:36:34.670 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=indexcoord] [serverID=1]
[2024/11/23 09:36:34.670 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=datacoord] [ServerID=1]
[2024/11/23 09:36:34.673 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/datacoord] [value="{\"ServerID\":1,\"ServerName\":\"datacoord\",\"Address\":\"172.20.0.6:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120128}"]
[2024/11/23 09:36:34.673 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=datacoord] [serverID=1]
[2024/11/23 09:36:34.673 +00:00] [INFO] [datacoord/server.go:258] ["DataCoord Register Finished"]
[2024/11/23 09:36:34.673 +00:00] [DEBUG] [rootcoord/kv_catalog.go:386] ["not found the user"] [key=root-coord/credential/users/root]
[2024/11/23 09:36:34.675 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["index builder schedule loop start"]
[2024/11/23 09:36:34.676 +00:00] [INFO] [datacoord/index_service.go:91] ["start create index for segment loop..."]
[2024/11/23 09:36:34.675 +00:00] [INFO] [datacoord/server.go:394] ["DataCoord (re)starts successfully and re-collecting segment stats from DataNodes"]
[2024/11/23 09:36:34.678 +00:00] [INFO] [datacoord/server.go:1053] ["re-collecting segment stats from DataNodes"] ["DataNode IDs"="[]"]
[2024/11/23 09:36:34.678 +00:00] [INFO] [datacoord/server.go:378] ["DataCoord startup successfully"]
[2024/11/23 09:36:34.678 +00:00] [DEBUG] [datacoord/service.go:237] ["DataCoord start done ..."]
[2024/11/23 09:36:34.678 +00:00] [DEBUG] [components/data_coord.go:53] ["DataCoord successfully started"]
[2024/11/23 09:36:34.683 +00:00] [WARN] [rootcoord/kv_catalog.go:962] ["fail to load grant privilege entity"] [key=root-coord/credential/grantee-privileges/public/Global/*.*] [type=Grant] [error="there is no value on key = by-dev/meta/root-coord/credential/grantee-privileges/public/Global/*.*"]
[2024/11/23 09:36:34.685 +00:00] [WARN] [rootcoord/kv_catalog.go:984] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection] [error="there is no value on key = by-dev/meta/root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection"]
[2024/11/23 09:36:34.685 +00:00] [DEBUG] [rootcoord/kv_catalog.go:989] ["not found the grantee id"] [key=root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection]
[2024/11/23 09:36:34.692 +00:00] [WARN] [rootcoord/kv_catalog.go:984] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeShowCollections] [error="there is no value on key = by-dev/meta/root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeShowCollections"]
[2024/11/23 09:36:34.692 +00:00] [DEBUG] [rootcoord/kv_catalog.go:989] ["not found the grantee id"] [key=root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeShowCollections]
[2024/11/23 09:36:34.694 +00:00] [WARN] [rootcoord/kv_catalog.go:962] ["fail to load grant privilege entity"] [key=root-coord/credential/grantee-privileges/public/Collection/*.*] [type=Grant] [error="there is no value on key = by-dev/meta/root-coord/credential/grantee-privileges/public/Collection/*.*"]
[2024/11/23 09:36:34.699 +00:00] [WARN] [rootcoord/kv_catalog.go:984] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail] [error="there is no value on key = by-dev/meta/root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail"]
[2024/11/23 09:36:34.699 +00:00] [DEBUG] [rootcoord/kv_catalog.go:989] ["not found the grantee id"] [key=root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail]
[2024/11/23 09:36:34.701 +00:00] [DEBUG] [rootcoord/service.go:146] ["RootCoord init done ..."]
[2024/11/23 09:36:34.701 +00:00] [INFO] [rootcoord/service.go:288] ["RootCoord Core start ..."]
[2024/11/23 09:36:34.701 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=rootcoord] [ServerID=1]
[2024/11/23 09:36:34.704 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/rootcoord] [value="{\"ServerID\":1,\"ServerName\":\"rootcoord\",\"Address\":\"172.20.0.6:53100\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120151}"]
[2024/11/23 09:36:34.704 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=rootcoord] [serverID=1]
[2024/11/23 09:36:34.704 +00:00] [INFO] [rootcoord/root_coord.go:281] ["RootCoord Register Finished"]
[2024/11/23 09:36:34.708 +00:00] [INFO] [rootcoord/proxy_manager.go:88] ["succeed to init sessions on etcd"] [sessions=null] [revision=18]
[2024/11/23 09:36:34.710 +00:00] [INFO] [rootcoord/root_coord.go:159] ["update rootcoord state"] [state=Healthy]
[2024/11/23 09:36:34.710 +00:00] [INFO] [rootcoord/root_coord.go:665] ["rootcoord startup successfully"]
[2024/11/23 09:36:34.710 +00:00] [DEBUG] [rootcoord/service.go:151] ["RootCoord start done ..."]
[2024/11/23 09:36:34.710 +00:00] [INFO] [components/root_coord.go:55] ["RootCoord successfully started"]
[2024/11/23 09:36:34.710 +00:00] [WARN] [rootcoord/proxy_client_manager.go:235] ["proxy client is empty, RefreshPrivilegeInfoCache will not send to any client"]
[2024/11/23 09:36:34.710 +00:00] [INFO] [rootcoord/quota_center.go:135] ["Start QuotaCenter"] [collectInterval/s=3]
[2024/11/23 09:36:34.710 +00:00] [INFO] [rootcoord/proxy_manager.go:111] ["start to watch etcd"]
[2024/11/23 09:36:34.716 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=rootcoord] [key=by-dev/meta/session/rootcoord]
[2024/11/23 09:36:34.718 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.718 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=datacoord] [key=by-dev/meta/session/datacoord]
[2024/11/23 09:36:34.719 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.6:53100]
[2024/11/23 09:36:34.719 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.6:53100] [serverID=1]
[2024/11/23 09:36:34.731 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:34.732 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.6:53100]
[2024/11/23 09:36:34.732 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.6:53100] [serverID=1]
[2024/11/23 09:36:34.846 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/11/23 09:36:34.850 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/11/23 09:36:34.850 +00:00] [INFO] [querynodev2/server.go:293] ["queryNode try to connect etcd success"] [MetaRootPath=by-dev/meta]
[2024/11/23 09:36:34.850 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:25] ["query node use concurrent safe scheduler"] [max_concurrency=3]
[2024/11/23 09:36:34.851 +00:00] [INFO] [querynodev2/server.go:299] ["queryNode init scheduler"] [policy=fifo]
[2024/11/23 09:36:34.851 +00:00] [INFO] [segments/segment_loader.go:112] ["SegmentLoader created"] [ioPoolSize=32]
[2024/11/23 09:36:34.913 +00:00] [WARN] [rootcoord/root_coord.go:199] ["zero ts was met, this should be only occurred in starting state"] [minBgDdlTs=454126362928873473] [minNormalDdlTs=0]
[2024/11/23 09:36:34.915 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=3604720435]
[2024/11/23 09:36:34.915 +00:00] [INFO] [querynodev2/server.go:353] ["query node init successfully"] [queryNodeID=1] [Address=172.20.0.6:21123]
[2024/11/23 09:36:34.915 +00:00] [DEBUG] [querynode/service.go:226] ["QueryNode init done ..."]
[2024/11/23 09:36:34.915 +00:00] [INFO] [querynodev2/server.go:372] ["query node start successfully"] [queryNodeID=1] [Address=172.20.0.6:21123] [mmapEnabled=false]
[2024/11/23 09:36:34.915 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=querynode] [ServerID=1]
[2024/11/23 09:36:34.915 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:169] ["start execute loop"]
[2024/11/23 09:36:34.918 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/querynode-1] [value="{\"ServerID\":1,\"ServerName\":\"querynode\",\"Address\":\"172.20.0.6:21123\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120158}"]
[2024/11/23 09:36:34.919 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=querynode] [serverID=1]
[2024/11/23 09:36:34.922 +00:00] [DEBUG] [querynode/service.go:231] ["QueryNode start done ..."]
[2024/11/23 09:36:34.922 +00:00] [DEBUG] [components/query_node.go:58] ["QueryNode successfully started"]
[2024/11/23 09:36:35.017 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=querynode] [key=by-dev/meta/session/querynode-1]
[2024/11/23 09:36:35.042 +00:00] [WARN] [retry/retry.go:40] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2024/11/23 09:36:35.043 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.20.0.6:53100]
[2024/11/23 09:36:35.043 +00:00] [DEBUG] [client/client.go:96] ["RootCoordClient GetSessions success"] [address=172.20.0.6:53100] [serverID=1]
[2024/11/23 09:36:35.055 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=100] ["heapuse "=19] ["total memory"=624] ["next GC"=32] ["new GOGC"=200] [gc-pause=56.766µs] [gc-pause-end=1732354595054979456]
[2024/11/23 09:36:35.123 +00:00] [DEBUG] [querycoord/service.go:165] ["QueryCoord report RootCoord ready"]
[2024/11/23 09:36:35.123 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.123 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.124 +00:00] [DEBUG] [querycoord/service.go:184] ["QueryCoord try to wait for DataCoord ready"]
[2024/11/23 09:36:35.125 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=datacoord] [key=datacoord] [address=172.20.0.6:13333]
[2024/11/23 09:36:35.126 +00:00] [DEBUG] [querycoord/service.go:193] ["QueryCoord report DataCoord ready"]
[2024/11/23 09:36:35.126 +00:00] [INFO] [querycoordv2/server.go:174] ["QueryCoord start init"] [meta-root-path=by-dev/meta] [address=172.20.0.6:19531]
[2024/11/23 09:36:35.126 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.127 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.128 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:35.128 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=querycoord] [address=172.20.0.6:19531] [id=1]
[2024/11/23 09:36:35.128 +00:00] [INFO] [querycoordv2/server.go:206] [QueryCoord] [State=Initializing]
[2024/11/23 09:36:35.128 +00:00] [INFO] [querycoordv2/server.go:210] ["query coordinator try to connect etcd success"]
[2024/11/23 09:36:35.129 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/23 09:36:38.128 +00:00] [next=2024/11/23 09:36:35.128 +00:00]
[2024/11/23 09:36:35.129 +00:00] [INFO] [querycoordv2/server.go:301] ["init meta"]
[2024/11/23 09:36:35.129 +00:00] [INFO] [querycoordv2/server.go:310] ["recover meta..."]
[2024/11/23 09:36:35.130 +00:00] [INFO] [querycoordv2/server.go:311] ["recover collections and partitions from kv store"] [traceID=1732354595130213622]
[2024/11/23 09:36:35.130 +00:00] [INFO] [querycoordv2/server.go:317] ["recovering collections..."] [collections="[]"]
[2024/11/23 09:36:35.131 +00:00] [INFO] [querycoordv2/server.go:342] ["QueryCoord server initMeta done"] [duration=2.407256ms]
[2024/11/23 09:36:35.131 +00:00] [INFO] [querycoordv2/server.go:234] ["init session"]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:238] ["init schedulers"]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:251] ["init dist controller"]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:261] ["init all available balancer"]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:270] ["use config balancer"] [balancer=ScoreBasedBalancer]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:277] ["init checker controller"]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:347] ["init observers"]
[2024/11/23 09:36:35.132 +00:00] [INFO] [querycoordv2/server.go:294] ["QueryCoord init success"]
[2024/11/23 09:36:35.132 +00:00] [DEBUG] [querycoord/service.go:98] ["QueryCoord init done ..."]
[2024/11/23 09:36:35.132 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=querycoord] [ServerID=1]
[2024/11/23 09:36:35.137 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/querycoord] [value="{\"ServerID\":1,\"ServerName\":\"querycoord\",\"Address\":\"172.20.0.6:19531\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120174}"]
[2024/11/23 09:36:35.137 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=querycoord] [serverID=1]
[2024/11/23 09:36:35.137 +00:00] [INFO] [datanode/service.go:289] ["RootCoord client is ready for DataNode"]
[2024/11/23 09:36:35.137 +00:00] [DEBUG] [datanode/service.go:297] ["starting DataCoord client for DataNode"]
[2024/11/23 09:36:35.137 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.138 +00:00] [INFO] [querycoordv2/server.go:392] ["start watcher..."]
[2024/11/23 09:36:35.140 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.141 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-1] [address=172.20.0.6:21123]
[2024/11/23 09:36:35.141 +00:00] [INFO] [task/scheduler.go:222] ["add executor for new QueryNode"] [nodeID=1]
[2024/11/23 09:36:35.142 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["start dist handler"] [nodeID=1]
[2024/11/23 09:36:35.142 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=datacoord] [key=datacoord] [address=172.20.0.6:13333]
[2024/11/23 09:36:35.143 +00:00] [INFO] [datanode/service.go:315] ["DataCoord client is ready for DataNode"]
[2024/11/23 09:36:35.143 +00:00] [INFO] [sync/once.go:74] ["DataNode server initializing"] [TimeTickChannelName=by-dev-datacoord-timetick-channel]
[2024/11/23 09:36:35.143 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.145 +00:00] [INFO] [meta/resource_manager.go:470] ["HandleNodeUp: add node to default resource group"] [rgName=__default_resource_group] [node=1]
[2024/11/23 09:36:35.145 +00:00] [INFO] [querycoordv2/server.go:682] ["HandleNodeUp: assign node to resource group"] [nodeID=1] [resourceGroup=__default_resource_group]
[2024/11/23 09:36:35.146 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.150 +00:00] [WARN] [dist/dist_handler.go:100] ["node last heart beat time lag too behind"] [now=2024/11/23 09:36:35.150 +00:00] [lastHeartBeatTime=1970/01/01 00:00:00.000 +00:00] [nodeID=1]
[2024/11/23 09:36:35.150 +00:00] [INFO] [querycoordv2/server.go:420] ["start cluster..."]
[2024/11/23 09:36:35.150 +00:00] [INFO] [querycoordv2/server.go:423] ["start job scheduler..."]
[2024/11/23 09:36:35.150 +00:00] [INFO] [querycoordv2/server.go:426] ["start task scheduler..."]
[2024/11/23 09:36:35.150 +00:00] [INFO] [querycoordv2/server.go:429] ["start checker controller..."]
[2024/11/23 09:36:35.150 +00:00] [INFO] [querycoordv2/server.go:432] ["start observers..."]
[2024/11/23 09:36:35.150 +00:00] [INFO] [querycoordv2/server.go:386] ["QueryCoord started"]
[2024/11/23 09:36:35.150 +00:00] [DEBUG] [querycoord/service.go:103] ["QueryCoord start done ..."]
[2024/11/23 09:36:35.150 +00:00] [DEBUG] [components/query_coord.go:57] ["QueryCoord successfully started"]
[2024/11/23 09:36:35.150 +00:00] [INFO] [observers/resource_observer.go:62] ["Start check resource group loop"]
[2024/11/23 09:36:35.150 +00:00] [INFO] [observers/target_observer.go:89] ["Start update next target loop"]
[2024/11/23 09:36:35.150 +00:00] [INFO] [observers/replica_observer.go:64] ["Start check replica loop"]
[2024/11/23 09:36:35.151 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:35.151 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=datanode] [address=172.20.0.6:21124] [id=1]
[2024/11/23 09:36:35.152 +00:00] [INFO] [datanode/data_node.go:247] ["DataNode server init rateCollector done"] ["node ID"=1]
[2024/11/23 09:36:35.152 +00:00] [INFO] [datanode/data_node.go:250] ["DataNode server init dispatcher client done"] ["node ID"=1]
[2024/11/23 09:36:35.152 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/23 09:36:35.152 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/23 09:36:35.152 +00:00] [INFO] [datanode/data_node.go:263] ["DataNode server init succeeded"] [MsgChannelSubName=by-dev-dataNode]
[2024/11/23 09:36:35.152 +00:00] [INFO] [datanode/service.go:327] ["current DataNode state"] [state=Initializing]
[2024/11/23 09:36:35.152 +00:00] [INFO] [datanode/service.go:193] ["DataNode gRPC services successfully initialized"]
[2024/11/23 09:36:35.152 +00:00] [INFO] [datanode/data_node.go:498] ["start id allocator done"] [role=datanode]
[2024/11/23 09:36:35.159 +00:00] [INFO] [storage/minio_chunk_manager.go:154] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/11/23 09:36:35.159 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=datanode] [ServerID=1]
[2024/11/23 09:36:35.159 +00:00] [INFO] [datanode/data_node.go:477] ["DataNode Background GC Start"]
[2024/11/23 09:36:35.159 +00:00] [INFO] [datanode/data_node.go:277] ["Start watch channel"] [prefix=channelwatch/1]
[2024/11/23 09:36:35.162 +00:00] [DEBUG] [sessionutil/session_util.go:718] ["watch services"] ["add kv"="key:\"by-dev/meta/session/datanode-1\" create_revision:23 mod_revision:23 version:1 value:\"{\\\"ServerID\\\":1,\\\"ServerName\\\":\\\"datanode\\\",\\\"Address\\\":\\\"172.20.0.6:21124\\\",\\\"TriggerKill\\\":true,\\\"Version\\\":\\\"2.3.1\\\",\\\"LeaseID\\\":7587882954988120184}\" lease:7587882954988120184 "]
[2024/11/23 09:36:35.163 +00:00] [DEBUG] [sessionutil/session_util.go:746] [WatchService] ["event type"=SessionAddEvent]
[2024/11/23 09:36:35.161 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/datanode-1] [value="{\"ServerID\":1,\"ServerName\":\"datanode\",\"Address\":\"172.20.0.6:21124\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120184}"]
[2024/11/23 09:36:35.163 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=datanode] [serverID=1]
[2024/11/23 09:36:35.163 +00:00] [INFO] [datanode/data_node.go:190] ["DataNode Register Finished"]
[2024/11/23 09:36:35.163 +00:00] [INFO] [datacoord/server.go:825] ["received datanode register"] [address=172.20.0.6:21124] [serverID=1]
[2024/11/23 09:36:35.163 +00:00] [INFO] [datacoord/channel_manager.go:348] ["register node with no reassignment"] ["registered node"=1]
[2024/11/23 09:36:35.165 +00:00] [INFO] [datanode/service.go:198] ["DataNode gRPC services successfully started"]
[2024/11/23 09:36:35.165 +00:00] [DEBUG] [components/data_node.go:56] ["Datanode successfully started"]
[2024/11/23 09:36:35.218 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=querycoord] [key=by-dev/meta/session/querycoord]
[2024/11/23 09:36:35.219 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=datanode] [key=by-dev/meta/session/datanode-1]
[2024/11/23 09:36:35.660 +00:00] [DEBUG] [datanode/timetick_sender.go:160] ["timeTickSender send datanode timetick message"] [toSendMsgs=null] [sendLastTss={}]
[2024/11/23 09:36:35.661 +00:00] [DEBUG] [datanode/timetick_sender.go:155] ["timeTickSender channelStatesCaches"] [sendedLastTss={}] [sizeBeforeClean=0] [sizeAfterClean=0]
[2024/11/23 09:36:35.849 +00:00] [DEBUG] [proxy/service.go:535] ["Proxy wait for RootCoord to be healthy done"]
[2024/11/23 09:36:35.852 +00:00] [DEBUG] [proxy/service.go:537] ["set RootCoord client for Proxy"]
[2024/11/23 09:36:35.852 +00:00] [DEBUG] [proxy/service.go:539] ["set RootCoord client for Proxy done"]
[2024/11/23 09:36:35.852 +00:00] [DEBUG] [proxy/service.go:543] ["create DataCoord client for Proxy"]
[2024/11/23 09:36:35.852 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.852 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.853 +00:00] [DEBUG] [proxy/service.go:549] ["create DataCoord client for Proxy done"]
[2024/11/23 09:36:35.853 +00:00] [DEBUG] [proxy/service.go:552] ["init DataCoord client for Proxy"]
[2024/11/23 09:36:35.853 +00:00] [DEBUG] [proxy/service.go:557] ["init DataCoord client for Proxy done"]
[2024/11/23 09:36:35.853 +00:00] [DEBUG] [proxy/service.go:559] ["Proxy wait for DataCoord to be healthy"]
[2024/11/23 09:36:35.853 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=datacoord] [key=datacoord] [address=172.20.0.6:13333]
[2024/11/23 09:36:35.854 +00:00] [DEBUG] [proxy/service.go:564] ["Proxy wait for DataCoord to be healthy done"]
[2024/11/23 09:36:35.854 +00:00] [DEBUG] [proxy/service.go:566] ["set DataCoord client for Proxy"]
[2024/11/23 09:36:35.854 +00:00] [DEBUG] [proxy/service.go:568] ["set DataCoord client for Proxy done"]
[2024/11/23 09:36:35.854 +00:00] [DEBUG] [proxy/service.go:572] ["create QueryCoord client for Proxy"]
[2024/11/23 09:36:35.854 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.854 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.855 +00:00] [DEBUG] [proxy/service.go:578] ["create QueryCoord client for Proxy done"]
[2024/11/23 09:36:35.855 +00:00] [DEBUG] [proxy/service.go:581] ["init QueryCoord client for Proxy"]
[2024/11/23 09:36:35.855 +00:00] [DEBUG] [proxy/service.go:586] ["init QueryCoord client for Proxy done"]
[2024/11/23 09:36:35.855 +00:00] [DEBUG] [proxy/service.go:588] ["Proxy wait for QueryCoord to be healthy"]
[2024/11/23 09:36:35.857 +00:00] [DEBUG] [sessionutil/session_util.go:559] ["SessionUtil GetSessions"] [prefix=querycoord] [key=querycoord] [address=172.20.0.6:19531]
[2024/11/23 09:36:35.858 +00:00] [DEBUG] [proxy/service.go:593] ["Proxy wait for QueryCoord to be healthy done"]
[2024/11/23 09:36:35.858 +00:00] [DEBUG] [proxy/service.go:595] ["set QueryCoord client for Proxy"]
[2024/11/23 09:36:35.858 +00:00] [DEBUG] [proxy/service.go:597] ["set QueryCoord client for Proxy done"]
[2024/11/23 09:36:35.858 +00:00] [DEBUG] [proxy/service.go:599] ["update Proxy's state to Initializing"]
[2024/11/23 09:36:35.858 +00:00] [DEBUG] [proxy/service.go:602] ["init Proxy"]
[2024/11/23 09:36:35.858 +00:00] [INFO] [proxy/proxy.go:190] ["init session for Proxy"]
[2024/11/23 09:36:35.858 +00:00] [DEBUG] [sessionutil/session_util.go:227] ["Session try to connect to etcd"]
[2024/11/23 09:36:35.859 +00:00] [DEBUG] [sessionutil/session_util.go:242] ["Session connect to etcd success"]
[2024/11/23 09:36:35.861 +00:00] [DEBUG] [sessionutil/session_util.go:285] [getServerID] [reuse=true]
[2024/11/23 09:36:35.861 +00:00] [INFO] [sessionutil/session_util.go:259] ["start server"] [name=proxy] [address=172.20.0.6:19529] [id=1]
[2024/11/23 09:36:35.861 +00:00] [INFO] [proxy/proxy.go:195] ["init session for Proxy done"]
[2024/11/23 09:36:35.861 +00:00] [INFO] [dependency/factory.go:83] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/23 09:36:35.861 +00:00] [INFO] [msgstream/mq_factory.go:17] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/23 09:36:35.861 +00:00] [INFO] [accesslog/log_writer.go:74] ["Access log save to /tmp/milvus_accesslog"]
[2024/11/23 09:36:35.861 +00:00] [INFO] [accesslog/log_writer.go:260] ["start time rotating of access log"]
[2024/11/23 09:36:35.863 +00:00] [DEBUG] [proxy/proxy.go:200] ["init access log for Proxy done"]
[2024/11/23 09:36:35.863 +00:00] [INFO] [proxy/proxy.go:206] ["Proxy init rateCollector done"] [nodeID=1]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:216] ["create id allocator done"] [role=proxy] [ProxyID=1]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:226] ["create timestamp allocator done"] [role=proxy] [ProxyID=1]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:237] ["create segment id assigner done"] [role=proxy] [ProxyID=1]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:242] ["create channels manager done"] [role=proxy]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:249] ["create task scheduler done"] [role=proxy]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:253] ["create channels time ticker done"] [role=proxy] [syncTimeTickInterval=100ms]
[2024/11/23 09:36:35.864 +00:00] [DEBUG] [proxy/proxy.go:256] ["create metrics cache manager done"] [role=proxy]
[2024/11/23 09:36:35.868 +00:00] [DEBUG] [rootcoord/service.go:507] [PolicyList] [role=rootcoord] [in=]
[2024/11/23 09:36:35.871 +00:00] [DEBUG] [rootcoord/service.go:507] ["PolicyList success"] [role=rootcoord] [in=]
[2024/11/23 09:36:35.875 +00:00] [INFO] [proxy/meta_cache.go:226] ["success to init meta cache"] [policy_infos="[\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Collection-*.*\\\",\\\"V2\\\":\\\"PrivilegeIndexDetail\\\"}\",\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Global-*.*\\\",\\\"V2\\\":\\\"PrivilegeDescribeCollection\\\"}\",\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Global-*.*\\\",\\\"V2\\\":\\\"PrivilegeShowCollections\\\"}\"]"]
[2024/11/23 09:36:35.875 +00:00] [DEBUG] [proxy/proxy.go:262] ["init meta cache done"] [role=proxy]
[2024/11/23 09:36:35.875 +00:00] [DEBUG] [proxy/service.go:607] ["init Proxy done"]
---Milvus Proxy successfully initialized and ready to serve!---
[2024/11/23 09:36:35.875 +00:00] [DEBUG] [proxy/service.go:359] ["init Proxy server done"]
[2024/11/23 09:36:35.875 +00:00] [DEBUG] [proxy/service.go:361] ["start Proxy server"]
[2024/11/23 09:36:35.876 +00:00] [DEBUG] [proxy/proxy.go:354] ["start task scheduler done"] [role=proxy]
[2024/11/23 09:36:35.876 +00:00] [DEBUG] [proxy/proxy.go:360] ["start id allocator done"] [role=proxy]
[2024/11/23 09:36:35.876 +00:00] [DEBUG] [proxy/proxy.go:366] ["start segment id assigner done"] [role=proxy]
[2024/11/23 09:36:35.877 +00:00] [DEBUG] [proxy/proxy.go:372] ["start channels time ticker done"] [role=proxy]
[2024/11/23 09:36:35.877 +00:00] [DEBUG] [proxy/proxy.go:381] ["update state code"] [role=proxy] [State=Healthy]
[2024/11/23 09:36:35.877 +00:00] [DEBUG] [sessionutil/session_util.go:406] ["service begin to register to etcd"] [serverName=proxy] [ServerID=1]
[2024/11/23 09:36:35.883 +00:00] [INFO] [sessionutil/session_util.go:435] ["put session key into etcd"] [key=by-dev/meta/session/proxy-1] [value="{\"ServerID\":1,\"ServerName\":\"proxy\",\"Address\":\"172.20.0.6:19529\",\"TriggerKill\":true,\"Version\":\"2.3.1\",\"LeaseID\":7587882954988120199}"]
[2024/11/23 09:36:35.883 +00:00] [INFO] [sessionutil/session_util.go:445] ["Service registered successfully"] [ServerName=proxy] [serverID=1]
[2024/11/23 09:36:35.883 +00:00] [INFO] [proxy/proxy.go:143] ["Proxy Register Finished"]
[2024/11/23 09:36:35.883 +00:00] [DEBUG] [rootcoord/proxy_manager.go:157] ["received proxy put event with session"] [session="Session:<ServerID: 1, ServerName: proxy, Version: 2.3.1>"]
[2024/11/23 09:36:35.883 +00:00] [INFO] [rootcoord/timeticksync.go:233] ["Add session for timeticksync"] [serverID=1]
[2024/11/23 09:36:35.883 +00:00] [INFO] [rootcoord/proxy_client_manager.go:125] ["succeed to create proxy client"] [address=172.20.0.6:19529] [serverID=1]
[2024/11/23 09:36:35.884 +00:00] [INFO] [proxy/service.go:627] ["start Proxy http server"]
[2024/11/23 09:36:35.886 +00:00] [DEBUG] [proxy/service.go:366] ["start Proxy server done"]
[2024/11/23 09:36:35.886 +00:00] [INFO] [components/proxy.go:55] ["Proxy successfully started"]
[2024/11/23 09:36:35.886 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2024/11/23 09:36:35.886 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2024/11/23 09:36:35.887 +00:00] [INFO] [tracer/tracer.go:71] ["Init tracer finished"] [Exporter=stdout]
[2024/11/23 09:36:35.923 +00:00] [INFO] [sessionutil/session_util.go:852] ["register session success"] [role=proxy] [key=by-dev/meta/session/proxy-1]
[2024/11/23 09:37:11.491 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=26] ["total memory"=2071] ["next GC"=62] ["new GOGC"=200] [gc-pause=62.076µs] [gc-pause-end=1732354631489545657]
[2024/11/23 09:38:10.384 +00:00] [INFO] [gc/gc_tuner.go:90] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=27] ["total memory"=2103] ["next GC"=62] ["new GOGC"=200] [gc-pause=144.911µs] [gc-pause-end=1732354690382159286]

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

dosubot[bot] commented 5 days ago

The errors you're encountering seem to be related to the initialization and state management of components in the Dify platform, particularly with Milvus components like rootcoord and querycoord, as well as MinIO storage.

  1. RootCoord and QueryCoord Errors: The messages "find no available rootcoord, check rootcoord state" and "find no available querycoord, check querycoord state" suggest that these components are not starting up correctly. This could be due to configuration issues, network problems, or resource constraints. Ensure that all necessary services are running and properly configured in your Docker setup. Check the logs for any additional error messages that might provide more context.

  2. MinIO Bucket Error: The error "Your previous request to create the named bucket succeeded and you already own it" indicates that the MinIO bucket already exists. This is generally a warning rather than a critical error. You can verify the bucket's existence and permissions to ensure that the application can access it correctly.

  3. Configuration Check: Double-check your .env file and Docker Compose configuration to ensure that all environment variables and service dependencies are correctly set up. The VECTOR_STORE=milvus setting indicates that Milvus is being used as the vector database, so ensure that Milvus is correctly configured and running [1].

  4. Etcd Connectivity: The logs show multiple attempts to connect to etcd, which are successful. However, ensure that etcd is fully operational and accessible by all components.

  5. Resource Allocation: Ensure that your Docker environment has sufficient resources (CPU, memory) allocated to handle the services you're running, as resource constraints can lead to initialization failures.

If these steps do not resolve the issue, consider checking for any updates or patches for the Dify platform or Milvus that might address these startup issues.

To continue talking to Dosu, mention @dosu.

taowang1993 commented 4 days ago

After investigation, I discovered that these errors are caused by code changes after 0.11.2.

There must be a bug in recent commits or pull requests.

crazywoola commented 4 days ago

Maybe it's related to python version. We are now supporting 3.12 in our code repo, maybe you can try to install the deps again with 3.12.