search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.89k stars 2.94k forks source link

[Bug]: Setting the user name and password does not take effect. #36987

Open wpx1997-stack opened 1 month ago

wpx1997-stack commented 1 month ago

Is there an existing issue for this?

Environment

- Milvus version:milvus:v2.4.13

Current Behavior

When using Docker Compose for deployment, setting the username and password does not take effect. The Docker Compose configuration information is as follows:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    restart: always
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - /mnt/milvus/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    restart: always
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
#    ports:
#      - "9001:9001"
#      - "9000:9000"
    volumes:
      - /mnt/milvus/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.13
    restart: always
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    logging: 
      driver: "json-file"
      options: 
            max-size: "1g"
    volumes:
      - /mnt/milvus/volumes/milvus:/var/lib/milvus
      - /mnt/milvus/volumes/milvus/config/milvus.yaml:/milvus/configs/milvus.yaml
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"
  attu:
    container_name: milvus-attu
    image: zilliz/attu:v2.4.8
    restart: always
    environment:
      MILVUS_URL: "192.168.1.141:19530"
    ports:
      - "3010:3000"

networks:
  default:  
    driver: bridge
  milvus:  
    external: true

The milvus.yaml sets the username and password as follows:

  security:
    authorizationEnabled: true
    # The superusers will ignore some system check processes
    # like the old password verification when updating the credential
    superUsers: root
    defaultRootPassword: 1234 # default password for root user
    tlsMode: 0

When using Attu, I found that I cannot log in. image

After entering the Docker container and checking the /milvus/configs/milvus.yaml file, I found that the settings have been successfully applied. image

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

xiaofan-luan commented 1 month ago

@wpx1997-stack the setup only works the first time you create the cluster.

if the cluster is already initialzied, you can not use config files to change root username.

you have to use API to create user and modify existing user's use name.

xiaofan-luan commented 1 month ago
image

if you take a look on this, only if credential is no there can you set the default pasword

xiaofan-luan commented 1 month ago

/assign @wpx1997-stack

yanliang567 commented 1 month ago

/unassign @wpx1997-stack please make sure you are setting the credential with the initial deployment.

iamkhalidbashir commented 2 weeks ago

This is an issue for me too I am getting this error with initial deployment (when the bucket is empty, the milvus standalone rdb is volumes are new and etcd is also new) fresh deployment image image

Image: milvusdb/milvus:v2.4.15

Please fix at priority

logs:

[2024/11/07 11:42:14.303 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=/var/log/milvus]
[2024/11/07 11:42:14.303 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=standalone-0.log]
[2024/11/07 11:42:14.303 +00:00] [INFO] [roles/roles.go:282] [setupPrometheusHTTPServer]
[2024/11/07 11:42:14.303 +00:00] [INFO] [http/server.go:160] ["management listen"] [addr=:9091]
[2024/11/07 11:42:14.304 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=7294235443]
[2024/11/07 11:42:14.304 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2024/11/07 11:42:14.305 +00:00] [INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2024/11/07 11:42:14.305 +00:00] [INFO] [rootcoord/service.go:196] ["init params done.."]
[2024/11/07 11:42:14.305 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.305 +00:00] [INFO] [datacoord/service.go:96] ["DataCoord listen on"] [address="[::]:13333"] [port=13333]
[2024/11/07 11:42:14.306 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.306 +00:00] [INFO] [components/index_coord.go:42] ["IndexCoord running ..."]
[2024/11/07 11:42:14.306 +00:00] [INFO] [querycoord/service.go:106] ["QueryCoord listen on"] [address="[::]:19531"] [port=19531]
[2024/11/07 11:42:14.306 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.307 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]
[2024/11/07 11:42:14.307 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.308 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]
[2024/11/07 11:42:14.309 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.310 +00:00] [INFO] [indexnode/service.go:78] ["IndexNode listen on"] [address="[::]:21121"] [port=21121]
[2024/11/07 11:42:14.311 +00:00] [INFO] [proxy/look_aside_balancer.go:225] ["Start check query node health loop"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/listener_manager.go:35] ["Proxy listen on external grpc listener"] [address=172.18.0.2:19530] [port=19530]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/listener_manager.go:45] ["Proxy listen on internal grpc listener"] [address=172.18.0.2:19529] [port=19529]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/listener_manager.go:79] ["Proxy server(http) and external grpc server share the same port"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/service.go:460] ["init Proxy server"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/service.go:479] ["Proxy init service's parameter table done"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/service.go:481] ["Proxy init http server's parameter table done"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [proxy/service.go:485] ["init Proxy's tracer done"] ["service name"="Proxy ip: 172.18.0.2, port: 19530"]
[2024/11/07 11:42:14.312 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.411 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.611 +00:00] [INFO] [rootcoord/service.go:216] ["etcd connect done ..."]
[2024/11/07 11:42:14.611 +00:00] [INFO] [rootcoord/service.go:276] ["start grpc "] [port=53100]
[2024/11/07 11:42:14.615 +00:00] [INFO] [datanode/service.go:253] ["DataNode address"] [address=172.18.0.2:21124]
[2024/11/07 11:42:14.615 +00:00] [INFO] [datanode/service.go:254] ["DataNode serverID"] [serverID=0]
[2024/11/07 11:42:14.617 +00:00] [INFO] [proxy/service.go:437] ["create Proxy internal grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"]
[2024/11/07 11:42:14.617 +00:00] [INFO] [proxy/service.go:524] ["register Proxy http server"]
[2024/11/07 11:42:14.618 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:42:14.710 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:42:14.711 +00:00] [INFO] [rootcoord/service.go:233] ["grpc init done ..."]
[2024/11/07 11:42:14.711 +00:00] [INFO] [rootcoord/service.go:236] ["RootCoord start to create DataCoord client"]
[2024/11/07 11:42:14.714 +00:00] [INFO] [indexnode/indexnode.go:209] ["IndexNode init"] [state=Initializing]
[2024/11/07 11:42:14.715 +00:00] [INFO] [querynodev2/server.go:278] ["QueryNode session info"] [metaPath=milvus-blainy/meta]
[2024/11/07 11:42:14.716 +00:00] [INFO] [datanode/service.go:263] ["initializing RootCoord client for DataNode"]
[2024/11/07 11:42:14.765 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:42:14.818 +00:00] [INFO] [rootcoord/service.go:245] ["RootCoord start to create QueryCoord client"]
[2024/11/07 11:42:14.818 +00:00] [INFO] [querycoord/service.go:180] ["QueryCoord try to wait for RootCoord ready"]
[2024/11/07 11:42:14.820 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:42:14.820 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.820 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.820 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:42:14.820 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:42:14.820 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:42:14.820 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.820 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.820 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:14.821 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:42:14.821 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.821 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.821 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:270\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:184\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:14.820 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:541\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:14.821 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:42:14.821 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.821 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.821 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:14.822 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:42:14.822 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.822 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.822 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:42:14.822 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.822 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:42:14.822 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:541\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:14.822 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:270\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:184\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:14.823 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querynode] [address=172.18.0.2:21123] [id=1]
[2024/11/07 11:42:14.823 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexcoord] [address=172.18.0.2:13333] [id=1]
[2024/11/07 11:42:14.824 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexnode] [address=172.18.0.2:21121] [id=1]
[2024/11/07 11:42:14.824 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querynode-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:14.824 +00:00] [INFO] [querynodev2/server.go:163] ["QueryNode init session"] [nodeID=1] ["node address"=172.18.0.2:21123]
[2024/11/07 11:42:14.824 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:42:14.824 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:42:14.824 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=rootcoord] [address=172.18.0.2:53100] [id=1]
[2024/11/07 11:42:14.825 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Initializing]
[2024/11/07 11:42:14.824 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="indexnode-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:14.825 +00:00] [INFO] [indexnode/indexnode.go:216] ["IndexNode init session successful"] [serverID=1]
[2024/11/07 11:42:14.826 +00:00] [INFO] [indexnode/indexnode.go:221] ["init index node done"] [nodeID=1] [Address=172.18.0.2:21121]
[2024/11/07 11:42:14.826 +00:00] [INFO] [indexnode/indexnode.go:232] [IndexNode] [State=Healthy]
[2024/11/07 11:42:14.826 +00:00] [INFO] [indexnode/indexnode.go:235] ["IndexNode start finished"] []
[2024/11/07 11:42:14.828 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=datacoord] [address=172.18.0.2:13333] [id=1]
[2024/11/07 11:42:14.829 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:42:17.826 +00:00] [next=2024/11/07 11:42:14.826 +00:00]
[2024/11/07 11:42:14.829 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=milvus-blainy/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/07 11:42:14.830 +00:00] [INFO] [datacoord/server.go:348] ["init rootcoord client done"]
[2024/11/07 11:42:14.832 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/indexnode-1] [value="{\"ServerID\":1,\"ServerName\":\"indexnode\",\"Address\":\"172.18.0.2:21121\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319772,\"HostName\":\"378c7bb1eec5\",\"EnableDisk\":true}"]
[2024/11/07 11:42:14.832 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexnode] [serverID=1]
[2024/11/07 11:42:14.832 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:42:17.830 +00:00] [next=2024/11/07 11:42:14.830 +00:00]
[2024/11/07 11:42:14.832 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=milvus-blainy/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/07 11:42:14.833 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
[2024/11/07 11:42:14.833 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=0]
[2024/11/07 11:42:14.836 +00:00] [INFO] [rootcoord/meta_table.go:282] ["create database"] [db=default] [ts=453765951607275521]
[2024/11/07 11:42:14.839 +00:00] [INFO] [rootcoord/meta_table.go:193] ["collections recovered from db"] [db_name=default] [collection_num=0] [partition_num=0]
[2024/11/07 11:42:14.840 +00:00] [INFO] [rootcoord/meta_table.go:209] ["RootCoord meta table reload done"] [duration=7.51421ms]
[2024/11/07 11:42:14.840 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.841 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.842 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.842 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.842 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.842 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.842 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:14.842 +00:00] [INFO] [rootcoord/dml_channels.go:215] ["init dml channels"] [prefix=milvus-blainy-rootcoord-dml] [num=16]
[2024/11/07 11:42:14.842 +00:00] [INFO] [rootcoord/root_coord.go:451] ["create TimeTick sync done"]
[2024/11/07 11:42:14.842 +00:00] [INFO] [rootcoord/root_coord.go:467] ["init proxy manager done"]
[2024/11/07 11:42:14.845 +00:00] [INFO] [rootcoord/root_coord.go:477] ["init credentials done"]
[2024/11/07 11:42:14.851 +00:00] [WARN] [rootcoord/kv_catalog.go:1026] ["fail to load grant privilege entity"] [key=root-coord/credential/grantee-privileges/public/Global/*.*] [type=Grant] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-privileges/public/Global/*.*]"]
[2024/11/07 11:42:14.853 +00:00] [WARN] [rootcoord/kv_catalog.go:1048] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection]"]
[2024/11/07 11:42:14.855 +00:00] [WARN] [rootcoord/kv_catalog.go:1026] ["fail to load grant privilege entity"] [key=root-coord/credential/grantee-privileges/public/Collection/*.*] [type=Grant] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-privileges/public/Collection/*.*]"]
[2024/11/07 11:42:14.858 +00:00] [WARN] [rootcoord/kv_catalog.go:1048] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail]"]
[2024/11/07 11:42:14.859 +00:00] [INFO] [rootcoord/root_coord.go:483] ["init rootcoord done"] [nodeID=1] [Address=172.18.0.2:53100]
[2024/11/07 11:42:14.859 +00:00] [INFO] [rootcoord/service.go:182] ["RootCoord init done ..."]
[2024/11/07 11:42:14.859 +00:00] [INFO] [rootcoord/service.go:316] ["RootCoord Core start ..."]
[2024/11/07 11:42:14.863 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/rootcoord] [value="{\"ServerID\":1,\"ServerName\":\"rootcoord\",\"Address\":\"172.18.0.2:53100\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319808,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:14.863 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=rootcoord] [serverID=1]
[2024/11/07 11:42:14.863 +00:00] [INFO] [rootcoord/root_coord.go:273] ["RootCoord Register Finished"]
[2024/11/07 11:42:14.864 +00:00] [INFO] [proxyutil/proxy_watcher.go:96] ["succeed to init sessions on etcd"] [sessions=null] [revision=15]
[2024/11/07 11:42:14.865 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Healthy]
[2024/11/07 11:42:14.865 +00:00] [WARN] [proxyutil/proxy_client_manager.go:263] ["proxy client is empty, RefreshPrivilegeInfoCache will not send to any client"]
[2024/11/07 11:42:14.865 +00:00] [INFO] [proxyutil/proxy_watcher.go:119] ["start to watch etcd"]
[2024/11/07 11:42:14.865 +00:00] [INFO] [rootcoord/quota_center.go:310] ["Start QuotaCenter"] [collectInterval=3s]
[2024/11/07 11:42:14.865 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="rootcoord-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:14.865 +00:00] [INFO] [rootcoord/root_coord.go:713] ["rootcoord startup successfully"]
[2024/11/07 11:42:14.865 +00:00] [INFO] [rootcoord/service.go:187] ["RootCoord start done ..."]
[2024/11/07 11:42:14.865 +00:00] [INFO] [components/root_coord.go:62] ["RootCoord successfully started"]
[2024/11/07 11:42:14.906 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=rootcoord] [key=milvus-blainy/meta/session/rootcoord]
[2024/11/07 11:42:15.022 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:15.022 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:541\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:15.023 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:270\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:184\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:15.261 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-blainy-prod] [root=files]
[2024/11/07 11:42:15.261 +00:00] [INFO] [datacoord/server.go:357] ["init chunk manager factory done"]
[2024/11/07 11:42:15.261 +00:00] [INFO] [datacoord/server.go:644] ["data coordinator connecting to metadata store"] [metaType=etcd]
[2024/11/07 11:42:15.261 +00:00] [INFO] [datacoord/server.go:657] ["data coordinator successfully connected to metadata store"] [metaType=etcd]
[2024/11/07 11:42:15.265 +00:00] [INFO] [datacoord/index_meta.go:97] ["indexMeta reloadFromKV done"] [duration=3.885086ms]
[2024/11/07 11:42:15.268 +00:00] [INFO] [datacoord/analyze_meta.go:70] ["analyzeMeta reloadFromKV done"] [duration=3.303644ms]
[2024/11/07 11:42:15.270 +00:00] [INFO] [datacoord/partition_stats_meta.go:67] ["DataCoord partitionStatsMeta reloadFromKV done"] [duration=1.469482ms]
[2024/11/07 11:42:15.271 +00:00] [INFO] [datacoord/compaction_task_meta.go:62] ["DataCoord compactionTaskMeta reloadFromKV done"] [duration=589.521µs]
[2024/11/07 11:42:15.274 +00:00] [INFO] [datacoord/meta.go:229] ["DataCoord meta reloadFromKV done"] [duration=3.299525ms]
[2024/11/07 11:42:15.280 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-blainy-prod] [root=files]
[2024/11/07 11:42:15.281 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=4]
[2024/11/07 11:42:15.281 +00:00] [INFO] [querynodev2/server.go:319] ["queryNode init scheduler"] [policy=fifo]
[2024/11/07 11:42:15.281 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [msgID=0]
[2024/11/07 11:42:15.281 +00:00] [WARN] [rootcoord/list_db_task.go:56] ["get current user from context failed"] [error="fail to get authorization from the md, authorization:[token]"]
[2024/11/07 11:42:15.281 +00:00] [INFO] [segments/segment_loader.go:566] ["SegmentLoader created"] [ioPoolSize=32]
[2024/11/07 11:42:15.281 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [msgID=0] ["num of databases"=1]
[2024/11/07 11:42:15.283 +00:00] [INFO] [querynodev2/server.go:230] ["set up knowhere build pool size"] [pool_size=2]
[2024/11/07 11:42:15.283 +00:00] [WARN] [rootcoord/show_collection_task.go:63] ["get current user from context failed"] [error="fail to get authorization from the md, authorization:[token]"]
[2024/11/07 11:42:15.292 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:42:18.277 +00:00] [next=2024/11/07 11:42:15.277 +00:00]
[2024/11/07 11:42:15.293 +00:00] [INFO] [datacoord/server.go:1257] ["all old data node down, enable auto balance!"]
[2024/11/07 11:42:15.295 +00:00] [INFO] [datacoord/channel_store_v2.go:71] ["channel store reload done"] [duration=964.041µs]
[2024/11/07 11:42:15.295 +00:00] [INFO] [datacoord/server.go:379] ["init datanode cluster done"]
[2024/11/07 11:42:15.295 +00:00] [INFO] [datacoord/server.go:561] ["DataCoord success to get DataNode sessions"] [sessions={}]
[2024/11/07 11:42:15.296 +00:00] [INFO] [datacoord/server.go:582] ["DataCoord Cluster Manager start up"]
[2024/11/07 11:42:15.296 +00:00] [INFO] [datacoord/channel_manager_v2.go:163] ["starting channel balance loop"]
[2024/11/07 11:42:15.296 +00:00] [INFO] [datacoord/channel_manager_v2.go:171] ["cluster start up"] [allNodes="[]"] [legacyNodes="[]"] [oldNodes="[]"] [newOnlines="[]"] [offLines="[]"]
[2024/11/07 11:42:15.296 +00:00] [INFO] [datacoord/server.go:587] ["DataCoord Cluster Manager start up successfully"]
[2024/11/07 11:42:15.299 +00:00] [INFO] [datacoord/server.go:386] ["init service discovery done"]
[2024/11/07 11:42:15.299 +00:00] [INFO] [datacoord/server.go:390] ["init compaction done"]
[2024/11/07 11:42:15.299 +00:00] [INFO] [datacoord/server.go:395] ["init segment manager done"]
[2024/11/07 11:42:15.300 +00:00] [INFO] [datacoord/garbage_collector.go:85] ["GC with option"] [enabled=true] [interval=1h0m0s] [scanInterval=168h0m0s] [missingTolerance=24h0m0s] [dropTolerance=3h0m0s]
[2024/11/07 11:42:15.302 +00:00] [INFO] [datacoord/server.go:410] ["init datacoord done"] [nodeID=1] [Address=172.18.0.2:13333]
[2024/11/07 11:42:15.312 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/indexcoord] [value="{\"ServerID\":1,\"ServerName\":\"indexcoord\",\"Address\":\"172.18.0.2:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319836,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:15.312 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexcoord] [serverID=1]
[2024/11/07 11:42:15.315 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/datacoord] [value="{\"ServerID\":1,\"ServerName\":\"datacoord\",\"Address\":\"172.18.0.2:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319839,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:15.315 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=datacoord] [serverID=1]
[2024/11/07 11:42:15.315 +00:00] [INFO] [datacoord/server.go:270] ["DataCoord Register Finished"]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/sync_segments_scheduler.go:77] ["SyncSegmentsScheduler started..."]
[2024/11/07 11:42:15.316 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="datacoord-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/server.go:424] ["DataCoord startup successfully"]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/task_scheduler.go:168] ["task scheduler loop start"]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/compaction.go:341] ["compactionPlanHandler start loop schedule"]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/compaction.go:360] ["compactionPlanHandler start loop check"] ["check result interval"=3s]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/compaction.go:380] ["compactionPlanHandler start clean check loop"] ["gc interval"=30m0s]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/import_scheduler.go:75] ["start import scheduler"]
[2024/11/07 11:42:15.316 +00:00] [INFO] [datacoord/index_service.go:113] ["start create index for segment loop..."]
[2024/11/07 11:42:15.317 +00:00] [INFO] [datacoord/import_checker.go:70] ["start import checker"]
[2024/11/07 11:42:15.317 +00:00] [INFO] [datacoord/compaction_trigger_v2.go:114] ["Compaction trigger manager start"]
[2024/11/07 11:42:15.409 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=datacoord] [key=milvus-blainy/meta/session/datacoord]
[2024/11/07 11:42:15.425 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:15.425 +00:00] [INFO] [datanode/service.go:274] ["RootCoord client is ready for DataNode"]
[2024/11/07 11:42:15.425 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:15.426 +00:00] [INFO] [querycoord/service.go:190] ["QueryCoord report RootCoord ready"]
[2024/11/07 11:42:15.428 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:15.429 +00:00] [INFO] [querycoord/service.go:201] ["QueryCoord try to wait for DataCoord ready"]
[2024/11/07 11:42:15.432 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:15.432 +00:00] [INFO] [datanode/service.go:293] ["DataCoord client is ready for DataNode"]
[2024/11/07 11:42:15.432 +00:00] [INFO] [datanode/data_node.go:237] ["DataNode server initializing"] [TimeTickChannelName=milvus-blainy-datacoord-timetick-channel]
[2024/11/07 11:42:15.438 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:15.438 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:15.438 +00:00] [INFO] [querycoord/service.go:210] ["QueryCoord report DataCoord ready"]
[2024/11/07 11:42:15.440 +00:00] [INFO] [querycoordv2/server.go:190] ["QueryCoord start init"] [meta-root-path=milvus-blainy/meta] [address=172.18.0.2:19531]
[2024/11/07 11:42:15.441 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=datanode] [address=172.18.0.2:21124] [id=1]
[2024/11/07 11:42:15.442 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="datanode-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:15.442 +00:00] [INFO] [datanode/data_node.go:257] ["DataNode server init rateCollector done"] [role=datanode] [nodeID=1]
[2024/11/07 11:42:15.442 +00:00] [INFO] [datanode/data_node.go:260] ["DataNode server init dispatcher client done"] [role=datanode] [nodeID=1]
[2024/11/07 11:42:15.442 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:42:15.442 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:42:15.442 +00:00] [INFO] [datanode/data_node.go:271] ["DataNode server init succeeded"] [role=datanode] [nodeID=1] [MsgChannelSubName=milvus-blainy-dataNode]
[2024/11/07 11:42:15.442 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/11/07 11:42:15.442 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/11/07 11:42:15.442 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querycoord] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:114\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:585\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:15.444 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querycoord] [address=172.18.0.2:19531] [id=1]
[2024/11/07 11:42:15.444 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/11/07 11:42:15.444 +00:00] [INFO] [querycoordv2/server.go:222] ["start init querycoord"] [State=Initializing]
[2024/11/07 11:42:15.444 +00:00] [INFO] [querycoordv2/server.go:226] ["query coordinator connecting to etcd."]
[2024/11/07 11:42:15.444 +00:00] [INFO] [querycoordv2/server.go:238] ["query coordinator successfully connected to etcd."]
[2024/11/07 11:42:15.444 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/11/07 11:42:15.444 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:114\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:585\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:15.446 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:42:18.445 +00:00] [next=2024/11/07 11:42:15.445 +00:00]
[2024/11/07 11:42:15.446 +00:00] [INFO] [querycoordv2/server.go:249] ["init ID allocator done"]
[2024/11/07 11:42:15.446 +00:00] [INFO] [querycoordv2/server.go:352] ["init meta"]
[2024/11/07 11:42:15.446 +00:00] [INFO] [querycoordv2/server.go:361] ["recover meta..."]
[2024/11/07 11:42:15.447 +00:00] [INFO] [meta/collection_manager.go:150] ["recover collections and partitions from kv store"] [traceID=1730979735447722433]
[2024/11/07 11:42:15.447 +00:00] [INFO] [querycoordv2/server.go:368] ["recovering collections..."] [collections="[]"]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:398] ["QueryCoord server initMeta done"] [duration=3.139604ms]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:261] ["init session"]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:265] ["init schedulers"]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:285] ["init proxy manager done"]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:288] ["init dist controller"]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:298] ["init checker controller"]
[2024/11/07 11:42:15.449 +00:00] [INFO] [querycoordv2/server.go:403] ["init observers"]
[2024/11/07 11:42:15.450 +00:00] [INFO] [querycoordv2/server.go:345] ["init querycoord done"] [nodeID=1] [Address=172.18.0.2:19531]
[2024/11/07 11:42:15.450 +00:00] [INFO] [querycoord/service.go:116] ["QueryCoord init done ..."]
[2024/11/07 11:42:15.454 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/querycoord] [value="{\"ServerID\":1,\"ServerName\":\"querycoord\",\"Address\":\"172.18.0.2:19531\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319864,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:15.454 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=querycoord] [serverID=1]
[2024/11/07 11:42:15.455 +00:00] [INFO] [querycoordv2/server.go:446] ["start watcher..."]
[2024/11/07 11:42:15.456 +00:00] [INFO] [querycoordv2/server.go:843] ["all old query node down, enable auto balance!"]
[2024/11/07 11:42:15.457 +00:00] [INFO] [proxyutil/proxy_watcher.go:96] ["succeed to init sessions on etcd"] [sessions=null] [revision=20]
[2024/11/07 11:42:15.458 +00:00] [INFO] [querycoordv2/server.go:499] ["start cluster..."]
[2024/11/07 11:42:15.458 +00:00] [INFO] [querycoordv2/server.go:502] ["start observers..."]
[2024/11/07 11:42:15.458 +00:00] [INFO] [observers/target_observer.go:150] ["Start update next target loop"]
[2024/11/07 11:42:15.458 +00:00] [INFO] [proxyutil/proxy_watcher.go:119] ["start to watch etcd"]
[2024/11/07 11:42:15.458 +00:00] [INFO] [observers/target_observer.go:164] ["target observer init done"]
[2024/11/07 11:42:15.458 +00:00] [INFO] [observers/replica_observer.go:72] ["Start check replica loop"]
[2024/11/07 11:42:15.458 +00:00] [INFO] [querycoordv2/server.go:508] ["start task scheduler..."]
[2024/11/07 11:42:15.459 +00:00] [INFO] [querycoordv2/server.go:511] ["start checker controller..."]
[2024/11/07 11:42:15.459 +00:00] [INFO] [querycoordv2/server.go:514] ["start job scheduler..."]
[2024/11/07 11:42:15.458 +00:00] [INFO] [observers/resource_observer.go:70] ["Start check resource group loop"]
[2024/11/07 11:42:15.459 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querycoord-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:15.459 +00:00] [INFO] [querycoordv2/server.go:440] ["QueryCoord started"]
[2024/11/07 11:42:15.459 +00:00] [INFO] [querycoord/service.go:121] ["QueryCoord start done ..."]
[2024/11/07 11:42:15.510 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=querycoord] [key=milvus-blainy/meta/session/querycoord]
[2024/11/07 11:42:15.645 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querycoord] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:114\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:585\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:42:15.675 +00:00] [INFO] [querynodev2/server.go:364] ["query node init successfully"] [queryNodeID=1] [Address=172.18.0.2:21123]
[2024/11/07 11:42:15.675 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:213] ["start execute loop"]
[2024/11/07 11:42:15.676 +00:00] [INFO] [querynodev2/server.go:391] ["query node start successfully"] [queryNodeID=1] [Address=172.18.0.2:21123] [mmapEnabled=false] [growingmmapEnable=false] [mmapVectorIndex=false] [mmapVectorField=false] [mmapScalarIndex=false] [mmapScalarField=false] [mmapChunkCache=true]
[2024/11/07 11:42:15.681 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/querynode-1] [value="{\"ServerID\":1,\"ServerName\":\"querynode\",\"Address\":\"172.18.0.2:21123\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{\"CurrentIndexVersion\":5},\"LeaseID\":7587882603024319872,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:15.681 +00:00] [INFO] [querycoordv2/server.go:687] ["add node to NodeManager"] [nodeID=1] [nodeAddr=172.18.0.2:21123]
[2024/11/07 11:42:15.681 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=querynode] [serverID=1]
[2024/11/07 11:42:15.681 +00:00] [INFO] [datacoord/server.go:989] ["received querynode register"] [address=172.18.0.2:21123] [serverID=1]
[2024/11/07 11:42:15.682 +00:00] [INFO] [datacoord/index_engine_version_manager.go:65] ["addOrUpdate version"] [nodeId=1] [minimal=0] [current=5]
[2024/11/07 11:42:15.685 +00:00] [INFO] [task/scheduler.go:240] ["add executor for new QueryNode"] [nodeID=1]
[2024/11/07 11:42:15.686 +00:00] [INFO] [dist/dist_handler.go:59] ["start dist handler"] [nodeID=1]
[2024/11/07 11:42:15.693 +00:00] [INFO] [meta/resource_manager.go:821] ["transfer node to resource group"] [rgName=__default_resource_group] [originalRG=_] [node=1]
[2024/11/07 11:42:15.693 +00:00] [INFO] [meta/resource_manager.go:460] ["HandleNodeUp: add node to resource group"] [rgName=__default_resource_group] [node=1] []
[2024/11/07 11:42:15.712 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=querynode] [key=milvus-blainy/meta/session/querynode-1]
[2024/11/07 11:42:15.871 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-blainy-prod] [root=files]
[2024/11/07 11:42:15.871 +00:00] [INFO] [syncmgr/sync_manager.go:66] ["sync manager initialized"] [initPoolSize=256]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/data_node.go:302] ["init datanode done"] [role=datanode] [nodeID=1] [Address=172.18.0.2:21124]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/service.go:305] ["current DataNode state"] [state=Initializing]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/service.go:188] ["DataNode gRPC services successfully initialized"]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/data_node.go:365] ["start id allocator done"] [role=datanode]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/data_node.go:344] ["DataNode Background GC Start"]
[2024/11/07 11:42:15.872 +00:00] [INFO] [importv2/scheduler.go:54] ["start import scheduler"]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/channel_checkpoint_updater.go:64] ["channel checkpoint updater start"]
[2024/11/07 11:42:15.872 +00:00] [INFO] [datanode/channel_manager.go:177] ["DataNode ChannelManager start"]
[2024/11/07 11:42:15.879 +00:00] [INFO] [datacoord/server.go:938] ["received datanode register"] [address=172.18.0.2:21124] [serverID=1]
[2024/11/07 11:42:15.879 +00:00] [INFO] [datacoord/channel_manager_v2.go:191] ["register node"] ["registered node"=1]
[2024/11/07 11:42:15.879 +00:00] [INFO] [datacoord/channel_manager_v2.go:197] ["register node with no reassignment"] ["registered node"=1]
[2024/11/07 11:42:15.879 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/datanode-1] [value="{\"ServerID\":1,\"ServerName\":\"datanode\",\"Address\":\"172.18.0.2:21124\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319878,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:15.879 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=datanode] [serverID=1]
[2024/11/07 11:42:15.879 +00:00] [INFO] [datanode/data_node.go:196] ["DataNode Register Finished"]
[2024/11/07 11:42:15.882 +00:00] [INFO] [datanode/service.go:193] ["DataNode gRPC services successfully started"]
[2024/11/07 11:42:15.913 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=datanode] [key=milvus-blainy/meta/session/datanode-1]
[2024/11/07 11:42:16.050 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:42:16.050 +00:00] [INFO] [proxy/proxy.go:214] ["init session for Proxy"]
[2024/11/07 11:42:16.053 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=proxy] [address=172.18.0.2:19529] [id=1]
[2024/11/07 11:42:16.053 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="proxy-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:42:16.053 +00:00] [INFO] [proxy/proxy.go:219] ["init session for Proxy done"]
[2024/11/07 11:42:16.053 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:42:16.053 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:42:16.053 +00:00] [INFO] [proxy/proxy.go:229] ["Proxy init rateCollector done"] [nodeID=1]
[2024/11/07 11:42:16.054 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:42:16.059 +00:00] [INFO] [proxy/meta_cache.go:380] ["success to init meta cache"] [policy_infos="[\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Collection-*.*\\\",\\\"V2\\\":\\\"PrivilegeIndexDetail\\\"}\",\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Global-*.*\\\",\\\"V2\\\":\\\"PrivilegeDescribeCollection\\\"}\"]"]
[2024/11/07 11:42:16.060 +00:00] [INFO] [proxy/proxy.go:301] ["init proxy done"] [nodeID=1] [Address=172.18.0.2:19529]
[2024/11/07 11:42:16.060 +00:00] [INFO] [proxy/service.go:465] ["init Proxy server done"]
[2024/11/07 11:42:16.060 +00:00] [INFO] [proxy/service.go:467] ["start Proxy server"]
[2024/11/07 11:42:16.065 +00:00] [INFO] [rootcoord/timeticksync.go:233] ["Add session for timeticksync"] [serverID=1]
[2024/11/07 11:42:16.065 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/proxy-1] [value="{\"ServerID\":1,\"ServerName\":\"proxy\",\"Address\":\"172.18.0.2:19529\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603024319888,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:42:16.065 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=proxy] [serverID=1]
[2024/11/07 11:42:16.065 +00:00] [INFO] [proxy/proxy.go:175] ["Proxy Register Finished"]
[2024/11/07 11:42:16.066 +00:00] [INFO] [proxyutil/proxy_client_manager.go:156] ["succeed to create proxy client"] [address=172.18.0.2:19529] [serverID=1]
[2024/11/07 11:42:16.066 +00:00] [INFO] [proxyutil/proxy_client_manager.go:156] ["succeed to create proxy client"] [address=172.18.0.2:19529] [serverID=1]
[2024/11/07 11:42:16.067 +00:00] [INFO] [proxy/service.go:623] ["start Proxy http server"]
[2024/11/07 11:42:16.067 +00:00] [INFO] [proxy/service.go:472] ["start Proxy server done"]
[2024/11/07 11:42:16.067 +00:00] [INFO] [components/proxy.go:62] ["Proxy successfully started"]
[2024/11/07 11:42:16.068 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]
[2024/11/07 11:42:16.115 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=proxy] [key=milvus-blainy/meta/session/proxy-1]
[2024/11/07 11:42:16.189 +00:00] [WARN] [dist/dist_handler.go:112] ["node last heart beat time lag too behind"] [now=2024/11/07 11:42:16.189 +00:00] [lastHeartBeatTime=1970/01/01 00:00:00.000 +00:00] [nodeID=1]
[2024/11/07 11:42:23.218 +00:00] [ERROR] [proxy/util.go:1006] ["Verify password failed"] [error="crypto/bcrypt: hashedSecret too short to be a bcrypted password"] [stack="github.com/milvus-io/milvus/internal/proxy.passwordVerify\n\t/workspace/source/internal/proxy/util.go:1006\ngithub.com/milvus-io/milvus/internal/proxy.AuthenticationInterceptor\n\t/workspace/source/internal/proxy/authentication_interceptor.go:90\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:42\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n\t/workspace/source/internal/proxy/accesslog/util.go:38\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Connect_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.4.15/milvuspb/milvus_grpc.pb.go:2821\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"]
[2024/11/07 11:42:23.218 +00:00] [WARN] [proxy/authentication_interceptor.go:91] ["fail to verify password"] [username=root]
[2024/11/07 11:42:24.315 +00:00] [ERROR] [proxy/util.go:1006] ["Verify password failed"] [error="crypto/bcrypt: hashedSecret too short to be a bcrypted password"] [stack="github.com/milvus-io/milvus/internal/proxy.passwordVerify\n\t/workspace/source/internal/proxy/util.go:1006\ngithub.com/milvus-io/milvus/internal/proxy.AuthenticationInterceptor\n\t/workspace/source/internal/proxy/authentication_interceptor.go:90\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:42\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n\t/workspace/source/internal/proxy/accesslog/util.go:38\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Connect_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.4.15/milvuspb/milvus_grpc.pb.go:2821\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"]
[2024/11/07 11:42:24.315 +00:00] [WARN] [proxy/authentication_interceptor.go:91] ["fail to verify password"] [username=root]
[2024/11/07 11:42:53.777 +00:00] [INFO] [config/manager.go:379] ["receive update event"] [event="{\"EventSource\":\"FileSource\",\"EventType\":\"UPDATE\",\"Key\":\"common.security.authorizationenabled\",\"Value\":\"false\",\"HasUpdated\":false}"]
[2024/11/07 11:42:53.778 +00:00] [INFO] [config/manager.go:379] ["receive update event"] [event="{\"EventSource\":\"FileSource\",\"EventType\":\"UPDATE\",\"Key\":\"commonsecurityauthorizationenabled\",\"Value\":\"false\",\"HasUpdated\":false}"]
[2024/11/07 11:43:10.723 +00:00] [INFO] [proxy/impl.go:5966] ["connect received"] [traceID=539cddc0670afdfb2da2766c494716c9] [sdk_type=nodejs] [sdk_version=2.4.8] [local_time="2024-11-07 11:43:10.715"] [user=root] [host=] [db=default]
[2024/11/07 11:43:10.723 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=539cddc0670afdfb2da2766c494716c9] [msgID=0]
[2024/11/07 11:43:10.723 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=539cddc0670afdfb2da2766c494716c9] [msgID=0] ["num of databases"=1]
[2024/11/07 11:43:10.724 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=539cddc0670afdfb2da2766c494716c9] [sdk_type=nodejs] [sdk_version=2.4.8] [local_time="2024-11-07 11:43:10.715"] [user=root] [host=] [identifier=453765966257192962] [last_active_time=2024/11/07 11:43:10.724 +00:00]
[2024/11/07 11:43:10.756 +00:00] [INFO] [proxy/impl.go:368] ["ListDatabases received"] [traceID=29a7e93243782b0b1ab2f9c44ef81b18] [role=proxy]
[2024/11/07 11:43:10.757 +00:00] [INFO] [proxy/impl.go:379] ["ListDatabases enqueued"] [traceID=29a7e93243782b0b1ab2f9c44ef81b18] [role=proxy]
[2024/11/07 11:43:10.758 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=29a7e93243782b0b1ab2f9c44ef81b18] [msgID=453765966257192963]
[2024/11/07 11:43:10.758 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=29a7e93243782b0b1ab2f9c44ef81b18] [msgID=453765966257192963] ["num of databases"=1]
[2024/11/07 11:43:10.758 +00:00] [INFO] [proxy/impl.go:389] ["ListDatabases done"] [traceID=29a7e93243782b0b1ab2f9c44ef81b18] [role=proxy] ["num of db"=1]
[2024/11/07 11:43:12.816 +00:00] [INFO] [proxy/impl.go:368] ["ListDatabases received"] [traceID=96a23d4dae592c67544ee9a6b34cc68f] [role=proxy]
[2024/11/07 11:43:12.816 +00:00] [INFO] [proxy/impl.go:379] ["ListDatabases enqueued"] [traceID=96a23d4dae592c67544ee9a6b34cc68f] [role=proxy]
[2024/11/07 11:43:12.817 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=96a23d4dae592c67544ee9a6b34cc68f] [msgID=453765966807957505]
[2024/11/07 11:43:12.817 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=96a23d4dae592c67544ee9a6b34cc68f] [msgID=453765966807957505] ["num of databases"=1]
[2024/11/07 11:43:12.817 +00:00] [INFO] [proxy/impl.go:389] ["ListDatabases done"] [traceID=96a23d4dae592c67544ee9a6b34cc68f] [role=proxy] ["num of db"=1]
[2024/11/07 11:43:12.830 +00:00] [INFO] [proxy/impl.go:368] ["ListDatabases received"] [traceID=bbae9d36cff627ad2aecd9c06f99d7fb] [role=proxy]
[2024/11/07 11:43:12.831 +00:00] [INFO] [proxy/impl.go:379] ["ListDatabases enqueued"] [traceID=bbae9d36cff627ad2aecd9c06f99d7fb] [role=proxy]
[2024/11/07 11:43:12.832 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=bbae9d36cff627ad2aecd9c06f99d7fb] [msgID=453765966807957507]
[2024/11/07 11:43:12.832 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=bbae9d36cff627ad2aecd9c06f99d7fb] [msgID=453765966807957507] ["num of databases"=1]
[2024/11/07 11:43:12.832 +00:00] [INFO] [proxy/impl.go:389] ["ListDatabases done"] [traceID=bbae9d36cff627ad2aecd9c06f99d7fb] [role=proxy] ["num of db"=1]
[2024/11/07 11:43:12.850 +00:00] [INFO] [rootcoord/root_coord.go:2823] ["received request to describe database "] [traceID=35f26021bde0d47fd8432a82290b0205] [dbName=default]
[2024/11/07 11:43:12.850 +00:00] [INFO] [rootcoord/root_coord.go:2847] ["done to describe database"] [traceID=35f26021bde0d47fd8432a82290b0205] [dbName=default] [ts=453765966807957512]
[2024/11/07 11:43:13.084 +00:00] [INFO] [querycoordv2/services.go:59] ["show collections request received"] [traceID=0d94644cc256d3fbb3bfa3f2b231973d] [collections="[]"]
[2024/11/07 11:43:13.923 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55460->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.924 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #16 SubChannel #17] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55436->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.923 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #4 SubChannel #5] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55472->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.923 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #6 SubChannel #7] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55446->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.924 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #8 SubChannel #9] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55516->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.924 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #10 SubChannel #11] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55426->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.924 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #40 SubChannel #41] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55502->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.924 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55496->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:13.924 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #18 SubChannel #19] grpc: addrConn.createTransport failed to connect to {Addr: \"milvus-etcd:2379\", ServerName: \"milvus-etcd\", }. Err: connection error: desc = \"error reading server preface: read tcp 172.18.0.2:55480->172.18.0.3:2379: read: connection reset by peer\""]
[2024/11/07 11:43:14.937 +00:00] [WARN] [sessionutil/session_util.go:530] ["session keepalive channel closed"]
[2024/11/07 11:43:14.937 +00:00] [INFO] [sessionutil/session_util.go:538] ["keepAlive channel close caused by etcd, try to KeepAliveOnce"] [serverName=indexnode]
[2024/11/07 11:43:14.941 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:15.317 +00:00] [INFO] [datacoord/compaction_policy_l0.go:33] ["start trigger l0CompactionPolicy..."]
[2024/11/07 11:43:15.317 +00:00] [INFO] [datacoord/compaction_policy_single.go:49] ["start trigger singleCompactionPolicy..."]
[2024/11/07 11:43:15.317 +00:00] [INFO] [datacoord/compaction_trigger.go:323] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=453765951606227257] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
[2024/11/07 11:43:15.432 +00:00] [WARN] [sessionutil/session_util.go:530] ["session keepalive channel closed"]
[2024/11/07 11:43:15.432 +00:00] [INFO] [sessionutil/session_util.go:538] ["keepAlive channel close caused by etcd, try to KeepAliveOnce"] [serverName=rootcoord]
[2024/11/07 11:43:15.432 +00:00] [WARN] [sessionutil/session_util.go:530] ["session keepalive channel closed"]
[2024/11/07 11:43:15.432 +00:00] [INFO] [sessionutil/session_util.go:538] ["keepAlive channel close caused by etcd, try to KeepAliveOnce"] [serverName=indexcoord]
[2024/11/07 11:43:15.432 +00:00] [WARN] [sessionutil/session_util.go:530] ["session keepalive channel closed"]
[2024/11/07 11:43:15.432 +00:00] [INFO] [sessionutil/session_util.go:538] ["keepAlive channel close caused by etcd, try to KeepAliveOnce"] [serverName=datacoord]
[2024/11/07 11:43:15.433 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:15.433 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:15.433 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:15.933 +00:00] [WARN] [sessionutil/session_util.go:530] ["session keepalive channel closed"]
[2024/11/07 11:43:15.933 +00:00] [INFO] [sessionutil/session_util.go:538] ["keepAlive channel close caused by etcd, try to KeepAliveOnce"] [serverName=querynode]
[2024/11/07 11:43:15.933 +00:00] [WARN] [sessionutil/session_util.go:530] ["session keepalive channel closed"]
[2024/11/07 11:43:15.933 +00:00] [INFO] [sessionutil/session_util.go:538] ["keepAlive channel close caused by etcd, try to KeepAliveOnce"] [serverName=querycoord]
[2024/11/07 11:43:15.934 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:15.934 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:16.344 +00:00] [WARN] [sessionutil/session_util.go:553] ["fail to retry keepAliveOnce"] [serverName=indexnode] [LeaseID=7587882603024319772] [error="etcdserver: requested lease not found"]
[2024/11/07 11:43:16.344 +00:00] [WARN] [sessionutil/session_util.go:882] ["connection lost detected, shuting down"]
[2024/11/07 11:43:16.344 +00:00] [ERROR] [indexnode/indexnode.go:142] ["Index Node disconnected from etcd, process will exit"] ["Server Id"=1] [stack="github.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Register.func1\n\t/workspace/source/internal/indexnode/indexnode.go:142"]
[2024/11/07 11:43:16.978 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=/var/log/milvus]
[2024/11/07 11:43:16.978 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=standalone-0.log]
[2024/11/07 11:43:16.978 +00:00] [INFO] [roles/roles.go:282] [setupPrometheusHTTPServer]
[2024/11/07 11:43:16.979 +00:00] [INFO] [http/server.go:160] ["management listen"] [addr=:9091]
[2024/11/07 11:43:16.980 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=7294235443]
[2024/11/07 11:43:16.980 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2024/11/07 11:43:16.980 +00:00] [INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2024/11/07 11:43:16.980 +00:00] [INFO] [rootcoord/service.go:196] ["init params done.."]
[2024/11/07 11:43:16.981 +00:00] [INFO] [datacoord/service.go:96] ["DataCoord listen on"] [address="[::]:13333"] [port=13333]
[2024/11/07 11:43:16.981 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:16.981 +00:00] [INFO] [components/index_coord.go:42] ["IndexCoord running ..."]
[2024/11/07 11:43:16.982 +00:00] [INFO] [querycoord/service.go:106] ["QueryCoord listen on"] [address="[::]:19531"] [port=19531]
[2024/11/07 11:43:16.983 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:16.985 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:16.984 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]
[2024/11/07 11:43:16.986 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:16.990 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]
[2024/11/07 11:43:16.990 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:16.991 +00:00] [INFO] [rootcoord/service.go:216] ["etcd connect done ..."]
[2024/11/07 11:43:16.991 +00:00] [INFO] [rootcoord/service.go:276] ["start grpc "] [port=53100]
[2024/11/07 11:43:16.993 +00:00] [INFO] [indexnode/service.go:78] ["IndexNode listen on"] [address="[::]:21121"] [port=21121]
[2024/11/07 11:43:16.993 +00:00] [INFO] [proxy/look_aside_balancer.go:225] ["Start check query node health loop"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/listener_manager.go:35] ["Proxy listen on external grpc listener"] [address=172.18.0.2:19530] [port=19530]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/listener_manager.go:45] ["Proxy listen on internal grpc listener"] [address=172.18.0.2:19529] [port=19529]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/listener_manager.go:79] ["Proxy server(http) and external grpc server share the same port"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/service.go:460] ["init Proxy server"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/service.go:479] ["Proxy init service's parameter table done"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/service.go:481] ["Proxy init http server's parameter table done"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]
[2024/11/07 11:43:16.994 +00:00] [INFO] [proxy/service.go:485] ["init Proxy's tracer done"] ["service name"="Proxy ip: 172.18.0.2, port: 19530"]
[2024/11/07 11:43:16.995 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:16.995 +00:00] [INFO] [datanode/service.go:253] ["DataNode address"] [address=172.18.0.2:21124]
[2024/11/07 11:43:16.995 +00:00] [INFO] [datanode/service.go:254] ["DataNode serverID"] [serverID=0]
[2024/11/07 11:43:16.997 +00:00] [INFO] [proxy/service.go:437] ["create Proxy internal grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"]
[2024/11/07 11:43:16.997 +00:00] [INFO] [proxy/service.go:524] ["register Proxy http server"]
[2024/11/07 11:43:16.997 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:17.000 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:43:17.000 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.000 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.000 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:541\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.001 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:43:17.001 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.001 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.001 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:541\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.087 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:43:17.092 +00:00] [INFO] [rootcoord/service.go:233] ["grpc init done ..."]
[2024/11/07 11:43:17.092 +00:00] [INFO] [rootcoord/service.go:236] ["RootCoord start to create DataCoord client"]
[2024/11/07 11:43:17.093 +00:00] [INFO] [querynodev2/server.go:278] ["QueryNode session info"] [metaPath=milvus-blainy/meta]
[2024/11/07 11:43:17.093 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[milvus-etcd:2379]"] [minVersion=1.3]
[2024/11/07 11:43:17.094 +00:00] [INFO] [rootcoord/service.go:245] ["RootCoord start to create QueryCoord client"]
[2024/11/07 11:43:17.095 +00:00] [INFO] [datanode/service.go:263] ["initializing RootCoord client for DataNode"]
[2024/11/07 11:43:17.096 +00:00] [INFO] [querycoord/service.go:180] ["QueryCoord try to wait for RootCoord ready"]
[2024/11/07 11:43:17.097 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:43:17.098 +00:00] [INFO] [indexnode/indexnode.go:209] ["IndexNode init"] [state=Initializing]
[2024/11/07 11:43:17.098 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:43:17.098 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.098 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.098 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:270\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:184\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.098 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:43:17.099 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.099 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.099 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.100 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:43:17.100 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.100 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.100 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/07 11:43:17.100 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.100 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/07 11:43:17.100 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:270\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:184\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.100 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.100 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querynode] [address=172.18.0.2:21123] [id=1]
[2024/11/07 11:43:17.100 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querynode-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:17.100 +00:00] [INFO] [querynodev2/server.go:163] ["QueryNode init session"] [nodeID=1] ["node address"=172.18.0.2:21123]
[2024/11/07 11:43:17.101 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:43:17.102 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexnode] [address=172.18.0.2:21121] [id=1]
[2024/11/07 11:43:17.102 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="indexnode-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:17.102 +00:00] [INFO] [indexnode/indexnode.go:216] ["IndexNode init session successful"] [serverID=1]
[2024/11/07 11:43:17.103 +00:00] [INFO] [indexnode/indexnode.go:221] ["init index node done"] [nodeID=1] [Address=172.18.0.2:21121]
[2024/11/07 11:43:17.103 +00:00] [INFO] [indexnode/indexnode.go:232] [IndexNode] [State=Healthy]
[2024/11/07 11:43:17.103 +00:00] [INFO] [indexnode/indexnode.go:235] ["IndexNode start finished"] []
[2024/11/07 11:43:17.107 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/indexnode-1] [value="{\"ServerID\":1,\"ServerName\":\"indexnode\",\"Address\":\"172.18.0.2:21121\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782421,\"HostName\":\"378c7bb1eec5\",\"EnableDisk\":true}"]
[2024/11/07 11:43:17.108 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexnode] [serverID=1]
[2024/11/07 11:43:17.126 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=indexnode] [key=milvus-blainy/meta/session/indexnode-1]
[2024/11/07 11:43:17.162 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:43:17.162 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:43:17.162 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:43:17.166 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=rootcoord] [address=172.18.0.2:53100] [id=1]
[2024/11/07 11:43:17.166 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Initializing]
[2024/11/07 11:43:17.166 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexcoord] [address=172.18.0.2:13333] [id=1]
[2024/11/07 11:43:17.168 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=datacoord] [address=172.18.0.2:13333] [id=1]
[2024/11/07 11:43:17.169 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:43:20.167 +00:00] [next=2024/11/07 11:43:17.167 +00:00]
[2024/11/07 11:43:17.169 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=milvus-blainy/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/07 11:43:17.169 +00:00] [INFO] [datacoord/server.go:348] ["init rootcoord client done"]
[2024/11/07 11:43:17.171 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/11/07 11:43:17.916 +00:00] [save=2024/11/07 11:43:20.917 +00:00] [next=2024/11/07 11:43:17.917 +00:00]
[2024/11/07 11:43:17.171 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=milvus-blainy/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/07 11:43:17.171 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
[2024/11/07 11:43:17.172 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=0]
[2024/11/07 11:43:17.175 +00:00] [INFO] [rootcoord/meta_table.go:282] ["create database"] [db=default] [ts=453765968145154049]
[2024/11/07 11:43:17.178 +00:00] [INFO] [rootcoord/meta_table.go:193] ["collections recovered from db"] [db_name=default] [collection_num=0] [partition_num=0]
[2024/11/07 11:43:17.179 +00:00] [INFO] [rootcoord/meta_table.go:209] ["RootCoord meta table reload done"] [duration=8.070492ms]
[2024/11/07 11:43:17.179 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.180 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.181 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.181 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.181 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.181 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:17.181 +00:00] [INFO] [rootcoord/dml_channels.go:215] ["init dml channels"] [prefix=milvus-blainy-rootcoord-dml] [num=16]
[2024/11/07 11:43:17.181 +00:00] [INFO] [rootcoord/root_coord.go:451] ["create TimeTick sync done"]
[2024/11/07 11:43:17.181 +00:00] [INFO] [rootcoord/root_coord.go:467] ["init proxy manager done"]
[2024/11/07 11:43:17.184 +00:00] [INFO] [rootcoord/root_coord.go:477] ["init credentials done"]
[2024/11/07 11:43:17.190 +00:00] [WARN] [rootcoord/kv_catalog.go:1026] ["fail to load grant privilege entity"] [key=root-coord/credential/grantee-privileges/public/Global/*.*] [type=Grant] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-privileges/public/Global/*.*]"]
[2024/11/07 11:43:17.192 +00:00] [WARN] [rootcoord/kv_catalog.go:1048] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-id/250dd41b686083b0/PrivilegeDescribeCollection]"]
[2024/11/07 11:43:17.194 +00:00] [WARN] [rootcoord/kv_catalog.go:1026] ["fail to load grant privilege entity"] [key=root-coord/credential/grantee-privileges/public/Collection/*.*] [type=Grant] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-privileges/public/Collection/*.*]"]
[2024/11/07 11:43:17.197 +00:00] [WARN] [rootcoord/kv_catalog.go:1048] ["fail to load the grantee id"] [key=root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail] [error="key not found[key=milvus-blainy/meta/root-coord/credential/grantee-id/e03326696e8a3b16/PrivilegeIndexDetail]"]
[2024/11/07 11:43:17.199 +00:00] [INFO] [rootcoord/root_coord.go:483] ["init rootcoord done"] [nodeID=1] [Address=172.18.0.2:53100]
[2024/11/07 11:43:17.199 +00:00] [INFO] [rootcoord/service.go:182] ["RootCoord init done ..."]
[2024/11/07 11:43:17.199 +00:00] [INFO] [rootcoord/service.go:316] ["RootCoord Core start ..."]
[2024/11/07 11:43:17.202 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:541\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.204 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/rootcoord] [value="{\"ServerID\":1,\"ServerName\":\"rootcoord\",\"Address\":\"172.18.0.2:53100\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782465,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:17.204 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=rootcoord] [serverID=1]
[2024/11/07 11:43:17.204 +00:00] [INFO] [rootcoord/root_coord.go:273] ["RootCoord Register Finished"]
[2024/11/07 11:43:17.206 +00:00] [INFO] [proxyutil/proxy_watcher.go:96] ["succeed to init sessions on etcd"] [sessions=null] [revision=16]
[2024/11/07 11:43:17.207 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Healthy]
[2024/11/07 11:43:17.207 +00:00] [INFO] [proxyutil/proxy_watcher.go:119] ["start to watch etcd"]
[2024/11/07 11:43:17.207 +00:00] [WARN] [proxyutil/proxy_client_manager.go:263] ["proxy client is empty, RefreshPrivilegeInfoCache will not send to any client"]
[2024/11/07 11:43:17.207 +00:00] [INFO] [rootcoord/quota_center.go:310] ["Start QuotaCenter"] [collectInterval=3s]
[2024/11/07 11:43:17.207 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="rootcoord-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:17.207 +00:00] [INFO] [rootcoord/root_coord.go:713] ["rootcoord startup successfully"]
[2024/11/07 11:43:17.207 +00:00] [INFO] [rootcoord/service.go:187] ["RootCoord start done ..."]
[2024/11/07 11:43:17.208 +00:00] [INFO] [components/root_coord.go:62] ["RootCoord successfully started"]
[2024/11/07 11:43:17.226 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=rootcoord] [key=milvus-blainy/meta/session/rootcoord]
[2024/11/07 11:43:17.301 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:270\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:184\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.301 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.608 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:17.609 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/11/07 11:43:17.609 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/11/07 11:43:17.609 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:563\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.610 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-blainy-prod] [root=files]
[2024/11/07 11:43:17.610 +00:00] [INFO] [datacoord/server.go:357] ["init chunk manager factory done"]
[2024/11/07 11:43:17.610 +00:00] [INFO] [datacoord/server.go:644] ["data coordinator connecting to metadata store"] [metaType=etcd]
[2024/11/07 11:43:17.610 +00:00] [INFO] [datacoord/server.go:657] ["data coordinator successfully connected to metadata store"] [metaType=etcd]
[2024/11/07 11:43:17.611 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/11/07 11:43:17.611 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/11/07 11:43:17.611 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:563\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.611 +00:00] [INFO] [datacoord/index_meta.go:97] ["indexMeta reloadFromKV done"] [duration=1.370122ms]
[2024/11/07 11:43:17.612 +00:00] [INFO] [datacoord/analyze_meta.go:70] ["analyzeMeta reloadFromKV done"] [duration=538.201µs]
[2024/11/07 11:43:17.613 +00:00] [INFO] [datacoord/partition_stats_meta.go:67] ["DataCoord partitionStatsMeta reloadFromKV done"] [duration=703.401µs]
[2024/11/07 11:43:17.613 +00:00] [INFO] [datacoord/compaction_task_meta.go:62] ["DataCoord compactionTaskMeta reloadFromKV done"] [duration=645.841µs]
[2024/11/07 11:43:17.616 +00:00] [INFO] [datacoord/meta.go:229] ["DataCoord meta reloadFromKV done"] [duration=2.174523ms]
[2024/11/07 11:43:17.618 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:43:20.616 +00:00] [next=2024/11/07 11:43:17.616 +00:00]
[2024/11/07 11:43:17.618 +00:00] [INFO] [datacoord/server.go:1257] ["all old data node down, enable auto balance!"]
[2024/11/07 11:43:17.619 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [msgID=0]
[2024/11/07 11:43:17.619 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [msgID=0] ["num of databases"=1]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/channel_store_v2.go:71] ["channel store reload done"] [duration=896.401µs]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/server.go:379] ["init datanode cluster done"]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/server.go:561] ["DataCoord success to get DataNode sessions"] [sessions={}]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/server.go:582] ["DataCoord Cluster Manager start up"]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/channel_manager_v2.go:163] ["starting channel balance loop"]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/channel_manager_v2.go:171] ["cluster start up"] [allNodes="[]"] [legacyNodes="[]"] [oldNodes="[]"] [newOnlines="[]"] [offLines="[]"]
[2024/11/07 11:43:17.620 +00:00] [INFO] [datacoord/server.go:587] ["DataCoord Cluster Manager start up successfully"]
[2024/11/07 11:43:17.622 +00:00] [INFO] [datacoord/server.go:386] ["init service discovery done"]
[2024/11/07 11:43:17.622 +00:00] [INFO] [datacoord/server.go:390] ["init compaction done"]
[2024/11/07 11:43:17.623 +00:00] [INFO] [datacoord/server.go:395] ["init segment manager done"]
[2024/11/07 11:43:17.623 +00:00] [INFO] [datacoord/garbage_collector.go:85] ["GC with option"] [enabled=true] [interval=1h0m0s] [scanInterval=168h0m0s] [missingTolerance=24h0m0s] [dropTolerance=3h0m0s]
[2024/11/07 11:43:17.624 +00:00] [INFO] [datacoord/server.go:410] ["init datacoord done"] [nodeID=1] [Address=172.18.0.2:13333]
[2024/11/07 11:43:17.628 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/indexcoord] [value="{\"ServerID\":1,\"ServerName\":\"indexcoord\",\"Address\":\"172.18.0.2:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782498,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:17.628 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexcoord] [serverID=1]
[2024/11/07 11:43:17.633 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/datacoord] [value="{\"ServerID\":1,\"ServerName\":\"datacoord\",\"Address\":\"172.18.0.2:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782501,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:17.633 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=datacoord] [serverID=1]
[2024/11/07 11:43:17.633 +00:00] [INFO] [datacoord/server.go:270] ["DataCoord Register Finished"]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/task_scheduler.go:168] ["task scheduler loop start"]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/compaction.go:341] ["compactionPlanHandler start loop schedule"]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/index_service.go:113] ["start create index for segment loop..."]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/sync_segments_scheduler.go:77] ["SyncSegmentsScheduler started..."]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/compaction.go:360] ["compactionPlanHandler start loop check"] ["check result interval"=3s]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/import_scheduler.go:75] ["start import scheduler"]
[2024/11/07 11:43:17.635 +00:00] [INFO] [datacoord/compaction_trigger_v2.go:114] ["Compaction trigger manager start"]
[2024/11/07 11:43:17.634 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="datacoord-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:17.635 +00:00] [INFO] [datacoord/server.go:424] ["DataCoord startup successfully"]
[2024/11/07 11:43:17.634 +00:00] [INFO] [datacoord/compaction.go:380] ["compactionPlanHandler start clean check loop"] ["gc interval"=30m0s]
[2024/11/07 11:43:17.635 +00:00] [INFO] [datacoord/import_checker.go:70] ["start import checker"]
[2024/11/07 11:43:17.704 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:17.704 +00:00] [INFO] [querycoord/service.go:190] ["QueryCoord report RootCoord ready"]
[2024/11/07 11:43:17.704 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:17.704 +00:00] [INFO] [datanode/service.go:274] ["RootCoord client is ready for DataNode"]
[2024/11/07 11:43:17.706 +00:00] [INFO] [querycoord/service.go:201] ["QueryCoord try to wait for DataCoord ready"]
[2024/11/07 11:43:17.709 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:17.709 +00:00] [INFO] [querycoord/service.go:210] ["QueryCoord report DataCoord ready"]
[2024/11/07 11:43:17.710 +00:00] [INFO] [querycoordv2/server.go:190] ["QueryCoord start init"] [meta-root-path=milvus-blainy/meta] [address=172.18.0.2:19531]
[2024/11/07 11:43:17.709 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:17.710 +00:00] [INFO] [datanode/service.go:293] ["DataCoord client is ready for DataNode"]
[2024/11/07 11:43:17.710 +00:00] [INFO] [datanode/data_node.go:237] ["DataNode server initializing"] [TimeTickChannelName=milvus-blainy-datacoord-timetick-channel]
[2024/11/07 11:43:17.713 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=datanode] [address=172.18.0.2:21124] [id=1]
[2024/11/07 11:43:17.713 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="datanode-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:17.713 +00:00] [INFO] [datanode/data_node.go:257] ["DataNode server init rateCollector done"] [role=datanode] [nodeID=1]
[2024/11/07 11:43:17.713 +00:00] [INFO] [datanode/data_node.go:260] ["DataNode server init dispatcher client done"] [role=datanode] [nodeID=1]
[2024/11/07 11:43:17.713 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:43:17.713 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:43:17.713 +00:00] [INFO] [datanode/data_node.go:271] ["DataNode server init succeeded"] [role=datanode] [nodeID=1] [MsgChannelSubName=milvus-blainy-dataNode]
[2024/11/07 11:43:17.713 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querycoord] [address=172.18.0.2:19531] [id=1]
[2024/11/07 11:43:17.714 +00:00] [INFO] [querycoordv2/server.go:222] ["start init querycoord"] [State=Initializing]
[2024/11/07 11:43:17.714 +00:00] [INFO] [querycoordv2/server.go:226] ["query coordinator connecting to etcd."]
[2024/11/07 11:43:17.714 +00:00] [INFO] [querycoordv2/server.go:238] ["query coordinator successfully connected to etcd."]
[2024/11/07 11:43:17.717 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=0001/01/01 00:00:00.000 +00:00] [save=2024/11/07 11:43:20.715 +00:00] [next=2024/11/07 11:43:17.715 +00:00]
[2024/11/07 11:43:17.717 +00:00] [INFO] [querycoordv2/server.go:249] ["init ID allocator done"]
[2024/11/07 11:43:17.717 +00:00] [INFO] [querycoordv2/server.go:352] ["init meta"]
[2024/11/07 11:43:17.717 +00:00] [INFO] [querycoordv2/server.go:361] ["recover meta..."]
[2024/11/07 11:43:17.719 +00:00] [INFO] [meta/collection_manager.go:150] ["recover collections and partitions from kv store"] [traceID=1730979797719019243]
[2024/11/07 11:43:17.719 +00:00] [INFO] [querycoordv2/server.go:368] ["recovering collections..."] [collections="[]"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:398] ["QueryCoord server initMeta done"] [duration=3.743885ms]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:261] ["init session"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:265] ["init schedulers"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:285] ["init proxy manager done"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:288] ["init dist controller"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:298] ["init checker controller"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:403] ["init observers"]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoordv2/server.go:345] ["init querycoord done"] [nodeID=1] [Address=172.18.0.2:19531]
[2024/11/07 11:43:17.721 +00:00] [INFO] [querycoord/service.go:116] ["QueryCoord init done ..."]
[2024/11/07 11:43:17.724 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/querycoord] [value="{\"ServerID\":1,\"ServerName\":\"querycoord\",\"Address\":\"172.18.0.2:19531\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782521,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:17.724 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=querycoord] [serverID=1]
[2024/11/07 11:43:17.725 +00:00] [INFO] [querycoordv2/server.go:446] ["start watcher..."]
[2024/11/07 11:43:17.726 +00:00] [INFO] [querycoordv2/server.go:843] ["all old query node down, enable auto balance!"]
[2024/11/07 11:43:17.727 +00:00] [INFO] [proxyutil/proxy_watcher.go:96] ["succeed to init sessions on etcd"] [sessions=null] [revision=21]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoordv2/server.go:499] ["start cluster..."]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoordv2/server.go:502] ["start observers..."]
[2024/11/07 11:43:17.728 +00:00] [INFO] [proxyutil/proxy_watcher.go:119] ["start to watch etcd"]
[2024/11/07 11:43:17.728 +00:00] [INFO] [observers/target_observer.go:150] ["Start update next target loop"]
[2024/11/07 11:43:17.728 +00:00] [INFO] [observers/target_observer.go:164] ["target observer init done"]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoordv2/server.go:508] ["start task scheduler..."]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoordv2/server.go:511] ["start checker controller..."]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoordv2/server.go:514] ["start job scheduler..."]
[2024/11/07 11:43:17.728 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querycoord-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoordv2/server.go:440] ["QueryCoord started"]
[2024/11/07 11:43:17.728 +00:00] [INFO] [querycoord/service.go:121] ["QueryCoord start done ..."]
[2024/11/07 11:43:17.728 +00:00] [INFO] [observers/resource_observer.go:70] ["Start check resource group loop"]
[2024/11/07 11:43:17.728 +00:00] [INFO] [observers/replica_observer.go:72] ["Start check replica loop"]
[2024/11/07 11:43:17.729 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=datacoord] [key=milvus-blainy/meta/session/datacoord]
[2024/11/07 11:43:17.729 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=querycoord] [key=milvus-blainy/meta/session/querycoord]
[2024/11/07 11:43:17.811 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/datacoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:563\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:461\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:58\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_arm64.s:1197\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/07 11:43:17.856 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-blainy-prod] [root=files]
[2024/11/07 11:43:17.856 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=4]
[2024/11/07 11:43:17.856 +00:00] [INFO] [querynodev2/server.go:319] ["queryNode init scheduler"] [policy=fifo]
[2024/11/07 11:43:17.856 +00:00] [INFO] [segments/segment_loader.go:566] ["SegmentLoader created"] [ioPoolSize=32]
[2024/11/07 11:43:17.857 +00:00] [INFO] [querynodev2/server.go:230] ["set up knowhere build pool size"] [pool_size=2]
[2024/11/07 11:43:18.138 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-blainy-prod] [root=files]
[2024/11/07 11:43:18.138 +00:00] [INFO] [syncmgr/sync_manager.go:66] ["sync manager initialized"] [initPoolSize=256]
[2024/11/07 11:43:18.138 +00:00] [INFO] [datanode/data_node.go:302] ["init datanode done"] [role=datanode] [nodeID=1] [Address=172.18.0.2:21124]
[2024/11/07 11:43:18.138 +00:00] [INFO] [datanode/service.go:305] ["current DataNode state"] [state=Initializing]
[2024/11/07 11:43:18.138 +00:00] [INFO] [datanode/service.go:188] ["DataNode gRPC services successfully initialized"]
[2024/11/07 11:43:18.138 +00:00] [INFO] [datanode/data_node.go:365] ["start id allocator done"] [role=datanode]
[2024/11/07 11:43:18.139 +00:00] [INFO] [datanode/data_node.go:344] ["DataNode Background GC Start"]
[2024/11/07 11:43:18.139 +00:00] [INFO] [datanode/channel_checkpoint_updater.go:64] ["channel checkpoint updater start"]
[2024/11/07 11:43:18.139 +00:00] [INFO] [importv2/scheduler.go:54] ["start import scheduler"]
[2024/11/07 11:43:18.139 +00:00] [INFO] [datanode/channel_manager.go:177] ["DataNode ChannelManager start"]
[2024/11/07 11:43:18.143 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/datanode-1] [value="{\"ServerID\":1,\"ServerName\":\"datanode\",\"Address\":\"172.18.0.2:21124\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782529,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:18.143 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=datanode] [serverID=1]
[2024/11/07 11:43:18.143 +00:00] [INFO] [datacoord/server.go:938] ["received datanode register"] [address=172.18.0.2:21124] [serverID=1]
[2024/11/07 11:43:18.143 +00:00] [INFO] [datacoord/channel_manager_v2.go:191] ["register node"] ["registered node"=1]
[2024/11/07 11:43:18.143 +00:00] [INFO] [datacoord/channel_manager_v2.go:197] ["register node with no reassignment"] ["registered node"=1]
[2024/11/07 11:43:18.143 +00:00] [INFO] [datanode/data_node.go:196] ["DataNode Register Finished"]
[2024/11/07 11:43:18.144 +00:00] [INFO] [datanode/service.go:193] ["DataNode gRPC services successfully started"]
[2024/11/07 11:43:18.215 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:18.218 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/11/07 11:43:18.218 +00:00] [INFO] [proxy/proxy.go:214] ["init session for Proxy"]
[2024/11/07 11:43:18.222 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=proxy] [address=172.18.0.2:19529] [id=1]
[2024/11/07 11:43:18.222 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="proxy-1\n"] [filePath=/tmp/milvus/server_id_8]
[2024/11/07 11:43:18.222 +00:00] [INFO] [proxy/proxy.go:219] ["init session for Proxy done"]
[2024/11/07 11:43:18.222 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2024/11/07 11:43:18.222 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2024/11/07 11:43:18.222 +00:00] [INFO] [proxy/proxy.go:229] ["Proxy init rateCollector done"] [nodeID=1]
[2024/11/07 11:43:18.222 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/07 11:43:18.228 +00:00] [INFO] [proxy/meta_cache.go:380] ["success to init meta cache"] [policy_infos="[\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Collection-*.*\\\",\\\"V2\\\":\\\"PrivilegeIndexDetail\\\"}\",\"{\\\"PType\\\":\\\"p\\\",\\\"V0\\\":\\\"public\\\",\\\"V1\\\":\\\"Global-*.*\\\",\\\"V2\\\":\\\"PrivilegeDescribeCollection\\\"}\"]"]
[2024/11/07 11:43:18.228 +00:00] [INFO] [proxy/proxy.go:301] ["init proxy done"] [nodeID=1] [Address=172.18.0.2:19529]
[2024/11/07 11:43:18.228 +00:00] [INFO] [proxy/service.go:465] ["init Proxy server done"]
[2024/11/07 11:43:18.228 +00:00] [INFO] [proxy/service.go:467] ["start Proxy server"]
[2024/11/07 11:43:18.232 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=datanode] [key=milvus-blainy/meta/session/datanode-1]
[2024/11/07 11:43:18.232 +00:00] [INFO] [rootcoord/timeticksync.go:233] ["Add session for timeticksync"] [serverID=1]
[2024/11/07 11:43:18.232 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/proxy-1] [value="{\"ServerID\":1,\"ServerName\":\"proxy\",\"Address\":\"172.18.0.2:19529\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{},\"LeaseID\":7587882603039782541,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:18.232 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=proxy] [serverID=1]
[2024/11/07 11:43:18.232 +00:00] [INFO] [proxy/proxy.go:175] ["Proxy Register Finished"]
[2024/11/07 11:43:18.233 +00:00] [INFO] [proxyutil/proxy_client_manager.go:156] ["succeed to create proxy client"] [address=172.18.0.2:19529] [serverID=1]
[2024/11/07 11:43:18.233 +00:00] [INFO] [proxyutil/proxy_client_manager.go:156] ["succeed to create proxy client"] [address=172.18.0.2:19529] [serverID=1]
[2024/11/07 11:43:18.234 +00:00] [INFO] [proxy/service.go:623] ["start Proxy http server"]
[2024/11/07 11:43:18.234 +00:00] [INFO] [proxy/service.go:472] ["start Proxy server done"]
[2024/11/07 11:43:18.234 +00:00] [INFO] [components/proxy.go:62] ["Proxy successfully started"]
[2024/11/07 11:43:18.237 +00:00] [INFO] [querynodev2/server.go:364] ["query node init successfully"] [queryNodeID=1] [Address=172.18.0.2:21123]
[2024/11/07 11:43:18.237 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:213] ["start execute loop"]
[2024/11/07 11:43:18.237 +00:00] [INFO] [querynodev2/server.go:391] ["query node start successfully"] [queryNodeID=1] [Address=172.18.0.2:21123] [mmapEnabled=false] [growingmmapEnable=false] [mmapVectorIndex=false] [mmapVectorField=false] [mmapScalarIndex=false] [mmapScalarField=false] [mmapChunkCache=true]
[2024/11/07 11:43:18.241 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=milvus-blainy/meta/session/querynode-1] [value="{\"ServerID\":1,\"ServerName\":\"querynode\",\"Address\":\"172.18.0.2:21123\",\"TriggerKill\":true,\"Version\":\"2.4.15\",\"IndexEngineVersion\":{\"CurrentIndexVersion\":5},\"LeaseID\":7587882603039782546,\"HostName\":\"378c7bb1eec5\"}"]
[2024/11/07 11:43:18.241 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=querynode] [serverID=1]
[2024/11/07 11:43:18.241 +00:00] [INFO] [querycoordv2/server.go:687] ["add node to NodeManager"] [nodeID=1] [nodeAddr=172.18.0.2:21123]
[2024/11/07 11:43:18.241 +00:00] [INFO] [datacoord/server.go:989] ["received querynode register"] [address=172.18.0.2:21123] [serverID=1]
[2024/11/07 11:43:18.241 +00:00] [INFO] [datacoord/index_engine_version_manager.go:65] ["addOrUpdate version"] [nodeId=1] [minimal=0] [current=5]
[2024/11/07 11:43:18.242 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]
[2024/11/07 11:43:18.244 +00:00] [INFO] [task/scheduler.go:240] ["add executor for new QueryNode"] [nodeID=1]
[2024/11/07 11:43:18.244 +00:00] [INFO] [dist/dist_handler.go:59] ["start dist handler"] [nodeID=1]
[2024/11/07 11:43:18.247 +00:00] [INFO] [meta/resource_manager.go:821] ["transfer node to resource group"] [rgName=__default_resource_group] [originalRG=_] [node=1]
[2024/11/07 11:43:18.247 +00:00] [INFO] [meta/resource_manager.go:460] ["HandleNodeUp: add node to resource group"] [rgName=__default_resource_group] [node=1] []
[2024/11/07 11:43:18.333 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=querynode] [key=milvus-blainy/meta/session/querynode-1]
[2024/11/07 11:43:18.333 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=proxy] [key=milvus-blainy/meta/session/proxy-1]
[2024/11/07 11:43:18.747 +00:00] [WARN] [dist/dist_handler.go:112] ["node last heart beat time lag too behind"] [now=2024/11/07 11:43:18.747 +00:00] [lastHeartBeatTime=1970/01/01 00:00:00.000 +00:00] [nodeID=1]
[2024/11/07 11:43:22.389 +00:00] [INFO] [proxy/impl.go:5966] ["connect received"] [traceID=ae743a28605176d5a171e38d051f4b58] [sdk_type=nodejs] [sdk_version=2.4.8] [local_time="2024-11-07 11:43:22.381"] [user=root] [host=] [db=default]
[2024/11/07 11:43:22.389 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=ae743a28605176d5a171e38d051f4b58] [msgID=0]
[2024/11/07 11:43:22.389 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=ae743a28605176d5a171e38d051f4b58] [msgID=0] ["num of databases"=1]
[2024/11/07 11:43:22.390 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=ae743a28605176d5a171e38d051f4b58] [sdk_type=nodejs] [sdk_version=2.4.8] [local_time="2024-11-07 11:43:22.381"] [user=root] [host=] [identifier=453765969309335554] [last_active_time=2024/11/07 11:43:22.390 +00:00]
[2024/11/07 11:43:22.419 +00:00] [INFO] [proxy/impl.go:368] ["ListDatabases received"] [traceID=49550b2c9f59ccdd8e9dbd74e8e7e1ac] [role=proxy]
[2024/11/07 11:43:22.420 +00:00] [INFO] [proxy/impl.go:379] ["ListDatabases enqueued"] [traceID=49550b2c9f59ccdd8e9dbd74e8e7e1ac] [role=proxy]
[2024/11/07 11:43:22.420 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=49550b2c9f59ccdd8e9dbd74e8e7e1ac] [msgID=453765969322442755]
[2024/11/07 11:43:22.420 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=49550b2c9f59ccdd8e9dbd74e8e7e1ac] [msgID=453765969322442755] ["num of databases"=1]
[2024/11/07 11:43:22.421 +00:00] [INFO] [proxy/impl.go:389] ["ListDatabases done"] [traceID=49550b2c9f59ccdd8e9dbd74e8e7e1ac] [role=proxy] ["num of db"=1]
[2024/11/07 11:43:23.384 +00:00] [INFO] [proxy/impl.go:368] ["ListDatabases received"] [traceID=70ed3f0daa8cff3d34b0c469905c3e6b] [role=proxy]
[2024/11/07 11:43:23.385 +00:00] [INFO] [proxy/impl.go:379] ["ListDatabases enqueued"] [traceID=70ed3f0daa8cff3d34b0c469905c3e6b] [role=proxy]
[2024/11/07 11:43:23.386 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=70ed3f0daa8cff3d34b0c469905c3e6b] [msgID=453765969571479553]
[2024/11/07 11:43:23.386 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=70ed3f0daa8cff3d34b0c469905c3e6b] [msgID=453765969571479553] ["num of databases"=1]
[2024/11/07 11:43:23.386 +00:00] [INFO] [proxy/impl.go:389] ["ListDatabases done"] [traceID=70ed3f0daa8cff3d34b0c469905c3e6b] [role=proxy] ["num of db"=1]
[2024/11/07 11:43:23.389 +00:00] [INFO] [proxy/impl.go:368] ["ListDatabases received"] [traceID=becbf0b9891eb836e527f62719b06ae9] [role=proxy]
[2024/11/07 11:43:23.391 +00:00] [INFO] [proxy/impl.go:379] ["ListDatabases enqueued"] [traceID=becbf0b9891eb836e527f62719b06ae9] [role=proxy]
[2024/11/07 11:43:23.392 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [traceID=becbf0b9891eb836e527f62719b06ae9] [msgID=453765969571479555]
[2024/11/07 11:43:23.392 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [traceID=becbf0b9891eb836e527f62719b06ae9] [msgID=453765969571479555] ["num of databases"=1]
[2024/11/07 11:43:23.392 +00:00] [INFO] [proxy/impl.go:389] ["ListDatabases done"] [traceID=becbf0b9891eb836e527f62719b06ae9] [role=proxy] ["num of db"=1]
[2024/11/07 11:43:23.405 +00:00] [INFO] [rootcoord/root_coord.go:2823] ["received request to describe database "] [traceID=ffd25e802a90c665d749296e692f68c9] [dbName=default]
[2024/11/07 11:43:23.406 +00:00] [INFO] [rootcoord/root_coord.go:2847] ["done to describe database"] [traceID=ffd25e802a90c665d749296e692f68c9] [dbName=default] [ts=453765969571479560]
[2024/11/07 11:43:23.598 +00:00] [INFO] [querycoordv2/services.go:59] ["show collections request received"] [traceID=67517d61daae4e6b7bb00ce2c64f5f01] [collections="[]"]
[2024/11/07 11:43:41.913 +00:00] [INFO] [proxy/impl.go:4880] [UpdateCredential] [traceID=b6c9f424994db02e3c763c43a3cebe2a] [username=root] [role=proxy]
[2024/11/07 11:43:41.917 +00:00] [ERROR] [proxy/util.go:1006] ["Verify password failed"] [error="crypto/bcrypt: hashedSecret too short to be a bcrypted password"] [stack="github.com/milvus-io/milvus/internal/proxy.passwordVerify\n\t/workspace/source/internal/proxy/util.go:1006\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).UpdateCredential\n\t/workspace/source/internal/proxy/impl.go:4915\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).UpdateCredential\n\t/workspace/source/internal/distributed/proxy/service.go:1016\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_UpdateCredential_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.4.15/milvuspb/milvus_grpc.pb.go:2423\ngithub.com/milvus-io/milvus/internal/proxy/connection.KeepActiveInterceptor\n\t/workspace/source/internal/proxy/connection/util.go:60\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.TraceLogInterceptor\n\t/workspace/source/internal/proxy/trace_log_interceptor.go:36\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryUpdateAccessInfoInterceptor\n\t/workspace/source/internal/proxy/accesslog/util.go:47\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.RateLimitInterceptor.func5\n\t/workspace/source/internal/proxy/rate_limit_interceptor.go:56\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/workspace/source/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func4\n\t/workspace/source/internal/proxy/privilege_interceptor.go:112\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.HookInterceptor\n\t/workspace/source/internal/proxy/hook_interceptor.go:56\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerHookInterceptor.func3\n\t/workspace/source/internal/proxy/hook_interceptor.go:21\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.DatabaseInterceptor.func2\n\t/workspace/source/internal/proxy/database_interceptor.go:15\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLogInterceptor\n\t/workspace/source/internal/proxy/accesslog/util.go:38\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startExternalGrpc.ChainUnaryServer.func6\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_UpdateCredential_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.4.15/milvuspb/milvus_grpc.pb.go:2425\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343\ngoogle.golang.org/grpc.
iamkhalidbashir commented 2 weeks ago

Even If I start the standalone server with image

And then later when I try to change the password of the root (by using old password as 'Milvus') it doesn't let me image

@yanliang567

xiaofan-luan commented 2 weeks ago

@iamkhalidbashir how did you deploy the cluster? user docker compose or k8s operator or Helm? can you share you deployment?

I think the problem might be you try to change milvus.yaml after instance is already created, which won't work.

the default password is a one time config and won't be used if it is already persistent into etcd

try to test default passowrd Milvus and see if you can change your password.

iamkhalidbashir commented 2 weeks ago

The root password is initialized image

This is my docker-compose deployment file image

Neither the config.yaml works nor the COMMON_SECURITY_DEFAULTROOTPASSWORD

I believe its an issue with attu which doesn't let me sign in when I use the password 'Milvus' for a config.yaml file where defaultRootPassword is set empty

iamkhalidbashir commented 2 weeks ago

Ok the is not with attu but with milvus I have tried multiple times to deploy with fresh ectd and fresh milvus instance every time niether the password 'Milvus' works or the password 'asdasdasd' which I have customized in config image

iamkhalidbashir commented 2 weeks ago

On checking the ectd keys I get following hash stored for user root image

and when comparing it with an online tool, the hash seems to be correct image

But still, it doesn't let me sign in image

iamkhalidbashir commented 2 weeks ago

Ok it worked with Basic authorization image

for the default API key it gave this error (i.e Bearer Token) image

I think attu might be using the Token Bearer method for which it was getting this error

This seems so frustrating as I was struggling for 2 days with this and had to debug the codebase of milvus to resolve this

xiaofan-luan commented 2 weeks ago 
if Params.CommonCfg.AuthorizationEnabled.GetAsBool() {
    if !validSourceID(ctx, md[strings.ToLower(util.HeaderSourceID)]) {
        authStrArr := md[strings.ToLower(util.HeaderAuthorize)]

        if len(authStrArr) < 1 {
            log.Warn("key not found in header")
            return nil, status.Error(codes.Unauthenticated, "missing authorization in header")
        }

        // token format: base64<username:password>
        // token := strings.TrimPrefix(authorization[0], "Bearer ")
        token := authStrArr[0]
        rawToken, err := crypto.Base64Decode(token)
        if err != nil {
            log.Warn("fail to decode the token", zap.Error(err))
            return nil, status.Error(codes.Unauthenticated, "invalid token format")
        }

        if !strings.Contains(rawToken, util.CredentialSeperator) {
            user, err := VerifyAPIKey(rawToken)
            if err != nil {
                log.Warn("fail to verify apikey", zap.Error(err))
                return nil, status.Error(codes.Unauthenticated, "auth check failure, please check api key is correct")
            }
            metrics.UserRPCCounter.WithLabelValues(user).Inc()
            userToken := fmt.Sprintf("%s%s%s", user, util.CredentialSeperator, util.PasswordHolder)
            md[strings.ToLower(util.HeaderAuthorize)] = []string{crypto.Base64Encode(userToken)}
            ctx = metadata.NewIncomingContext(ctx, md)
        } else {
            // username+password authentication
            username, password := parseMD(rawToken)
            if !passwordVerify(ctx, username, password, globalMetaCache) {
                log.Warn("fail to verify password", zap.String("username", username))
                // NOTE: don't use the merr, because it will cause the wrong retry behavior in the sdk
                return nil, status.Error(codes.Unauthenticated, "auth check failure, please check username and password are correct")
            }
            metrics.UserRPCCounter.WithLabelValues(username).Inc()
        }
    }
xiaofan-luan commented 2 weeks ago

if you take a look here, you request goes to verify api key which is not supported by opensource user.

xiaofan-luan commented 2 weeks ago

could you try username password see it working?

iamkhalidbashir commented 2 weeks ago

Username password works with nodejs sdk and rest api But it dosent work with attu

- Khalid

On Sat, 9 Nov 2024 at 1:16 AM Xiaofan @.***> wrote:

could you try username password see it working?

— Reply to this email directly, view it on GitHub https://github.com/milvus-io/milvus/issues/36987#issuecomment-2465678871, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGS5WW2CPZVHKEQUD326MUTZ7UL2NAVCNFSM6AAAAABQFIRJPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRVGY3TQOBXGE . You are receiving this because you were mentioned.Message ID: @.***>

xiaofan-luan commented 2 weeks ago

So which means the username password has already work. but for some reason the username:password style is not working

xiaofan-luan commented 2 weeks ago

@SimFG could you help on it?

xiaofan-luan commented 2 weeks ago

I think the crypto.Base64Decode could be wrongly used at attu and nodejs @shanghaikid could you also help on testing this?

shanghaikid commented 2 weeks ago

I tested this configuration, everything is working. it works even you put defaultRootPassword: "123456" in the config file. https://github.com/zilliztech/attu/issues/684#issuecomment-2464190169

@iamkhalidbashir

xiaofan-luan commented 2 weeks ago

please make sure your milvus is the latest and attu is the latest as well and test on it. We can not reproduce in house

shanghaikid commented 1 week ago

I found out the issue: in milvus.yaml, the default password can only accept up to 20 characters, like 12345678901234567890. Adding just one more character, like 123456789012345678901, doesn’t work. It seems there’s no documentation mentioning this limit.

However, for created users, passwords up to 123456789012345678901 do work, as long as they don’t exceed this constraint:encrypt password failed: bcrypt: password length exceeds 72 bytes. It seems the default password doesn’t follow the same rule.

SimFG commented 1 week ago

@shanghaikid Is this so strange? I'll verify it later.