search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.75k stars 2.93k forks source link

[Bug]: querynode crash when running cluster in k8 #34983

Closed msr120295 closed 2 months ago

msr120295 commented 3 months ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4.6
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): ubuntu - microk8s
- CPU/Memory: 20 Core 100GB RAM
- GPU: GTX Tesla M10 8GB rAM 4 PCs
- Others:

Current Behavior

i tried to run milvus cluster in k8 using helm chart. installed using default helm chart values.

this is k8 pods image

but querynode always error and when i see the logs. this is the error precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000] [2024/07/24 22:34:26.248 +00:00] [WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"] [2024/07/24 22:34:26.248 +00:00] [ERROR] [querynodev2/server.go:369] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:369\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:284\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] [2024/07/24 22:34:26.249 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] [2024/07/24 22:34:26.249 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]

i see that the issue is when querynode trying to connect with s3 minio . is there any solution regarding this ? i tried different container image like 2.3.18 / 2.3.16 / 2.4.5 still same. querynode fails

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

[2024/07/24 22:34:26.034 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"] 2024/07/24 22:34:26 maxprocs: Leaving GOMAXPROCS=20: CPU quota undefined

__  _________ _   ____  ______

/ |/ / _/ /| | / / / / / /
/ /|/ // // /| |/ / // /\ \
// ///__/_/__/_/

Welcome to use Milvus! Version: v2.4.6 Built: Tue Jul 16 05:16:09 UTC 2024 GitCommit: ce9ab52 GoVersion: go version go1.21.10 linux/amd64

TotalMem: 105404379136 UsedMem: 29528064

open pid file: /run/milvus/querynode.pid lock pid file: /run/milvus/querynode.pid [2024/07/24 22:34:26.062 +00:00] [INFO] [roles/roles.go:307] ["starting running Milvus components"] [2024/07/24 22:34:26.062 +00:00] [INFO] [roles/roles.go:170] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so] [2024/07/24 22:34:26.075 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/07/24 22:34:26.077 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"my-milvus-etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"] [2024/07/24 22:34:26.077 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3] [2024/07/24 22:34:26.080 +00:00] [DEBUG] [config/etcd_source.go:91] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[my-milvus-etcd:2379]"] [2024/07/24 22:34:26.081 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource] [2024/07/24 22:34:26.081 +00:00] [INFO] [paramtable/component_param.go:3566] ["DeployModeEnv is not set, use default"] [default=0.5] [2024/07/24 22:34:26.084 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}] [2024/07/24 22:34:26.084 +00:00] [INFO] [roles/roles.go:256] [setupPrometheusHTTPServer] [2024/07/24 22:34:26.084 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/07/24 22:34:26.084 +00:00] [INFO] [http/server.go:152] ["management listen"] [addr=:9091] [2024/07/24 22:34:26.084 +00:00] [DEBUG] [querynode/service.go:104] [QueryNode] [port=21123] [2024/07/24 22:34:26.084 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3] [2024/07/24 22:34:26.085 +00:00] [DEBUG] [querynode/service.go:124] ["QueryNode connect to etcd successfully"] [2024/07/24 22:34:26.186 +00:00] [DEBUG] [querynode/service.go:134] [QueryNode] [State=Initializing] [2024/07/24 22:34:26.186 +00:00] [INFO] [querynodev2/server.go:286] ["QueryNode session info"] [metaPath=by-dev/meta] [2024/07/24 22:34:26.187 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3] [2024/07/24 22:34:26.190 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"] [2024/07/24 22:34:26.192 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"] [2024/07/24 22:34:26.195 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true] [2024/07/24 22:34:26.198 +00:00] [DEBUG] [sessionutil/session_util.go:373] ["Session get serverID success"] [key=id] [ServerId=116] [2024/07/24 22:34:26.198 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querynode] [address=10.1.29.184:21123] [id=116] [2024/07/24 22:34:26.199 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querynode-116\n"] [filePath=/tmp/milvus/server_id_8] [2024/07/24 22:34:26.199 +00:00] [INFO] [querynodev2/server.go:160] ["QueryNode init session"] [nodeID=116] ["node address"=10.1.29.184:21123] [2024/07/24 22:34:26.199 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=false] [mqType=kafka] [2024/07/24 22:34:26.204 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=milvus-bucket] [root=file] [2024/07/24 22:34:26.205 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=20] [2024/07/24 22:34:26.205 +00:00] [INFO] [querynodev2/server.go:326] ["queryNode init scheduler"] [policy=fifo] [2024/07/24 22:34:26.205 +00:00] [INFO] [segments/segment_loader.go:548] ["SegmentLoader created"] [ioPoolSize=160] WARNING: Logging before InitGoogleLogging() is written to STDERR I20240724 22:34:26.205444 47 knowhere_config.cc:107] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 I20240724 22:34:26.205590 47 knowhere_config.cc:118] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 I20240724 22:34:26.205642 47 knowhere_config.cc:54] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.3.5-hotfix I20240724 22:34:26.206149 47 thread_pool.h:172] [KNOWHERE][InitGlobalSearchThreadPool][milvus] Init global search thread pool with size 80 I20240724 22:34:26.206229 47 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO I20240724 22:34:26.206250 47 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX2 I20240724 22:34:26.206295 47 Common.cpp:35] [SERVER][SetIndexSliceSize][milvus] set config index slice size (byte): 16777216 I20240724 22:34:26.206351 47 Common.cpp:41] [SERVER][SetHighPriorityThreadCoreCoefficient][milvus] set high priority thread pool core coefficient: 10 I20240724 22:34:26.206379 47 Common.cpp:48] [SERVER][SetMiddlePriorityThreadCoreCoefficient][milvus] set middle priority thread pool core coefficient: 5 I20240724 22:34:26.206404 47 Common.cpp:55] [SERVER][SetLowPriorityThreadCoreCoefficient][milvus] set low priority thread pool core coefficient: 1 [2024/07/24 22:34:26.206 +00:00] [INFO] [querynodev2/server.go:227] ["set up knowhere build pool size"] [pool_size=10] I20240724 22:34:26.206501 47 thread_pool.h:152] [KNOWHERE][InitGlobalBuildThreadPool][milvus] Init global build thread pool with size 10 I20240724 22:34:26.206533 47 Common.cpp:62] [SERVER][SetDefaultExecEvalExprBatchSize][milvus] set default expr eval batch size: 8192 I20240724 22:34:26.206629 47 MinioChunkManager.cpp:161] [SERVER][InitSDKAPIDefault][milvus] init aws with log level:fatal I20240724 22:34:26.243103 47 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000] E20240724 22:34:26.246778 47 MinioChunkManager.cpp:237] [SERVER][PreCheck][milvus] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000] [2024/07/24 22:34:26.248 +00:00] [WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"] [2024/07/24 22:34:26.248 +00:00] [ERROR] [querynodev2/server.go:369] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:369\nsync.(Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:284\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] [2024/07/24 22:34:26.249 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] [2024/07/24 22:34:26.249 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"] panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]

Anything else?

No response

yanliang567 commented 3 months ago

/assign @LoveEachDay could you please take a look at this issue /unassign

xiaofan-luan commented 3 months ago

WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck],

msr120295 commented 3 months ago

Hello, I want to ask, what is the best version of milvus to run ? i tried image from 2.3.0 to 2.4.6 still error. querynode crashloopback because of s3

xiaofan-luan commented 3 months ago

[address=my-milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"

Did you follow this document to set the right persmisison? https://milvus.io/docs/eks.md

image

msr120295 commented 3 months ago

i use minio for s3. i didnt use external minio. i follow this link https://milvus.io/docs/install_cluster-helm.md. i follow default helm values. is that ok ?

xiaofan-luan commented 3 months ago

but did you enable any access control? I'm assuming you only get this when you enable some kind of access control

xiaofan-luan commented 3 months ago

I suggest you post out all the changes you've been made on helm chart so we can check it. I don't believe you will hit this issue if you follow the default setup

msr120295 commented 3 months ago

i follow this helm https://artifacthub.io/packages/helm/milvus-helm/milvus/4.2.0

this is my helm values, i only change image tag from 2.4.6 to 2.4.6-gpu version

cluster: enabled: true

image: all: repository: milvusdb/milvus tag: v2.4.6-gpu pullPolicy: IfNotPresent tools: repository: milvusdb/milvus-config-tool tag: v0.1.2 pullPolicy: IfNotPresent

nodeSelector: {} tolerations: [] affinity: {} labels: {} annotations: {} customConfigMap: ""

service: type: ClusterIP port: 19530 portName: milvus nodePort: "" annotations: {} labels: {} loadBalancerSourceRanges:

ingress: enabled: false annotations: nginx.ingress.kubernetes.io/backend-protocol: GRPC nginx.ingress.kubernetes.io/listen-ports-ssl: '[19530]' nginx.ingress.kubernetes.io/proxy-body-size: 4m nginx.ingress.kubernetes.io/ssl-redirect: "true" labels: {} rules:

serviceAccount: create: false name: annotations: labels:

metrics: enabled: true

serviceMonitor: enabled: false interval: "30s" scrapeTimeout: "10s" additionalLabels: {}

livenessProbe: enabled: true initialDelaySeconds: 90 periodSeconds: 30 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5

readinessProbe: enabled: true initialDelaySeconds: 90 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5

log: level: "info" file: maxSize: 300 maxAge: 10 maxBackups: 20 format: "text"

persistence: mountPath: "/milvus/logs"

enabled: false
annotations:
  helm.sh/resource-policy: keep
persistentVolumeClaim:
  existingClaim: ""
  storageClass:
  accessModes: ReadWriteMany
  size: 10Gi
  subPath: ""

heaptrack: image: repository: milvusdb/heaptrack tag: v0.1.0 pullPolicy: IfNotPresent

proxy: enabled: true replicas: 1 resources: {} nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false http: enabled: true
debugMode: enabled: false

tls: enabled: false strategy: {}

rootCoordinator: enabled: true replicas: 1
resources: {} nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false activeStandby: enabled: false strategy: {}

service: port: 53100 annotations: {} labels: {} clusterIP: ""

queryCoordinator: enabled: true replicas: 1
resources: {} nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false activeStandby: enabled: false
strategy: {}

service: port: 19531 annotations: {} labels: {} clusterIP: ""

queryNode: enabled: true

replicas: 1 resources: requests: nvidia.com/gpu: "1" limits: nvidia.com/gpu: "1"

nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false disk: enabled: true
size: enabled: false
profiling: enabled: false

strategy: {}

indexCoordinator: enabled: true replicas: 1
resources: {} nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false
activeStandby: enabled: false strategy: {}

service: port: 31000 annotations: {} labels: {} clusterIP: ""

indexNode: enabled: true replicas: 1 resources: requests: nvidia.com/gpu: "1" limits: nvidia.com/gpu: "1" nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false
disk: enabled: true
size: enabled: false strategy: {}

dataCoordinator: enabled: true replicas: 1
resources: {} nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false activeStandby: enabled: false
strategy: {}

service: port: 13333 annotations: {} labels: {} clusterIP: ""

dataNode: enabled: true replicas: 1 resources: {} nodeSelector: {} affinity: {} tolerations: [] extraEnv: [] heaptrack: enabled: false profiling: enabled: false strategy: {} mixCoordinator: enabled: false

attu: enabled: false

minio: enabled: true name: minio mode: distributed image: tag: "RELEASE.2023-03-20T20-16-18Z" pullPolicy: IfNotPresent accessKey: minioadmin secretKey: minioadmin existingSecret: "" bucketName: "milvus-bucket" rootPath: file useIAM: false iamEndpoint: "" region: "" useVirtualHost: false podDisruptionBudget: enabled: false resources: requests: memory: 2Gi

service: type: ClusterIP port: 9000

persistence: enabled: true existingClaim: "" storageClass: accessMode: ReadWriteOnce size: 500Gi

livenessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5

readinessProbe: enabled: true initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 1 successThreshold: 1 failureThreshold: 5

startupProbe: enabled: true initialDelaySeconds: 0 periodSeconds: 10 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 60

etcd: enabled: true name: etcd replicaCount: 3 pdb: create: false image: repository: "milvusdb/etcd" tag: "3.5.5-r4" pullPolicy: IfNotPresent

service: type: ClusterIP port: 2379 peerPort: 2380

auth: rbac: enabled: false

persistence: enabled: true storageClass: accessMode: ReadWriteOnce size: 10Gi

livenessProbe: enabled: true timeoutSeconds: 10

readinessProbe: enabled: true periodSeconds: 20 timeoutSeconds: 10

autoCompactionMode: revision autoCompactionRetention: "1000"

extraEnvVars:

pulsar: enabled: false

kafka: enabled: false

msr120295 commented 3 months ago

@xiaofan-luan

i have added s3 permission and using stand alone minio. outside from milvus helm. but still issue like this

WARN] [initcore/init_core.go:168] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck],

xiaofan-luan commented 3 months ago

are u using the right AK/SK or enabled IAM?

accessKey: minioadmin secretKey: minioadmin bucketName: "milvus-bucket" rootPath: file useIAM: false

msr120295 commented 3 months ago

no, i am not using IAM (identity access management)

xiaofan-luan commented 3 months ago

you can probably test your S3 connectivity with https://github.com/zhagnlu/milvus-store-tool 

From your log this seems to be clearly a S3 problem

msr120295 commented 3 months ago

i tried to click your link but it seems 404 not found for milvus store tool image

xiaofan-luan commented 3 months ago

https://github.com/zhagnlu/milvus-store-tool

xiaofan-luan commented 3 months ago

try this

locustbaby commented 3 months ago

Hi @msr120295 , from the picture, if you want to install milvus cluster with kafka, you can run the cmd helm upgrade --install my-milvus milvus/milvus --set pulsar.enabled=false --set kafka.enabled=true

And I want to confirm if you're going to use S3 or another oss as your datastore?

msr120295 commented 3 months ago

yes @locustbaby . i want to use minio as s3 storage. can you provide the correct helm chart values ?

xiaofan-luan commented 3 months ago

minio: enabled: true name: minio mode: distributed image: tag: "RELEASE.2023-03-20T20-16-18Z" pullPolicy: IfNotPresent accessKey: minioadmin secretKey: minioadmin existingSecret: "" bucketName: "milvus-bucket" rootPath: file useIAM: false iamEndpoint: "" region: "" useVirtualHost: false podDisruptionBudget: enabled: false resources: requests: memory: 2Gi

So the helm chart looks as is?

xiaofan-luan commented 3 months ago

can you share all the step you do to install the cluster? including you you setup your minio?

xiaofan-luan commented 3 months ago

here is a general guide investigate on this failure

This error suggests there's an issue with the authorization parameters in your S3-compatible API request, specifically with the date format in the X-Amz-Credential parameter. Here's how you can address this problem:

Check the date format: Ensure that the date in your X-Amz-Credential parameter is in the format "yyyyMMdd" (e.g., 20240813 for August 13, 2024). Verify your system clock: Make sure your system's clock is synchronized and set to the correct time. A significant time difference between your client and the server can cause authorization failures. Review your SDK or client code: If you're using an SDK or custom code to generate the request:

Ensure you're using the correct date formatting function. Check if there are any timezone issues affecting the date generation.

Examine the full credential string: The X-Amz-Credential typically follows this format:

////aws4_request Verify that all parts are correct, not just the date. Use debugging tools: Enable debug logging in your client or SDK to see the exact request being sent. This can help identify where the incorrect date is coming from. Check for middleware or proxies: If you're using any middleware or proxies, ensure they're not modifying the authorization headers. Regenerate the signature: If you're manually constructing the authorization header, try regenerating the entire signature with the correct date format. Consult API documentation: Review the documentation for the specific S3-compatible service you're using to ensure you're following their exact requirements for authorization.
xiaofan-luan commented 3 months ago

X-Amz-Credential is automatically generated by minio client so please check your system clock and see it's offset is huge

msr120295 commented 3 months ago

ok thanks @xiaofan-luan it works now

SalBakraa commented 3 months ago

Hi, I have been running into this issue when using docker compose with milvus-standalone.

milvus-standalone  | [2024/08/18 09:18:49.081 +00:00] [WARN] [grpcclient/client.go:478] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:631\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:452\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:54\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | I20240818 09:18:49.113718    83 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
milvus-standalone  | E20240818 09:18:49.116004    83 MinioChunkManager.cpp:237] [SERVER][PreCheck][milvus] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
milvus-standalone  | [2024/08/18 09:18:49.121 +00:00] [WARN] [initcore/init_core.go:217] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"]
milvus-standalone  | [2024/08/18 09:18:49.121 +00:00] [ERROR] [querynodev2/server.go:359] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:359\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:276\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
milvus-standalone  | [2024/08/18 09:18:49.121 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
milvus-standalone  | [2024/08/18 09:18:49.121 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
milvus-standalone  | panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]

My system clock was off, so I synced it. However the error still presists.

$ timedatectl
               Local time: Sun 2024-08-18 12:25:17 +03
           Universal time: Sun 2024-08-18 09:25:17 UTC
                 RTC time: Sun 2024-08-18 09:25:17
                Time zone: Asia/Riyadh (+03, +0300)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

The following is my docker-compose.yml:

networks:
  default:
    driver: bridge  # Define custom network as bridge

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    restart: on-failure
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
      - default

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    restart: on-failure
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
      - default

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.8
    restart: on-failure
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"
    networks:
      - default

volumes:
  milvus_data:
xiaofan-luan commented 3 months ago

\"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n | \t/workspace/source/internal/util/initcore/init_core.go:218\n | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n | \t/workspace/source/internal/util/initcore/init_core.go:171\n | github.com/milvus-io/milvus/internal/querynodev2.(QueryNode).InitSegcore\n | \t/workspace/source/internal/querynodev2/server.go:244\n | github.com/milvus-io/milvus/internal/querynodev2.(QueryNode).Init.func1\n | \t/workspace/source/internal/querynodev2/server.go:357\n | sync.(Once).doSlow\n | \t/usr/local/go/src/sync/once.go:74\n | sync.(Once).Do\n | \t/usr/local/go/src/sync/once.go:65\n | github.com/milvus-io/milvus/internal/querynodev2.(QueryNode).Init\n | \t/workspace/source/internal/querynodev2/server.go:276\n | github.com/milvus-io/milvus/internal/distributed/querynode.(Server).init\n | \t/workspace/source/internal/distributed/querynode/service.go:135\n | github.com/milvus-io/milvus/internal/distributed/querynode.(Server).Run\n | \t/workspace/source/internal/distributed/querynode/service.go:231\n | github.com/milvus-io/milvus/cmd/components.(QueryNode).Run\n | \t/workspace/source/cmd/components/query_node.go:55\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:121\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) withstack.withStack (2) errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:359\nsync.(Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:276\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]

i think you might need to ask your GCP team for support. we don't have too much detail about how this is happened. It seems that you already have the right config

SalBakraa commented 3 months ago

Can you expand on what GCP stands for?I am running milvus locally on my workstation.

xiaofan-luan commented 3 months ago

Can you expand on what GCP stands for?I am running milvus locally on my workstation.

So you deployed minio locally on your work station?

xiaofan-luan commented 3 months ago

did you enable ntp on all the nodes?

xiaofan-luan commented 3 months ago

from the S3 SDK document you sould get a request to minio like this:

GET -------- HTTP method / -------- Path. For API stage endpoint, it should be /{stage-name}/{resource-path} -------- Query string key-value pair. Leave it blank if the request doesn't have a query string. content-type:application/json -------- Header key-value pair. One header per line. host:0123456789.execute-api.us-east-1.amazonaws.com -------- Host and x-amz-date are required headers for all signed requests.
x-amz-date:20220806T024003Z

content-type;host;x-amz-date -------- A list of signed headers d167e99c53f15b0c105101d468ae35a3dc9187839ca081095e340f3649a04501 -------- Hash of the payload

Could you use wired shark to capture a pack to S3 see what's inside? An easier way is to use this tool https://github.com/zhagnlu/milvus-store-tool to trigger request to S3. I'm pretty sure you have some wrong configuration(might not be milvus configurations)

SalBakraa commented 3 months ago

So you deployed minio locally on your work station?

I am running the everything on a local workstation using docker compose. sudo docker compose up

from the S3 SDK document you sould get a request to minio like this:

GET -------- HTTP method / -------- Path. For API stage endpoint, it should be /{stage-name}/{resource-path} -------- Query string key-value pair. Leave it blank if the request doesn't have a query string. content-type:application/json -------- Header key-value pair. One header per line. host:0123456789.execute-api.us-east-1.amazonaws.com -------- Host and x-amz-date are required headers for all signed requests. x-amz-date:20220806T024003Z

content-type;host;x-amz-date -------- A list of signed headers d167e99c53f15b0c105101d468ae35a3dc9187839ca081095e340f3649a04501 -------- Hash of the payload

Could you use wired shark to capture a pack to S3 see what's inside? An easier way is to use this tool https://github.com/zhagnlu/milvus-store-tool to trigger request to S3. I'm pretty sure you have some wrong configuration(might not be milvus configurations)

The thing is, I get no issues when running the docker images on my machine while it's offline. The issue only shows up when running minio while the machine is connected to the internet. Is there a way to permanently disable minio from connecting to the internet? I don't need to use AWS or any cloud providers for my project.

Running MinioTool while connected to internet

Running MinioTool while offline

xiaofan-luan commented 3 months ago

what's the configuration difference between the two cluster?

SalBakraa commented 2 months ago

The difference is here:

Offline docker_compose.yml:

networks:
  default:
    driver: bridge  # Define custom network as bridge
  offline-network:
    driver: bridge  # Define custom network as bridge
    internal: true # Block internet access

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    restart: on-failure
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
      - offline-network

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    restart: on-failure
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
      - offline-network

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.8
    restart: on-failure
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"
    networks:
      - default
      - offline-network

volumes:
  milvus_data:
grebois commented 2 months ago

Same error with 2.4.9:

I20240827 17:59:45.147747    56 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
E20240827 17:59:45.148984    56 MinioChunkManager.cpp:237] [SERVER][PreCheck][milvus] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
[2024/08/27 17:59:45.150 +00:00] [WARN] [initcore/init_core.go:217] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"]
[2024/08/27 17:59:45.150 +00:00] [ERROR] [querynodev2/server.go:359] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:359\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:276\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
[2024/08/27 17:59:45.150 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
[2024/08/27 17:59:45.150 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]

Using this helm values

cluster:
  enabled: false
etcd:
  replicaCount: 1
minio:
  mode: standalone
pulsar:
  enabled: false
xiaofan-luan commented 2 months ago

Same error with 2.4.9:

I20240827 17:59:45.147747    56 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
E20240827 17:59:45.148984    56 MinioChunkManager.cpp:237] [SERVER][PreCheck][milvus] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
[2024/08/27 17:59:45.150 +00:00] [WARN] [initcore/init_core.go:217] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"]
[2024/08/27 17:59:45.150 +00:00] [ERROR] [querynodev2/server.go:359] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:359\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:276\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
[2024/08/27 17:59:45.150 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
[2024/08/27 17:59:45.150 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=milvus-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]

Using this helm values

cluster:
  enabled: false
etcd:
  replicaCount: 1
minio:
  mode: standalone
pulsar:
  enabled: false

did you got a chance to check the clock skew between your milvus and your minio?

grebois commented 2 months ago

Hi @xiaofan-luan is this what you mean?

> kubectl get pods
NAME                                 READY   STATUS             RESTARTS        AGE
milvus-attu-5c47cbf894-zmwzp         1/1     Running            1 (4h36m ago)   13h
milvus-etcd-0                        1/1     Running            1 (4h36m ago)   13h
milvus-minio-57c5cb9b56-7v7nn        1/1     Running            1 (4h36m ago)   13h
milvus-standalone-6bc68fb797-sfksf   0/1     CrashLoopBackOff   174 (82s ago)   13h
> kubectl exec -it milvus-minio-57c5cb9b56-7v7nn -- date
Wed Aug 28 11:29:52 UTC 2024
> kubectl exec -it milvus-standalone-6bc68fb797-sfksf -- date
error: Internal error occurred: unable to upgrade connection: container not found ("standalone")
> kubectl exec -it milvus-etcd-0 -- date
Wed Aug 28 11:30:40 UTC 2024
> kubectl exec -it milvus-attu-5c47cbf894-zmwzp -- date
Wed Aug 28 11:30:51 UTC 2024

I cant run date into the standalone pod:

> kubectl delete pod milvus-standalone-6bc68fb797-sfksf && kubectl get pods
pod "milvus-standalone-6bc68fb797-sfksf" deleted
NAME                                 READY   STATUS    RESTARTS        AGE
milvus-attu-5c47cbf894-zmwzp         1/1     Running   1 (4h39m ago)   13h
milvus-etcd-0                        1/1     Running   1 (4h39m ago)   13h
milvus-minio-57c5cb9b56-7v7nn        1/1     Running   1 (4h39m ago)   13h
milvus-standalone-6bc68fb797-4zh8b   0/1     Error     0               2s
> kubectl exec -it milvus-standalone-6bc68fb797-4zh8b  -- date
error: Internal error occurred: unable to upgrade connection: container not found ("standalone")
> kubectl exec -it milvus-standalone-6bc68fb797-4zh8b  -- date
error: Internal error occurred: unable to upgrade connection: container not found ("standalone")

This is currently happening in 10 multiple cluster at the same time, all freshly installed. Worth mentioning that this happens in installation that are located in UAE, an exact same deployment in Europe work just fine. Not sure if thats somehow related.

grebois commented 2 months ago

@xiaofan-luan same problem while using docker compose:

root@d000065:~# wget https://github.com/milvus-io/milvus/releases/download/v2.4.9/milvus-standalone-docker-compose.yml -O docker-compose.yml
--2024-08-28 14:06:03--  https://github.com/milvus-io/milvus/releases/download/v2.4.9/milvus-standalone-docker-compose.yml
Resolving github.com (github.com)... 20.233.83.145
Connecting to github.com (github.com)|20.233.83.145|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/208728772/8ffe36d4-f65a-4091-ae99-21d935e02f94?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240828%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240828T140604Z&X-Amz-Expires=300&X-Amz-Signature=8fbb3c220bcbb3aed3317a4f0771ff57cf98671925b2fb82430bbfe5f416c16c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=208728772&response-content-disposition=attachment%3B%20filename%3Dmilvus-standalone-docker-compose.yml&response-content-type=application%2Foctet-stream [following]
--2024-08-28 14:06:04--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/208728772/8ffe36d4-f65a-4091-ae99-21d935e02f94?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240828%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240828T140604Z&X-Amz-Expires=300&X-Amz-Signature=8fbb3c220bcbb3aed3317a4f0771ff57cf98671925b2fb82430bbfe5f416c16c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=208728772&response-content-disposition=attachment%3B%20filename%3Dmilvus-standalone-docker-compose.yml&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1766 (1.7K) [application/octet-stream]
Saving to: ‘docker-compose.yml’

docker-compose.yml                         100%[=======================================================================================>]   1.72K  --.-KB/s    in 0s

2024-08-28 14:06:05 (53.8 MB/s) - ‘docker-compose.yml’ saved [1766/1766]

root@d000065:~# docker compose up
WARN[0000] /root/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
[+] Running 24/24
 ✔ standalone Pulled                                                                                                                                                 81.6s
   ✔ 7646c8da3324 Pull complete                                                                                                                                      10.0s
   ✔ e071a2bda6d5 Pull complete                                                                                                                                      19.6s
   ✔ 653cde9beea1 Pull complete                                                                                                                                      20.4s
   ✔ 865722af4c96 Pull complete                                                                                                                                      20.4s
   ✔ 79099f9f1d08 Pull complete                                                                                                                                      78.1s
   ✔ 535376b75c84 Pull complete                                                                                                                                      78.1s
   ✔ eb2f8229d375 Pull complete                                                                                                                                      78.1s
   ✔ 4f4fb700ef54 Pull complete                                                                                                                                      78.1s
 ✔ etcd Pulled                                                                                                                                                       39.4s
   ✔ dbba69284b27 Pull complete                                                                                                                                      27.3s
   ✔ 270b322b3c62 Pull complete                                                                                                                                      27.4s
   ✔ 7c21e2da1038 Pull complete                                                                                                                                      30.1s
   ✔ cb4f77bfee6c Pull complete                                                                                                                                      31.2s
   ✔ e5485096ca5d Pull complete                                                                                                                                      32.5s
   ✔ 3ea3736f61e1 Pull complete                                                                                                                                      32.6s
   ✔ 1e815a2c4f55 Pull complete                                                                                                                                      34.7s
 ✔ minio Pulled                                                                                                                                                      17.0s
   ✔ c7e856e03741 Pull complete                                                                                                                                      11.1s
   ✔ c1ff217ec952 Pull complete                                                                                                                                      11.1s
   ✔ b12cc8972a67 Pull complete                                                                                                                                      11.1s
   ✔ 4324e307ea00 Pull complete                                                                                                                                      11.1s
   ✔ 152089595ebc Pull complete                                                                                                                                      11.1s
   ✔ 05f217fb8612 Pull complete                                                                                                                                      13.7s
[+] Running 4/4
 ✔ Network root_default         Created                                                                                                                               0.1s
 ✔ Container milvus-minio       Started                                                                                                                               1.5s
 ✔ Container milvus-etcd        Started                                                                                                                               1.5s
 ✔ Container milvus-standalone  Started                                                                                                                               0.5s
Attaching to milvus-etcd, milvus-minio, milvus-standalone
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"1000"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"4294967296"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_SNAPSHOT_COUNT","variable-value":"50000"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","-advertise-client-urls=http://127.0.0.1:2379","-listen-client-urls","http://0.0.0.0:2379","--data-dir","/etcd"]}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/etcd","dir-type":"member"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"19002cfc6","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":32,"max-cpu-available":32,"member-initialized":true,"name":"default","data-dir":"/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":50000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://127.0.0.1:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":4294967296,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"1µs","auto-compaction-interval":"1µs","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
milvus-etcd        | {"level":"warn","ts":1724854145.7097147,"caller":"fileutil/fileutil.go:57","msg":"check file permission","error":"directory \"/etcd\" exist, but the permission is \"drwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/etcd/member/snap/db","took":"77.036µs"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.709Z","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.710Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","commit-index":33}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.710Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=()"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.710Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became follower at term 2"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.710Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 33, applied: 0, lastindex: 33, lastterm: 2]"}
milvus-etcd        | {"level":"warn","ts":"2024-08-28T14:09:05.711Z","caller":"auth/store.go:1233","msg":"simple token is not cryptographically signed"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.711Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":21}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.712Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":4294967296,"quota-size":"4.3 GB"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.713Z","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.5.5","cluster-version":"to_be_decided"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.713Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.714Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.714Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.714Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","cluster-version":"3.5"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.714Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.714Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://127.0.0.1:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.714Z","caller":"embed/etcd.go:584","msg":"serving peer traffic","address":"127.0.0.1:2380"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:05.715Z","caller":"embed/etcd.go:556","msg":"cmux::serve","address":"127.0.0.1:2380"}
milvus-minio       | WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
milvus-minio       |          Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
milvus-minio       | WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
milvus-minio       | MinIO Object Storage Server
milvus-minio       | Copyright: 2015-2023 MinIO, Inc.
milvus-minio       | License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
milvus-minio       | Version: RELEASE.2023-03-20T20-16-18Z (go1.19.7 linux/amd64)
milvus-minio       |
milvus-minio       | Status:         1 Online, 0 Offline.
milvus-minio       | API: http://172.18.0.2:9000  http://127.0.0.1:9000
milvus-minio       | Console: http://172.18.0.2:9001 http://127.0.0.1:9001
milvus-minio       |
milvus-minio       | Documentation: https://min.io/docs/minio/linux/index.html
milvus-minio       | Warning: The standard parity is set to 0. This can lead to data loss.
milvus-standalone  | [2024/08/28 14:09:06.011 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
milvus-standalone  | 2024/08/28 14:09:06 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined
milvus-standalone  |
milvus-standalone  |     __  _________ _   ____  ______
milvus-standalone  |    /  |/  /  _/ /| | / / / / / __/
milvus-standalone  |   / /|_/ // // /_| |/ / /_/ /\ \
milvus-standalone  |  /_/  /_/___/____/___/\____/___/
milvus-standalone  |
milvus-standalone  | Welcome to use Milvus!
milvus-standalone  | Version:   v2.4.9
milvus-standalone  | Built:     Fri Aug 16 08:41:42 UTC 2024
milvus-standalone  | GitCommit: 4e098f1
milvus-standalone  | GoVersion: go version go1.21.11 linux/amd64
milvus-standalone  |
milvus-standalone  | TotalMem: 134863863808
milvus-standalone  | UsedMem: 31449088
milvus-standalone  |
milvus-standalone  | open pid file: /run/milvus/standalone.pid
milvus-standalone  | lock pid file: /run/milvus/standalone.pid
milvus-standalone  | [2024/08/28 14:09:06.024 +00:00] [INFO] [roles/roles.go:328] ["starting running Milvus components"]
milvus-standalone  | [2024/08/28 14:09:06.024 +00:00] [INFO] [roles/roles.go:177] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
milvus-standalone  | [2024/08/28 14:09:06.029 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
milvus-standalone  | [2024/08/28 14:09:06.030 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
milvus-standalone  | [2024/08/28 14:09:06.030 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-minio       |
milvus-minio       |  You are running an older version of MinIO released 1 year ago
milvus-minio       |  Update: Run `mc admin update`
milvus-minio       |
milvus-minio       |
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 2"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 2"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 2"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 3"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 3"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.032Z","caller":"embed/serve.go:100","msg":"ready to serve client requests"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.032Z","caller":"etcdserver/server.go:2054","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://127.0.0.1:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.032Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.032Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
milvus-etcd        | {"level":"info","ts":"2024-08-28T14:09:07.032Z","caller":"embed/serve.go:146","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"[::]:2379"}
milvus-standalone  | [2024/08/28 14:09:07.033 +00:00] [DEBUG] [config/etcd_source.go:91] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd:2379]"]
milvus-standalone  | [2024/08/28 14:09:07.035 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
milvus-standalone  | [2024/08/28 14:09:07.037 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
milvus-standalone  | [2024/08/28 14:09:07.037 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
milvus-standalone  | [2024/08/28 14:09:07.038 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
milvus-standalone  | [2024/08/28 14:09:07.038 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
milvus-standalone  | [2024/08/28 14:09:07.038 +00:00] [INFO] [roles/roles.go:277] [setupPrometheusHTTPServer]
milvus-standalone  | [2024/08/28 14:09:07.038 +00:00] [INFO] [http/server.go:152] ["management listen"] [addr=:9091]
milvus-standalone  | [2024/08/28 14:09:07.039 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=121377477427]
milvus-standalone  | [2024/08/28 14:09:07.039 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
milvus-standalone  | [2024/08/28 14:09:07.039 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.040 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.040 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."]
milvus-standalone  | [2024/08/28 14:09:07.040 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.040 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.041 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.041 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"]
milvus-standalone  | [2024/08/28 14:09:07.041 +00:00] [INFO] [proxy/look_aside_balancer.go:204] ["Start check query node health loop"]
milvus-standalone  | [2024/08/28 14:09:07.041 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]
milvus-standalone  | [2024/08/28 14:09:07.041 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]
milvus-standalone  | [2024/08/28 14:09:07.041 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.042 +00:00] [INFO] [datanode/service.go:256] ["DataNode address"] [address=172.18.0.4:21124]
milvus-standalone  | [2024/08/28 14:09:07.042 +00:00] [INFO] [datanode/service.go:257] ["DataNode serverID"] [serverID=0]
milvus-standalone  | [2024/08/28 14:09:07.042 +00:00] [INFO] [proxy/service.go:398] ["Proxy internal server listen on tcp"] [port=19529]
milvus-standalone  | [2024/08/28 14:09:07.043 +00:00] [INFO] [proxy/service.go:405] ["Proxy internal server already listen on tcp"] [port=19529]
milvus-standalone  | [2024/08/28 14:09:07.043 +00:00] [INFO] [proxy/service.go:437] ["create Proxy internal grpc server"] ["enforcement policy"="{\"MinTime\":5000000000,\"PermitWithoutStream\":true}"] ["server parameters"="{\"MaxConnectionIdle\":0,\"MaxConnectionAge\":0,\"MaxConnectionAgeGrace\":0,\"Time\":60000000000,\"Timeout\":10000000000}"]
milvus-standalone  | [2024/08/28 14:09:07.043 +00:00] [INFO] [proxy/service.go:530] ["Proxy server listen on tcp"] [port=19530]
milvus-standalone  | [2024/08/28 14:09:07.043 +00:00] [INFO] [proxy/service.go:533] ["Proxy server already listen on tcp"] [port=19530]
milvus-standalone  | [2024/08/28 14:09:07.043 +00:00] [INFO] [proxy/service.go:614] ["register Proxy http server"]
milvus-standalone  | [2024/08/28 14:09:07.043 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [grpcclient/client.go:457] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [grpcclient/client.go:478] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:631\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:452\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:54\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [grpcclient/client.go:464] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.045 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:631\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:452\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:54\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.142 +00:00] [INFO] [querynodev2/server.go:278] ["QueryNode session info"] [metaPath=by-dev/meta]
milvus-standalone  | [2024/08/28 14:09:07.142 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[etcd:2379]"] [minVersion=1.3]
milvus-standalone  | [2024/08/28 14:09:07.142 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
milvus-standalone  | [2024/08/28 14:09:07.142 +00:00] [INFO] [datanode/service.go:266] ["initializing RootCoord client for DataNode"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [INFO] [indexnode/indexnode.go:209] ["IndexNode init"] [state=Initializing]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:457] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:457] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:478] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:273\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:188\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:478] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:169\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:100\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:464] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [grpcclient/client.go:464] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:169\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:100\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.143 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n  | \t/workspace/source/internal/distributed/datanode/service.go:273\n  | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n  | \t/workspace/source/internal/distributed/datanode/service.go:188\n  | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n  | \t/workspace/source/cmd/components/data_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querynode] [address=172.18.0.4:21123] [id=2]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexnode] [address=172.18.0.4:21121] [id=2]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querynode-2\n"] [filePath=/tmp/milvus/server_id_8]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [querynodev2/server.go:163] ["QueryNode init session"] [nodeID=2] ["node address"=172.18.0.4:21123]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="indexnode-2\n"] [filePath=/tmp/milvus/server_id_8]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [indexnode/indexnode.go:216] ["IndexNode init session successful"] [serverID=2]
milvus-standalone  | [2024/08/28 14:09:07.144 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=true] [mqType=rocksmq]
milvus-standalone  | WARNING: Logging before InitGoogleLogging() is written to STDERR
milvus-standalone  | I20240828 14:09:07.144827    46 knowhere_config.cc:107] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384
milvus-standalone  | I20240828 14:09:07.144910    46 knowhere_config.cc:118] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0
milvus-standalone  | I20240828 14:09:07.144932    46 knowhere_config.cc:54] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.3.7
milvus-standalone  | I20240828 14:09:07.145040    46 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
milvus-standalone  | I20240828 14:09:07.145051    46 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX2
milvus-standalone  | I20240828 14:09:07.145076    46 Common.cpp:35] [SERVER][SetIndexSliceSize][milvus] set config index slice size (byte): 16777216
milvus-standalone  | I20240828 14:09:07.145107    46 Common.cpp:41] [SERVER][SetHighPriorityThreadCoreCoefficient][milvus] set high priority thread pool core coefficient: 10
milvus-standalone  | I20240828 14:09:07.145121    46 Common.cpp:48] [SERVER][SetMiddlePriorityThreadCoreCoefficient][milvus] set middle priority thread pool core coefficient: 5
milvus-standalone  | I20240828 14:09:07.145134    46 Common.cpp:55] [SERVER][SetLowPriorityThreadCoreCoefficient][milvus] set low priority thread pool core coefficient: 1
milvus-standalone  | I20240828 14:09:07.145193    46 thread_pool.h:152] [KNOWHERE][InitGlobalBuildThreadPool][milvus] Init global build thread pool with size 24
milvus-standalone  | [2024/08/28 14:09:07.145 +00:00] [INFO] [indexnode/indexnode.go:221] ["init index node done"] [nodeID=2] [Address=172.18.0.4:21121]
milvus-standalone  | [2024/08/28 14:09:07.145 +00:00] [INFO] [indexnode/indexnode.go:232] [IndexNode] [State=Healthy]
milvus-standalone  | [2024/08/28 14:09:07.145 +00:00] [INFO] [indexnode/indexnode.go:235] ["IndexNode start finished"] []
milvus-standalone  | [2024/08/28 14:09:07.146 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/indexnode-2] [value="{\"ServerID\":2,\"ServerName\":\"indexnode\",\"Address\":\"172.18.0.4:21121\",\"TriggerKill\":true,\"Version\":\"2.4.9\",\"IndexEngineVersion\":{},\"LeaseID\":7587881034873744914,\"HostName\":\"9e0b82804345\",\"EnableDisk\":true}"]
milvus-standalone  | [2024/08/28 14:09:07.146 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexnode] [serverID=2]
milvus-standalone  | [2024/08/28 14:09:07.155 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
milvus-standalone  | [2024/08/28 14:09:07.155 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
milvus-standalone  | [2024/08/28 14:09:07.156 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
milvus-standalone  | [2024/08/28 14:09:07.156 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexcoord] [address=172.18.0.4:13333] [id=2]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=rootcoord] [address=172.18.0.4:53100] [id=2]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Initializing]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=a-bucket] [root=files]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=32]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [querynodev2/server.go:319] ["queryNode init scheduler"] [policy=fifo]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=datacoord] [address=172.18.0.4:13333] [id=2]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [segments/segment_loader.go:548] ["SegmentLoader created"] [ioPoolSize=256]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [datacoord/server.go:345] ["init rootcoord client done"]
milvus-standalone  | I20240828 14:09:07.157577    63 thread_pool.h:172] [KNOWHERE][InitGlobalSearchThreadPool][milvus] Init global search thread pool with size 32
milvus-standalone  | I20240828 14:09:07.157644    63 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
milvus-standalone  | I20240828 14:09:07.157653    63 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX2
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [querynodev2/server.go:230] ["set up knowhere build pool size"] [pool_size=16]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/08/28 14:08:25.622 +00:00] [save=2024/08/28 14:09:10.157 +00:00] [next=2024/08/28 14:09:07.157 +00:00]
milvus-standalone  | [2024/08/28 14:09:07.157 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
milvus-standalone  | I20240828 14:09:07.157780   114 thread_pool.h:53] [KNOWHERE][operator()][knowhere_build0] Successfully set priority of knowhere thread.
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=a-bucket] [root=files]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/server.go:354] ["init chunk manager factory done"]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/server.go:632] ["data coordinator connecting to metadata store"] [metaType=etcd]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/server.go:645] ["data coordinator successfully connected to metadata store"] [metaType=etcd]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/08/28 14:08:25.622 +00:00] [save=2024/08/28 14:09:10.157 +00:00] [next=2024/08/28 14:09:07.157 +00:00]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
milvus-standalone  | I20240828 14:09:07.158383    63 thread_pool.h:188] [KNOWHERE][SetGlobalBuildThreadPoolSize][milvus] Global build thread pool size has already been set to 16
milvus-standalone  | I20240828 14:09:07.158419    63 Common.cpp:62] [SERVER][SetDefaultExecEvalExprBatchSize][milvus] set default expr eval batch size: 8192
milvus-standalone  | I20240828 14:09:07.158455    63 MinioChunkManager.cpp:161] [SERVER][InitSDKAPIDefault][milvus] init aws with log level:fatal
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/index_meta.go:97] ["indexMeta reloadFromKV done"] [duration=291.612µs]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=1]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/analyze_meta.go:70] ["analyzeMeta reloadFromKV done"] [duration=94.876µs]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/partition_stats_meta.go:67] ["DataCoord partitionStatsMeta reloadFromKV done"] [duration=118.31µs]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [datacoord/compaction_task_meta.go:62] ["DataCoord compactionTaskMeta reloadFromKV done"] [duration=135.208µs]
milvus-standalone  | [2024/08/28 14:09:07.158 +00:00] [INFO] [rootcoord/meta_table.go:190] ["collections recovered from db"] [db_name=default] [collection_num=0] [partition_num=0]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/meta.go:229] ["DataCoord meta reloadFromKV done"] [duration=294.467µs]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/server.go:1245] ["all old data node down, enable auto balance!"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [rootcoord/meta_table.go:208] ["RootCoord meta table reload done"] [duration=1.090118ms]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/channel_store_v2.go:71] ["channel store reload done"] [duration=94.836µs]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/server.go:368] ["init datanode cluster done"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/server.go:549] ["DataCoord success to get DataNode sessions"] [sessions={}]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/server.go:570] ["DataCoord Cluster Manager start up"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/channel_manager_v2.go:162] ["starting channel balance loop"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_0]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/channel_manager_v2.go:170] ["cluster start up"] [allNodes="[]"] [legacyNodes="[]"] [oldNodes="[]"] [newOnlines="[]"] [offLines="[]"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [datacoord/server.go:575] ["DataCoord Cluster Manager start up successfully"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_1]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_2]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_3]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_4]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_5]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_6]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_7]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_8]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_9]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_10]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_11]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [grpcclient/client.go:457] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_12]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_13]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_14]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [server/rocksmq_impl.go:413] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_15]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [rootcoord/dml_channels.go:215] ["init dml channels"] [prefix=by-dev-rootcoord-dml] [num=16]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [rootcoord/root_coord.go:451] ["create TimeTick sync done"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [rootcoord/root_coord.go:467] ["init proxy manager done"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [grpcclient/client.go:478] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n  | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n  | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n  | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n  | \t/workspace/source/internal/datacoord/meta.go:234\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n  | \t/workspace/source/internal/datacoord/server.go:659\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n  | \t/workspace/source/internal/datacoord/server.go:658\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [INFO] [rootcoord/root_coord.go:477] ["init credentials done"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [grpcclient/client.go:249] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [grpcclient/client.go:464] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
milvus-standalone  | [2024/08/28 14:09:07.159 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:477\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:470\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:557\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n  | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n  | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n  | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n  | \t/workspace/source/internal/datacoord/meta.go:234\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n  | \t/workspace/source/internal/datacoord/server.go:659\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n  | \t/workspace/source/internal/datacoord/server.go:658\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [rootcoord/meta_table.go:1273] ["role already exists"] [role=admin]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [rootcoord/meta_table.go:1273] ["role already exists"] [role=public]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [datacoord/server.go:375] ["init service discovery done"]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [datacoord/server.go:379] ["init compaction done"]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [datacoord/server.go:384] ["init segment manager done"]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [datacoord/garbage_collector.go:83] ["GC with option"] [enabled=true] [interval=1h0m0s] [scanInterval=168h0m0s] [missingTolerance=24h0m0s] [dropTolerance=3h0m0s]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [rootcoord/root_coord.go:483] ["init rootcoord done"] [nodeID=2] [Address=172.18.0.4:53100]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [rootcoord/service.go:312] ["RootCoord Core start ..."]
milvus-standalone  | [2024/08/28 14:09:07.160 +00:00] [INFO] [datacoord/server.go:399] ["init datacoord done"] [nodeID=2] [Address=172.18.0.4:13333]
milvus-standalone  | [2024/08/28 14:09:07.161 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/rootcoord] [value="{\"ServerID\":2,\"ServerName\":\"rootcoord\",\"Address\":\"172.18.0.4:53100\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.9\",\"IndexEngineVersion\":{},\"LeaseID\":7587881034873744964,\"HostName\":\"9e0b82804345\"}"]
milvus-standalone  | [2024/08/28 14:09:07.161 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=rootcoord] [serverID=2]
milvus-standalone  | [2024/08/28 14:09:07.161 +00:00] [INFO] [rootcoord/root_coord.go:273] ["RootCoord Register Finished"]
milvus-standalone  | [2024/08/28 14:09:07.161 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/indexcoord] [value="{\"ServerID\":2,\"ServerName\":\"indexcoord\",\"Address\":\"172.18.0.4:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.9\",\"IndexEngineVersion\":{},\"LeaseID\":7587881034873744967,\"HostName\":\"9e0b82804345\"}"]
milvus-standalone  | [2024/08/28 14:09:07.161 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexcoord] [serverID=2]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [proxyutil/proxy_watcher.go:96] ["succeed to init sessions on etcd"] [sessions=null] [revision=27]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Healthy]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [WARN] [proxyutil/proxy_client_manager.go:263] ["proxy client is empty, RefreshPrivilegeInfoCache will not send to any client"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [proxyutil/proxy_watcher.go:119] ["start to watch etcd"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [rootcoord/quota_center.go:303] ["Start QuotaCenter"] [collectInterval=3s]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/datacoord] [value="{\"ServerID\":2,\"ServerName\":\"datacoord\",\"Address\":\"172.18.0.4:13333\",\"Exclusive\":true,\"TriggerKill\":true,\"Version\":\"2.4.9\",\"IndexEngineVersion\":{},\"LeaseID\":7587881034873744972,\"HostName\":\"9e0b82804345\"}"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=datacoord] [serverID=2]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/server.go:267] ["DataCoord Register Finished"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/task_scheduler.go:168] ["task scheduler loop start"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/compaction.go:341] ["compactionPlanHandler start loop schedule"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/sync_segments_scheduler.go:71] ["SyncSegmentsScheduler started..."]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/compaction.go:360] ["compactionPlanHandler start loop check"] ["check result interval"=3s]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/index_service.go:114] ["start create index for segment loop..."]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/compaction.go:380] ["compactionPlanHandler start clean check loop"] ["gc interval"=30m0s]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/import_checker.go:73] ["start import checker"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/compaction_trigger_v2.go:118] ["Compaction trigger manager start"]
milvus-standalone  | [2024/08/28 14:09:07.162 +00:00] [INFO] [datacoord/import_scheduler.go:75] ["start import scheduler"]
milvus-standalone  | [2024/08/28 14:09:07.189 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="datacoord-2\n"] [filePath=/tmp/milvus/server_id_8]
milvus-standalone  | [2024/08/28 14:09:07.189 +00:00] [INFO] [datacoord/server.go:413] ["DataCoord startup successfully"]
milvus-standalone  | [2024/08/28 14:09:07.189 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="rootcoord-2\n"] [filePath=/tmp/milvus/server_id_8]
milvus-standalone  | [2024/08/28 14:09:07.190 +00:00] [INFO] [rootcoord/root_coord.go:713] ["rootcoord startup successfully"]
milvus-standalone  | [2024/08/28 14:09:07.190 +00:00] [INFO] [components/root_coord.go:58] ["RootCoord successfully started"]
milvus-standalone  | I20240828 14:09:07.205859    63 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
milvus-standalone  | E20240828 14:09:07.207074    63 MinioChunkManager.cpp:237] [SERVER][PreCheck][milvus] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
milvus-standalone  | [2024/08/28 14:09:07.208 +00:00] [WARN] [initcore/init_core.go:217] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n"]
milvus-standalone  | [2024/08/28 14:09:07.208 +00:00] [ERROR] [querynodev2/server.go:359] ["QueryNode init segcore failed"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:359\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:276\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
milvus-standalone  | [2024/08/28 14:09:07.208 +00:00] [ERROR] [querynode/service.go:136] ["QueryNode init error: "] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:136\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:231\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
milvus-standalone  | [2024/08/28 14:09:07.208 +00:00] [ERROR] [components/query_node.go:56] ["QueryNode starts error"] [error="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]"] [errorVerbose="[UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/initcore.HandleCStatus\n  | \t/workspace/source/internal/util/initcore/init_core.go:218\n  | github.com/milvus-io/milvus/internal/util/initcore.InitRemoteChunkManager\n  | \t/workspace/source/internal/util/initcore/init_core.go:171\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).InitSegcore\n  | \t/workspace/source/internal/querynodev2/server.go:244\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n  | \t/workspace/source/internal/querynodev2/server.go:357\n  | sync.(*Once).doSlow\n  | \t/usr/local/go/src/sync/once.go:74\n  | sync.(*Once).Do\n  | \t/usr/local/go/src/sync/once.go:65\n  | github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n  | \t/workspace/source/internal/querynodev2/server.go:276\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n  | \t/workspace/source/internal/distributed/querynode/service.go:135\n  | github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n  | \t/workspace/source/internal/distributed/querynode/service.go:231\n  | github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n  | \t/workspace/source/cmd/components/query_node.go:55\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:121\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format \"yyyyMMdd\"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:56\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:121"]
milvus-standalone  | panic: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:400, exception:AuthorizationQueryParametersError, errmessage:Unable to parse ExceptionName: AuthorizationQueryParametersError Message: Error parsing the X-Amz-Credential parameter; incorrect date format. This date in the credential must be in the format "yyyyMMdd"., params:params, bucket=a-bucket, prefix=justforconnectioncheck], configuration:[address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
...
milvus-standalone exited with code 134
^CGracefully stopping... (press Ctrl+C again to force)
[+] Stopping 3/3
 ✔ Container milvus-standalone  Stopped                                                                                                                               0.0s
 ✔ Container milvus-minio       Stopped                                                                                                                               0.9s
 ✔ Container milvus-etcd        Stopped                                                                                                                               0.4s
canceled
grebois commented 2 months ago

@xiaofan-luan I can confirm what @SalBakraa mentioned, adding internal: true to the docker compose fixes the problem:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.9
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus
    internal: true

but its a bit tricky to port this to kubernetes without using network policies

zwd1208 commented 2 months ago

@grebois Could you provide the output of the ifconfig and route -n commands from your local machine? I would like to confirm if there are any issues related to routing conflicts.

grebois commented 2 months ago

Sure:

root@d000065:~# ifconfig
br-90565eb617be: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.0.1  netmask 255.255.0.0  broadcast 172.19.255.255
        inet6 fe80::42:a6ff:fe98:81ce  prefixlen 64  scopeid 0x20<link>
        ether 02:42:a6:98:81:ce  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 746 (746.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-a935bb90430b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::42:ddff:fef1:c85  prefixlen 64  scopeid 0x20<link>
        ether 02:42:dd:f1:0c:85  txqueuelen 0  (Ethernet)
        RX packets 48  bytes 3691 (3.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 52  bytes 15198 (15.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.0.1  netmask 255.255.255.0  broadcast 10.42.0.255
        inet6 fe80::9877:77ff:fe56:5619  prefixlen 64  scopeid 0x20<link>
        ether 86:bd:6a:10:23:e8  txqueuelen 1000  (Ethernet)
        RX packets 31859526  bytes 10405774773 (10.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 51822908  bytes 75604367810 (75.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:bc:18:09:88  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.70.183  netmask 255.255.255.0  broadcast 192.168.70.255
        inet6 fe80::5a11:22ff:fec5:262f  prefixlen 64  scopeid 0x20<link>
        ether 58:11:22:c5:26:2f  txqueuelen 1000  (Ethernet)
        RX packets 9173914  bytes 7234770960 (7.2 GB)
        RX errors 0  dropped 20  overruns 0  frame 0
        TX packets 10563911  bytes 9303219073 (9.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.65  netmask 255.255.255.248  broadcast 10.0.0.71
        inet6 fe80::76fe:ceff:feea:d6ec  prefixlen 64  scopeid 0x20<link>
        ether 74:fe:ce:ea:d6:ec  txqueuelen 1000  (Ethernet)
        RX packets 53276287  bytes 73446436543 (73.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24857983  bytes 1643635126 (1.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::7409:e1ff:fece:dccb  prefixlen 64  scopeid 0x20<link>
        ether 76:09:e1:ce:dc:cb  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 5394010  bytes 3958848916 (3.9 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5394010  bytes 3958848916 (3.9 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth55815796: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::5c8c:3dff:fecb:a710  prefixlen 64  scopeid 0x20<link>
        ether 5e:8c:3d:cb:a7:10  txqueuelen 0  (Ethernet)
        RX packets 137907  bytes 29786986 (29.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 141213  bytes 25441500 (25.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth07a5b4f4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::fb:a2ff:fea5:412d  prefixlen 64  scopeid 0x20<link>
        ether 02:fb:a2:a5:41:2d  txqueuelen 0  (Ethernet)
        RX packets 135030  bytes 29958171 (29.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 139516  bytes 24993748 (24.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth0997dad8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::289b:ff:fe7e:8280  prefixlen 64  scopeid 0x20<link>
        ether 2a:9b:00:7e:82:80  txqueuelen 0  (Ethernet)
        RX packets 33  bytes 2362 (2.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5907  bytes 248978 (248.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth17ed3ae: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e4b0:1eff:fec1:ff36  prefixlen 64  scopeid 0x20<link>
        ether e6:b0:1e:c1:ff:36  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33  bytes 2622 (2.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth18a3457: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::28d1:7eff:fe8b:ec07  prefixlen 64  scopeid 0x20<link>
        ether 2a:d1:7e:8b:ec:07  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33  bytes 2622 (2.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth1adc1f29: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::90fe:41ff:feb6:98ab  prefixlen 64  scopeid 0x20<link>
        ether 92:fe:41:b6:98:ab  txqueuelen 0  (Ethernet)
        RX packets 1492730  bytes 1755423374 (1.7 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 999730  bytes 5103677236 (5.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth1bc2057c: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::240c:13ff:fe0d:b938  prefixlen 64  scopeid 0x20<link>
        ether ae:08:27:4a:1d:18  txqueuelen 0  (Ethernet)
        RX packets 172089  bytes 35308612 (35.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 174875  bytes 88512709 (88.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth1f8e9caa: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::38d8:e5ff:fed3:689f  prefixlen 64  scopeid 0x20<link>
        ether 3a:d8:e5:d3:68:9f  txqueuelen 0  (Ethernet)
        RX packets 532233  bytes 817058602 (817.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 550413  bytes 297860937 (297.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth200b76af: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::7092:18ff:febc:3aee  prefixlen 64  scopeid 0x20<link>
        ether 72:92:18:bc:3a:ee  txqueuelen 0  (Ethernet)
        RX packets 768170  bytes 125205031 (125.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 780428  bytes 80606019 (80.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth302cadfe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::5c6a:deff:fece:a5d4  prefixlen 64  scopeid 0x20<link>
        ether 5e:6a:de:ce:a5:d4  txqueuelen 0  (Ethernet)
        RX packets 33  bytes 2362 (2.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5915  bytes 249314 (249.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth3ba7e01e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::f4e4:7ff:fefb:4336  prefixlen 64  scopeid 0x20<link>
        ether 2a:bc:80:86:2a:38  txqueuelen 0  (Ethernet)
        RX packets 514776  bytes 157309719 (157.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 620789  bytes 97477451 (97.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth3bbe6041: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::b0d3:3fff:feda:ab85  prefixlen 64  scopeid 0x20<link>
        ether b2:d3:3f:da:ab:85  txqueuelen 0  (Ethernet)
        RX packets 255734  bytes 59391619 (59.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 345196  bytes 259286133 (259.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth3d6dc2c: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::7001:75ff:fe7a:f270  prefixlen 64  scopeid 0x20<link>
        ether 72:01:75:7a:f2:70  txqueuelen 0  (Ethernet)
        RX packets 301  bytes 32040 (32.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 357  bytes 36746 (36.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth500e0188: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::78bc:34ff:fea9:e734  prefixlen 64  scopeid 0x20<link>
        ether 7a:bc:34:a9:e7:34  txqueuelen 0  (Ethernet)
        RX packets 3922770  bytes 7414160408 (7.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3093172  bytes 2381699773 (2.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth7febe159: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::4b5:eff:fefc:631a  prefixlen 64  scopeid 0x20<link>
        ether 06:b5:0e:fc:63:1a  txqueuelen 0  (Ethernet)
        RX packets 31358163  bytes 57158043738 (57.1 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 50364998  bytes 73693312803 (73.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth853adb2f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::40ce:deff:fe82:50a4  prefixlen 64  scopeid 0x20<link>
        ether 42:ce:de:82:50:a4  txqueuelen 0  (Ethernet)
        RX packets 91436  bytes 55915633 (55.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 102224  bytes 12237406 (12.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth8cced951: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::ec3b:68ff:feef:1df8  prefixlen 64  scopeid 0x20<link>
        ether ee:3b:68:ef:1d:f8  txqueuelen 0  (Ethernet)
        RX packets 1668402  bytes 5883948497 (5.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2302718  bytes 3845550793 (3.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth9a0862c8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::407d:43ff:fe80:f15f  prefixlen 64  scopeid 0x20<link>
        ether 42:7d:43:80:f1:5f  txqueuelen 0  (Ethernet)
        RX packets 5655472  bytes 4086657899 (4.0 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7152662  bytes 56297041076 (56.2 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethab81f3c: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e0d0:f9ff:fe90:4667  prefixlen 64  scopeid 0x20<link>
        ether e2:d0:f9:90:46:67  txqueuelen 0  (Ethernet)
        RX packets 53  bytes 7664 (7.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 93  bytes 19703 (19.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethada34fc6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::9086:92ff:fe36:45cd  prefixlen 64  scopeid 0x20<link>
        ether 92:86:92:36:45:cd  txqueuelen 0  (Ethernet)
        RX packets 72946  bytes 19642885 (19.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 76772  bytes 22940888 (22.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb43bd053: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::a0fd:a7ff:fee0:6033  prefixlen 64  scopeid 0x20<link>
        ether a2:fd:a7:e0:60:33  txqueuelen 0  (Ethernet)
        RX packets 138279  bytes 31308903 (31.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 141547  bytes 25214492 (25.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethf17e5c6d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::246e:a3ff:fe27:6267  prefixlen 64  scopeid 0x20<link>
        ether 26:6e:a3:27:62:67  txqueuelen 0  (Ethernet)
        RX packets 9231  bytes 766087 (766.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13939  bytes 2147344 (2.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethf2ef5f50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::b837:cff:fe69:e2f5  prefixlen 64  scopeid 0x20<link>
        ether ba:37:0c:69:e2:f5  txqueuelen 0  (Ethernet)
        RX packets 141434  bytes 32230190 (32.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 143400  bytes 25648721 (25.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wt0: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1280
        inet 100.81.191.184  netmask 255.255.0.0  destination 100.81.191.184
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 122941  bytes 39203288 (39.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 139208  bytes 77309260 (77.3 MB)
        TX errors 77  dropped 15 overruns 0  carrier 0  collisions 0

root@d000065:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.70.1    0.0.0.0         UG    100    0        0 eno1
10.0.0.64       0.0.0.0         255.255.255.248 U     0      0        0 enp5s0
10.42.0.0       0.0.0.0         255.255.255.0   U     0      0        0 cni0
100.81.0.0      0.0.0.0         255.255.0.0     U     0      0        0 wt0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-a935bb90430b
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-90565eb617be
192.168.70.0    0.0.0.0         255.255.255.0   U     100    0        0 eno1
192.168.70.1    0.0.0.0         255.255.255.255 UH    100    0        0 eno1
root@d000065:~#
zwd1208 commented 2 months ago

@grebois Is the network internal enabled now? Additionally, could you please check by using docker network ls, and then inspect the results with docker network inspect <milvus network id> ? Please replace with the actual network ID of your Milvus Docker network when executing the command. And could you also check the difference of the result when internal disabled ?

grebois commented 2 months ago

@zwd1208 I am remotely connected to this host, so I cannot really disable the network, but even though, we tried without network once and it worked, the outputs I am sharing are with internal: true on docker compose, this is with internet:

root@d000065:~# docker network ls
NETWORK ID     NAME                    DRIVER    SCOPE
a681e9b2b544   bridge                  bridge    local
4c61fb2c2b16   host                    host      local
03ca522b5a4c   milvus                  bridge    local
09f5f99fbbef   none                    null      local
90565eb617be   root_internal_network   bridge    local
a935bb90430b   root_public_network     bridge    local

root@d000065:~# docker network inspect 03ca522b5a4c
[
    {
        "Name": "milvus",
        "Id": "03ca522b5a4c85b03397a146f3e6573f5e6dcb780e34a43695e06bf6a4290698",
        "Created": "2024-08-29T10:59:58.561628253Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "d330b199ec697e6376351a7754e3ef07dc564141051cd3d071374eed34418b1f": {
                "Name": "milvus-etcd",
                "EndpointID": "bc72a071dd18ed854c0e779a02b9ec3bb6e148e81f081181be3f76387e9966da",
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            },
            "d598960c53035ac8f4ed7aa2a4f37f6825b8f9792d192957d1fdf5c490021950": {
                "Name": "milvus-minio",
                "EndpointID": "704b096102191ecb232dc0a6c4bfff8f4944ef67f000392ed00782d51d1fb605",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "root",
            "com.docker.compose.version": "2.29.2"
        }
    }
]

This is with internal: true :

root@d000065:~# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
a681e9b2b544   bridge    bridge    local
4c61fb2c2b16   host      host      local
d59fed8e69a3   milvus    bridge    local
09f5f99fbbef   none      null      local

root@d000065:~# docker network inspect d59fed8e69a3
[
    {
        "Name": "milvus",
        "Id": "d59fed8e69a365e7d52209f1dd2c00823015462d8d907d3d6e385e7bcd6d8be5",
        "Created": "2024-08-29T14:08:37.454833831Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": true,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "62da83eed8d64c9e8acdbb08956f32986c5f2cc22f867a92fe781466fd0c60a9": {
                "Name": "milvus-etcd",
                "EndpointID": "f7c5aee74f8012a1780d69bab9d1ae183b32d497a01d168a3909bd2de608ea44",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "c239f3daf409ef01c5d40c24145ce0e19464cc3f8375296bba170cb58f77b162": {
                "Name": "milvus-standalone",
                "EndpointID": "39e0dc3aab7cb4757ba71b716be239137ecbf71a76f4aeba7ca022f8bae1edaa",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            },
            "f1506755c60bf8f77acbc1dc1bf72b062830623d7d4162f60e44d382fd3e554e": {
                "Name": "milvus-minio",
                "EndpointID": "2f81811745c79e30cd12ea24fb795f435ed500d9ba8dccf95439987ee6a550fa",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "root",
            "com.docker.compose.version": "2.29.2"
        }
    }
]
grebois commented 2 months ago

Same problem @ 2.4.10 in docker-compose, helm chart and milvus-operator.

xiaofan-luan commented 2 months ago

Quick Question, Are you trying to deploy your milvus on UAE and Europe but share the same S3/Minio?

We recommend is that you need separate storage for each of your env.

Once you enable network, then ntp triggered and change the time to local time so there is a clock skew among different regions.

Again this is a rough guess, I think you want to first make your your milvus, etcd and object storage is on same region and same time

zwd1208 commented 2 months ago

@grebois I was unable to reproduce this issue. Could you provide the configurations for deployment using Docker Compose, Helm Chart, and Operator?

grebois commented 2 months ago

Quick Question, Are you trying to deploy your milvus on UAE and Europe but share the same S3/Minio?

We recommend is that you need separate storage for each of your env.

Once you enable network, then ntp triggered and change the time to local time so there is a clock skew among different regions.

Again this is a rough guess, I think you want to first make your your milvus, etcd and object storage is on same region and same time

No, I tried deploying it standalone, each with it's "own" minio, as part of the docker-compose.

@grebois I was unable to reproduce this issue. Could you provide the configurations for deployment using Docker Compose, Helm Chart, and Operator?

No changes, all default configuration, I can give you access to one of these machines for you to try it.

zwd1208 commented 2 months ago

No changes, all default configuration, I can give you access to one of these machines for you to try it.

@grebois Alright, you can send the access information to my email at weida.zhu@zilliz.com

grebois commented 2 months ago

@zwd1208 Sent!

zwd1208 commented 2 months ago

@grebois I've find the root cause. First, I'd like to confirm what type of VM your machine is？

And the reason is: Milvus uses the AWS SDK to access MinIO. The AWS SDK attempts to access the local address http://169.254.169.254 during initialization to retrieve the region information. But this address also exists on your machine, it fails to return the region information correctly, causing the AWS SDK to make an error when constructing the access URL. The log information below also indicates this reason.

img_v3_02ei_ff08e645-104f-469f-94ff-24bba65cff6g This also explains why it works correctly when the network internal: true applied, as in this scenario, the AWS SDK is unable to access the address 169.254.169.254.

To resolve this issue, you can add the environment variable MINIO_REGION: us-east-1 in the Milvus environment.

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.10
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
      MINIO_REGION: us-east-1  # add default region
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"
grebois commented 2 months ago

@zwd1208 thanks a lot for your outstanding support, the IP is not actually configured, this is all bare metal, it's just the admin panel IP of a backup 5g Router we don't really use, Huawei Hg8245h, just to give a bit more background to other people that might face the same problem, will try setting the MINIO_REGION: us-east-1 and also limit the access to this IP with iptables, great work, highly appreciated.

xiaofan-luan commented 2 months ago

what a bug! Thanks for the support! @zwd1208