[Bug]: can't config storage to s3 when run in docker-compose

[zhangcheng-vitongue]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

I had followed the doc to configure docker with s3 storage: 

https://milvus.io/docs/configure-docker.md
https://milvus.io/docs/deploy_s3.md

Current Behavior

[2023/11/07 07:31:13.479 +00:00] [WARN] [storage/minio_object_storage.go:115] ["failed to check blob bucket exist"] [bucket=wired-milvus-df962b7] [error="Get \"https://minio:9000/wired-milvus-df962b7/?location=\": http: server gave HTTP response to HTTPS client"]
[2023/11/07 07:31:13.479 +00:00] [ERROR] [storage/minio_object_storage.go:132] ["retry func failed"] ["retry time"=8] [error="Get \"https://minio:9000/wired-milvus-df962b7/?location=\": http: server gave HTTP response to HTTPS client"] [stack="github.com/milvus-io/milvus/internal/storage.newMinioClient\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_object_storage.go:132\ngithub.com/milvus-io/milvus/internal/storage.newMinioChunkManagerWithConfig\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_chunk_manager.go:77\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).newChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:52\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:61\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).newChunkManagerFactory\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:460\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:335\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:323\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:126\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:249\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:50\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
[2023/11/07 07:31:13.479 +00:00] [WARN] [storage/minio_object_storage.go:115] ["failed to check blob bucket exist"] [bucket=wired-milvus-df962b7] [error="Get \"https://minio:9000/wired-milvus-df962b7/?location=\": http: server gave HTTP response to HTTPS client"]
[2023/11/07 07:31:13.479 +00:00] [ERROR] [storage/minio_object_storage.go:132] ["retry func failed"] ["retry time"=8] [error="Get \"https://minio:9000/wired-milvus-df962b7/?location=\": http: server gave HTTP response to HTTPS client"] [stack="github.com/milvus-io/milvus/internal/storage.newMinioClient\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_object_storage.go:132\ngithub.com/milvus-io/milvus/internal/storage.newMinioChunkManagerWithConfig\n\t/go/src/github.com/milvus-io/milvus/internal/storage/minio_chunk_manager.go:77\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).newChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:52\ngithub.com/milvus-io/milvus/internal/storage.(*ChunkManagerFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/storage/factory.go:61\ngithub.com/milvus-io/milvus/internal/util/dependency.(*DefaultFactory).NewPersistentStorageChunkManager\n\t/go/src/github.com/milvus-io/milvus/internal/util/dependency/factory.go:150\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:298\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:267\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:128\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:223\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

I think you didn't enable ssl on minio but milvus connect with SSL

[zhangcheng-vitongue]

@xiaofan-luan

minio:
  address: s3.ap-southeast-1.amazonaws.com
  port: 443
  accessKeyID: ******
  secretAccessKey: *******
  useSSL: true
  bucketName: wired-milvus-df962b7
  rootPath: "" # The root path where the message is stored in MinIO/S3
  # Whether to useIAM role to access S3/GCS instead of access/secret keys
  # For more information, refer to
  # aws: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html
  # gcp: https://cloud.google.com/storage/docs/access-control/iam
  # aliyun (ack): https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/use-rrsa-to-enforce-access-control
  # aliyun (ecs): https://www.alibabacloud.com/help/en/elastic-compute-service/latest/attach-an-instance-ram-role
  useIAM: false
  # Cloud Provider of S3. Supports: "aws", "gcp", "aliyun".
  # You can use "aws" for other cloud provider supports S3 API with signature v4, e.g.: minio
  # You can use "gcp" for other cloud provider supports S3 API with signature v2
  # You can use "aliyun" for other cloud provider uses virtual host style bucket
  # When useIAM enabled, only "aws", "gcp", "aliyun" is supported for now
  cloudProvider: aws
  # Custom endpoint for fetch IAM role credentials. when useIAM is true & cloudProvider is "aws".
  # Leave it empty if you want to use AWS default endpoint
  iamEndpoint:
  # Log level for aws sdk log.
  # Supported level:  off, fatal, error, warn, info, debug, trace
  logLevel: ddebug
  # Cloud data center region
  region: ""
  # Cloud whether use virtual host bucket mode
  useVirtualHost: false
  # timeout for request time in milliseconds
  requestTimeoutMs: 3000

the above is my milvus.yaml config, it enabled the ssl

[abeatbeyondlab]

docker compose install seems not working ... having the same issue

[LoveEachDay]

Have you removed the default env MINIO_ADDRESS in your docker-compose.yml? By default the environment variables has higher precedence than milvus.yaml.

[zhangcheng-vitongue]

@LoveEachDay no,

the docker-compose.yml contains this line:

MINIO_ADDRESS: minio:9000

[yanliang567]

/assign @LoveEachDay /unassign

[LoveEachDay]

@zhangcheng-vitongue You need to remove MINIO_ADDRESS: minio:9000 this line before running docker-compose up .

[xiaofan-luan]

@zhangcheng-vitongue You need to remove MINIO_ADDRESS: minio:9000 this line before running docker-compose up .

We should remove it by default and it is very misleading

[LoveEachDay]

No, by default we will local minio with docker-compose. We'd specify the minio address explicitly.

And we'd better refine our milvus doc at https://milvus.io/docs/deploy_s3.md#Configure-S3-with-Docker-Compose