[Bug]: Aliyun cloud provider still not working in v2.3.0-beta

elonzh commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version: v2.3.0-beta
- Deployment mode(standalone or cluster): standalone (deployed by helm chart)
- MQ type(rocksmq, pulsar or kafka): rocksmq

Current Behavior

Milvus server complains ["QueryNode init error: "] [error="attempt #0: Please use virtual hosted style to access.]

Expected Behavior

aliyun clouldProvider works as normal

Steps To Reproduce

Using `aliyun` as cloudProvider

externalS3:
  enabled: true
  cloudProvider: "aliyun"
  host: "oss-cn-shanghai-internal.aliyuncs.com"
  accessKey: "..."
  secretKey: "..."
  useSSL: false
  bucketName: "..."
  rootPath: "milvus"

Milvus Log

...
[2023/03/21 04:34:17.251 +00:00] [DEBUG] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord]
[2023/03/21 04:34:17.251 +00:00] [DEBUG] [client/client.go:100] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:159] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:159] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:319] ["ClientBase ReCall grpc first call get error "] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/tracer/stack_trace.go:51 github.com/milvus-io/milvus/internal/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:318 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:504 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:195 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:319] ["ClientBase ReCall grpc first call get error "] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/tracer/stack_trace.go:51 github.com/milvus-io/milvus/internal/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:318 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:378 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:174 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/03/21 04:34:17.251 +00:00] [DEBUG] [client/client.go:100] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2023/03/21 04:34:17.251 +00:00] [DEBUG] [client/client.go:94] ["QueryCoordClient msess key not existed"] [key=querycoord]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:159] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:328] ["ClientBase ReCall grpc second call get error"] [role=querycoord] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/tracer/stack_trace.go:51 github.com/milvus-io/milvus/internal/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:378 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:174 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/03/21 04:34:17.251 +00:00] [WARN] [grpcclient/client.go:159] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2023/03/21 04:34:17.252 +00:00] [WARN] [grpcclient/client.go:328] ["ClientBase ReCall grpc second call get error"] [role=datacoord] [error="err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/tracer/stack_trace.go:51 github.com/milvus-io/milvus/internal/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:504 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:195 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/03/21 04:34:17.252 +00:00] [WARN] [rootcoord/quota_center.go:129] ["quotaCenter sync metrics failed"] [error="err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/tracer/stack_trace.go:51 github.com/milvus-io/milvus/internal/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:378 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:174 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n"]
[2023/03/21 04:34:17.264 +00:00] [ERROR] [datacoord/server.go:445] ["chunk manager init failed"] [error="attempt #0: Please use virtual hosted style to access.: attempt #1: Please use virtual hosted style to access.: attempt #2: Please use virtual hosted style to access.: attempt #3: Please use virtual hosted style to access.: attempt #4: Please use virtual hosted style to access.: attempt #5: Please use virtual hosted style to access.: attempt #6: Please use virtual hosted style to access.: attempt #7: Please use virtual hosted style to access.: attempt #8: Please use virtual hosted style to access.: attempt #9: Please use virtual hosted style to access.: attempt #10: Please use virtual hosted style to access.: attempt #11: Please use virtual hosted style to access.: attempt #12: Please use virtual hosted style to access.: attempt #13: Please use virtual hosted style to access.: attempt #14: Please use virtual hosted style to access.: attempt #15: Please use virtual hosted style to access.: attempt #16: Please use virtual hosted style to access.: attempt #17: Please use virtual hosted style to access.: attempt #18: Please use virtual hosted style to access.: attempt #19: Please use virtual hosted style to access."] [stack="github.com/milvus-io/milvus/internal/datacoord.(*Server).newChunkManagerFactory\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:445\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:317\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:307\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:104\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:209\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:91"]
[2023/03/21 04:34:17.265 +00:00] [ERROR] [datacoord/service.go:105] ["dataCoord init error"] [error="attempt #0: Please use virtual hosted style to access.: attempt #1: Please use virtual hosted style to access.: attempt #2: Please use virtual hosted style to access.: attempt #3: Please use virtual hosted style to access.: attempt #4: Please use virtual hosted style to access.: attempt #5: Please use virtual hosted style to access.: attempt #6: Please use virtual hosted style to access.: attempt #7: Please use virtual hosted style to access.: attempt #8: Please use virtual hosted style to access.: attempt #9: Please use virtual hosted style to access.: attempt #10: Please use virtual hosted style to access.: attempt #11: Please use virtual hosted style to access.: attempt #12: Please use virtual hosted style to access.: attempt #13: Please use virtual hosted style to access.: attempt #14: Please use virtual hosted style to access.: attempt #15: Please use virtual hosted style to access.: attempt #16: Please use virtual hosted style to access.: attempt #17: Please use virtual hosted style to access.: attempt #18: Please use virtual hosted style to access.: attempt #19: Please use virtual hosted style to access."] [stack="github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:105\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:209\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:91"]
[2023/03/21 04:34:17.323 +00:00] [ERROR] [querynode/query_node.go:276] ["QueryNode init vector storage failed"] [error="attempt #0: Please use virtual hosted style to access.: attempt #1: Please use virtual hosted style to access.: attempt #2: Please use virtual hosted style to access.: attempt #3: Please use virtual hosted style to access.: attempt #4: Please use virtual hosted style to access.: attempt #5: Please use virtual hosted style to access.: attempt #6: Please use virtual hosted style to access.: attempt #7: Please use virtual hosted style to access.: attempt #8: Please use virtual hosted style to access.: attempt #9: Please use virtual hosted style to access.: attempt #10: Please use virtual hosted style to access.: attempt #11: Please use virtual hosted style to access.: attempt #12: Please use virtual hosted style to access.: attempt #13: Please use virtual hosted style to access.: attempt #14: Please use virtual hosted style to access.: attempt #15: Please use virtual hosted style to access.: attempt #16: Please use virtual hosted style to access.: attempt #17: Please use virtual hosted style to access.: attempt #18: Please use virtual hosted style to access.: attempt #19: Please use virtual hosted style to access."] [stack="github.com/milvus-io/milvus/internal/querynode.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/query_node.go:276\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynode.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/query_node.go:251\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:122\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:203\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:91"]
[2023/03/21 04:34:17.324 +00:00] [ERROR] [querynode/service.go:123] ["QueryNode init error: "] [error="attempt #0: Please use virtual hosted style to access.: attempt #1: Please use virtual hosted style to access.: attempt #2: Please use virtual hosted style to access.: attempt #3: Please use virtual hosted style to access.: attempt #4: Please use virtual hosted style to access.: attempt #5: Please use virtual hosted style to access.: attempt #6: Please use virtual hosted style to access.: attempt #7: Please use virtual hosted style to access.: attempt #8: Please use virtual hosted style to access.: attempt #9: Please use virtual hosted style to access.: attempt #10: Please use virtual hosted style to access.: attempt #11: Please use virtual hosted style to access.: attempt #12: Please use virtual hosted style to access.: attempt #13: Please use virtual hosted style to access.: attempt #14: Please use virtual hosted style to access.: attempt #15: Please use virtual hosted style to access.: attempt #16: Please use virtual hosted style to access.: attempt #17: Please use virtual hosted style to access.: attempt #18: Please use virtual hosted style to access.: attempt #19: Please use virtual hosted style to access."] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:203\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:91"]
panic: attempt #0: Please use virtual hosted style to access.: attempt #1: Please use virtual hosted style to access.: attempt #2: Please use virtual hosted style to access.: attempt #3: Please use virtual hosted style to access.: attempt #4: Please use virtual hosted style to access.: attempt #5: Please use virtual hosted style to access.: attempt #6: Please use virtual hosted style to access.: attempt #7: Please use virtual hosted style to access.: attempt #8: Please use virtual hosted style to access.: attempt #9: Please use virtual hosted style to access.: attempt #10: Please use virtual hosted style to access.: attempt #11: Please use virtual hosted style to access.: attempt #12: Please use virtual hosted style to access.: attempt #13: Please use virtual hosted style to access.: attempt #14: Please use virtual hosted style to access.: attempt #15: Please use virtual hosted style to access.: attempt #16: Please use virtual hosted style to access.: attempt #17: Please use virtual hosted style to access.: attempt #18: Please use virtual hosted style to access.: attempt #19: Please use virtual hosted style to access.

goroutine 260 [running]:
panic({0x35da060, 0xc001480b40})
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc0003d5f20 sp=0xc0003d5e60 pc=0x125fcf7
github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run(0xc000bad540?)
    /go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:55 +0x56 fp=0xc0003d5f58 sp=0xc0003d5f20 pc=0x2e5c2d6
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
    /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:91 +0x15c fp=0xc0003d5fe0 sp=0xc0003d5f58 pc=0x2e5ea1c
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003d5fe8 sp=0xc0003d5fe0 pc=0x12950c1
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...]
    /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:78 +0x151

goroutine 1 [semacquire]:
runtime.gopark(0x1e44b9d?, 0x519f1a0?, 0x40?, 0xc2?, 0x1235e25?)
    /usr/local/go/src/runtime/proc.go:361 +0xd6 fp=0xc001425748 sp=0xc001425728 pc=0x1262f56
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:367
runtime.semacquire1(0xc0004efdf8, 0x90?, 0x1, 0x0)
    /usr/local/go/src/runtime/sema.go:144 +0x1f3 fp=0xc0014257b0 sp=0xc001425748 pc=0x12740d3
sync.runtime_Semacquire(0x125e59f?)
    /usr/local/go/src/runtime/sema.go:56 +0x25 fp=0xc0014257e0 sp=0xc0014257b0 pc=0x1290d65
sync.(*WaitGroup).Wait(0xc000bad540?)
    /usr/local/go/src/sync/waitgroup.go:136 +0x52 fp=0xc001425808 sp=0xc0014257e0 pc=0x12a3872
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc001425e58, 0x1, {0xc000bf0aa0?, 0xe?})
    /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:310 +0x965 fp=0xc001425e08 sp=0xc001425808 pc=0x2e5e225
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xc000c02930, {0xc00004e090?, 0x3, 0x3}, 0xc0005fa3c0)
    /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:112 +0x66e fp=0xc001425ee0 sp=0xc001425e08 pc=0x2e6a9ae
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc00004e090?, 0x3, 0x3})
    /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x21e fp=0xc001425f58 sp=0xc001425ee0 pc=0x2e6a23e
main.main()
    /go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e fp=0xc001425f80 sp=0xc001425f58 pc=0x2e6d08e
runtime.main()
    /usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc001425fe0 sp=0xc001425f80 pc=0x1262b92
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001425fe8 sp=0xc001425fe0 pc=0x12950c1
...

Anything else?

No response

xiaofan-luan commented 1 year ago

Thanks for the feedback. /assign @haorenfsa could you help on investigation

sre-ci-robot commented 1 year ago

@xiaofan-luan: GitHub didn't allow me to assign the following users: could, you, help, on, investigation.

Note that only milvus-io members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/milvus-io/milvus/issues/22880#issuecomment-1477348389): >Thanks for the feedback. >/assign @haorenfsa could you help on investigation Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

yanliang567 commented 1 year ago

/assign @haorenfsa /unassign

haorenfsa commented 1 year ago

Hi, @elonzh Here's my configuration works in lastest helm chart (4.0.12):

externalS3:
  enabled: true
  cloudProvider: "aliyun"
  host: "oss-cn-shanghai-internal.aliyuncs.com"
  accessKey: "..."
  secretKey: "..."
  useSSL: true
  port: 443
  bucketName: "..."
  rootPath: "milvus"

make sure you have specify the port

You can also use 80 port if you prefer:

externalS3:
  # ...
  useSSL: false
  port: 80

Sry for the mess of the externalS3 configuration, we're seeking a way to make it better #22185

haorenfsa commented 1 year ago

/unassign

haorenfsa commented 1 year ago

@yanliang567 this can be closed

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

milvus-io / milvus