Can't connect to Azure Storage using private endpoint

[mark4409]

Using Thanos 0.26.0 on the latest docker.

I'm having difficulty using an Azure Storage account using private endpoints. DNS is configured correctly and the log file does show it connecting to the correct IP for the private endpoint. It fails though with the message:

thanos-thanos-sidecar-2-1 | level=info ts=2022-07-04T09:47:01.720820509Z caller=factory.go:49 msg="loading bucket configuration" thanos-thanos-sidecar-2-1 | level=error ts=2022-07-04T09:47:45.510628057Z caller=main.go:132 err="Put \"https://xxxxxxxxxxx.blob.core.windows.net/metrics?restype=container&timeout=61\": read tcp 172.20.0.4:38996->10.16.76.4:443: read: connection reset by peer\nAzure API return unexpected error: *url.Error\n\ngithub.com/thanos-io/thanos/pkg/objstore/azure.NewBucketWithConfig\n\t/app/pkg/objstore/azure/azure.go:199\ngithub.com/thanos-io/thanos/pkg/objstore/azure.NewBucket\n\t/app/pkg/objstore/azure/azure.go:185\ngithub.com/thanos-io/thanos/pkg/objstore/client.NewBucket\n\t/app/pkg/objstore/client/factory.go:67\nmain.runSidecar\n\t/app/cmd/thanos/sidecar.go:306\nmain.registerSidecar.func1\n\t/app/cmd/thanos/sidecar.go:73\nmain.main\n\t/app/cmd/thanos/main.go:130\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\ncreate AZURE client\ngithub.com/thanos-io/thanos/pkg/objstore/client.NewBucket\n\t/app/pkg/objstore/client/factory.go:82\nmain.runSidecar\n\t/app/cmd/thanos/sidecar.go:306\nmain.registerSidecar.func1\n\t/app/cmd/thanos/sidecar.go:73\nmain.main\n\t/app/cmd/thanos/main.go:130\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\npreparing sidecar command failed\nmain.main\n\t/app/cmd/thanos/main.go:132\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581" I have tried connecting to the container using Azure Storage Explorer using the provided url and access key, and it connects fine.

If I remove the private endpoint so that it sits again on a public IP address, thanos connects to it without an issue. The url and the IP address in the logline are correct so that shows it is resolving correctly.

Is it a case that private endpoints are not yet supported?

[marevers]

What does your object store configuration file look like? It looks like it's trying to connect to the regular host name, rather than the private endpoint. Normally, that would be https://xxxxxxxxxxxxx.privatelink.blob.core.windows.net.

Config:

    type: AZURE
    config:
      storage_account: "account"
      storage_account_key: "key"
      container: "container"
      endpoint: "privatelink.blob.core.windows.net"

[mark4409]

The endpoint is correct - it uses DNS forwarding to resolve to the private endpoint - https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns#on-premises-workloads-using-a-dns-forwarder

The IP address that its trying to connect to suggests that this is working correctly. Other than that, the configuration file looks exactly as you've displayed and works fine if I remove the private endpoint so that it sits on a public address.

I tested the private endpoint connection on a windows device on the same network using storage explorer and the same account key that I set in the config file, and it connected fine to the endpoint.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.