search

thanos-io / objstore

Go module providing unified interface and efficient clients to work with various object storage providers until like GCS, S3, Azure, SWIFT, COS and more.
Apache License 2.0
111 stars 76 forks source link

azure v0.34.0 - gopanic: send on closed channel #101

Closed ahurtaud closed 8 months ago

ahurtaud commented 8 months ago

Hello,

with the latest version of thanos v0.34.0 running a compactor on an Azure Container storage. We are seeing from time to time some go panics while uploading the blocks. I think this is new to v0.34.0 and was working ok with v0.33.x

We are running in the official docker image with the following config:

args 

      --log.level=debug
      --data-dir=/data
      --objstore.config=
        type: AZURE
        config:
          container: "argos-argos-amadeus-com"
          endpoint: "privatelink.blob.core.windows.net"
          storage_account: "$(OBJSTORE_ACCESS_KEY)"
          storage_account_key: "$(OBJSTORE_SECRET_KEY)"
          max_retries: 0
          http_config:
            insecure_skip_verify: true
      --wait
      --wait-interval=30m
      --compact.cleanup-interval=30m
      --compact.concurrency=8
      --downsample.concurrency=2
      --compact.blocks-fetch-concurrency=1
      --block-files-concurrency=1
      --block-meta-fetch-concurrency=5
      --web.disable
      --compact.progress-interval=10m
      --http-address=0.0.0.0:8080
      --consistency-delay=30m
      --retention.resolution-raw=35d
      --retention.resolution-5m=100d
      --retention.resolution-1h=100d
      --compact.skip-block-with-out-of-order-chunks
Go Panic stack trace 

ts=2024-02-26T12:19:24.936631613Z caller=downsample.go:391 level=info msg="downsampled block" from=01HQ3G1QHTRFY7KSAJ2RJRXXT4 to=01HQJNH9DAP3BQCEKK8HT619MG duration=2m36.3914078s duration_ms=156391
ts=2024-02-26T12:19:37.839394217Z caller=azure.go:311 level=debug msg="uploading blob" blob=01HQJNH9DAP3BQCEKK8HT619MG/chunks/000001
panic: send on closed channel

goroutine 1233414 [running]:
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.staticBuffer.Put(...)
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:83
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).write(0xc0013c9400, {{0xc01d0a2000, 0x300000, 0x300000}, {0xc004e3c5a0, 0x58}, 0x300000})
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:179 +0x39a
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).sendChunk.func1()
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:153 +0xb6
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer.func1()
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:62 +0x35
created by github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer in goroutine 1233411
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:60 +0xc6
panic: send on closed channel

goroutine 1233413 [running]:
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.staticBuffer.Put(...)
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:83
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).write(0xc0013c9400, {{0xc01c5a8000, 0x300000, 0x300000}, {0xc00529a240, 0x58}, 0x300000})
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:191 +0x37c
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).sendChunk.func1()
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:153 +0xb6
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer.func1()
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:62 +0x35
created by github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer in goroutine 1233411
    /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:60 +0xc6
MichaHoffmann commented 8 months ago

Oh, it looks like the azblob library is very old. It could be this issue; in fact the stack traces look pretty similar.

ahurtaud commented 8 months ago

@MichaHoffmann Hello Michael, can we please make sure Thanos updates the go modules to get this fix in the next thanos versions? I think the dependency is not updated. I dont have right now the project setup on my localdev, so i guess its faster for anyone of you to upgrade and create a Pull request there. If it's not that easy for you, tell me and I'll try to find some time to do it.

Thank you :)

https://github.com/thanos-io/thanos/blob/main/go.mod#L67

yeya24 commented 7 months ago

@ahurtaud I just openend a pr to bump up the version. https://github.com/thanos-io/thanos/pull/7228

Sometimes maintainers might miss updating the dependency on the main repo. When anyone notices it, feel free to upgrade the go mod on Thanos and open a pr. Leaving a comment in the issue helps as well.