search

Lucretius / vault_raft_snapshot_agent

⛔️ DEPRECATED ⛔️ An agent which provides periodic snapshotting capabilities of Vault's Raft backend
MIT License
78 stars 42 forks source link

[FEAT] snapshot upload to azure china blob #26

Open vikramhansawat opened 1 year ago

vikramhansawat commented 1 year ago

This feature request will allow vault_raft_snapshot_agent to upload the snapshots to AzureChinaCloud. Currently only AzurePublicCloud is supported. FYI, you can see that by default blob.core.windows.net is used as a blob storage url:

Nov 18 10:24:27 server000000 vault_raft_snapshot_agent[20019]: 2022/11/18 10:24:27 Reading configuration...
Nov 18 10:24:27 000000 vault_raft_snapshot_agent[20019]: 2022/11/18 10:24:27 Successfully created local snapshot to /data/vault/raft/snapshots/raft_snapshot-1668767067323825425.snap
Nov 18 10:24:27 000000 /usr/bin/vault_raft_snapshot_agent[20019]: 2022/11/18 10:24:27 ==> REQUEST/RESPONSE (Try=1/55.398133ms, OpTime=55.540135ms) -- REQUEST ERROR
                                                                                                        PUT https://XXX.blob.core.windows.net/serverbackup/raft_snapshot-1668767067323825425.snap?timeout=61
                                                                                                        Authorization: REDACTED
                                                                                                        Content-Length: [30667]
                                                                                                        User-Agent: [Azure-Storage/0.7 (go1.17.12; linux)]
                                                                                                        X-Ms-Blob-Cache-Control: []
                                                                                                        X-Ms-Blob-Content-Disposition: []
                                                                                                        X-Ms-Blob-Content-Encoding: []
                                                                                                        X-Ms-Blob-Content-Language: []
                                                                                                        X-Ms-Blob-Content-Type: []
                                                                                                        X-Ms-Blob-Type: [BlockBlob]
                                                                                                        X-Ms-Client-Request-Id: [66a7f0ae-243f-4d23-6256-64598cd1338c]
                                                                                                        X-Ms-Version: [2018-11-09]
                                                                                                        x-ms-date: [Fri, 18 Nov 2022 10:24:27 GMT]
                                                                                                        --------------------------------------------------------------------------------
                                                                                                        ERROR:
                                                                                                     -> github.com/Azure/azure-pipeline-go/pipeline.NewError, /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/error.go:154
                                                                                                     HTTP request failed

                                                                                                     Put "https://XXXX.blob.core.windows.net/vaultserverbackup/raft_snapshot-1668767067323825425.snap?timeout=61": dial tcp: lookup XXXX.blob.core.windo

                                                                                                     goroutine 1 [running]:
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.stack()
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_request_log.go:146 +0x65
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.NewRequestLogPolicyFactory.func1.1({0xdfe988, 0xc000034cc0}, {0xc00093e368})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_request_log.go:96 +0x733
                                                                                                     github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc506c0, {0xdfe988, 0xc000034cc0}, {0xd})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/core.go:43 +0x2f
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.(*SharedKeyCredential).New.func1({0xdfe988, 0xc000034cc0}, {0xdf8475800})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_credential_shared_key.go:66 +0x3cd
                                                                                                     github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xdfe918, {0xdfe988, 0xc000034cc0}, {0x7})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/core.go:43 +0x2f
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.NewRetryPolicyFactory.func1.1({0xdfe918, 0xc0000a6180}, {0xc00093e668})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_retry.go:204 +0x96e
                                                                                                     github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc09500, {0xdfe918, 0xc0000a6180}, {0x16})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/core.go:43 +0x2f
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1({0xdfe918, 0xc0000a6180}, {0xc00093e708})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_unique_request_id.go:19 +0x163
                                                                                                     github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc09500, {0xdfe918, 0xc0000a6180}, {0xa})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/core.go:43 +0x2f
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1({0xdfe918, 0xc0000a6180}, {0xc000010430})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zc_policy_telemetry.go:34 +0x11c
                                                                                                     github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0000735c0, {0xdfe918, 0xc0000a6180}, {0x41a025})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/core.go:43 +0x2f
                                                                                                     github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do(0xc5c0c0, {0xdfe918, 0xc0000a6180}, {0xdece40, 0xc00023f6b0}, {0xc000038d88})
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.1/pipeline/core.go:129 +0x54
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.blockBlobClient.Upload({{{{0xc000038d80, 0x5}, {0x0, 0x0}, 0x0, {0xc000038d88, 0x25}, {0xc0000cc040, 0x39}, {0x0, ...}, ...
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/zz_generated_block_blob.go:436 +0x4b8
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.BlockBlobURL.Upload({{{{{...}, {...}}}}, {{{{...}, {...}, 0x0, {...}, {...}, {...}, _, {...}, ...}, ...}}}, ...)
                                                                                                             /root/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.8.0/azblob/url_block_blob.go:64 +0x1fc
                                                                                                     github.com/Azure/azure-storage-blob-go/azblob.(*uploadStreamToBlockBlobOptions).end(0xc0000e2000, {0xdfe918, 0xc0000a6180})
vikramhansawat commented 1 year ago

@Lucretius can you please review the pull request?