I am hoping someone can point us in the right direction for this problem.
We recently upgraded to the latest Microsoft go 1.23.2 and azure SDK v1.16.0.
Previously we were using 1.22.5 and v1.14.0, so not very old.
After upgrading we have noticed that one of our automated tests has started failing about 20% of the time with a go panic.
This panic is from an "index out of range" error deep in the stack upon calling CommitBlockList.
Here is the panic:
panic: runtime error: index out of range [12] with length 12
:1 +0x31 fp=0xc0007db450 sp=0xc0007db428 pc=0x962151
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc0005a5640)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144 +0xf0 fp=0xc0007db490 sp=0xc0007db450 pc=0x92f2b0
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.includeResponsePolicy(0xc0005a5640)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/runtime/policy_include_response.go:19 +0x1c fp=0xc0007db4d0 sp=0xc0007db490 pc=0x95d21c
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.PolicyFunc.Do(0xcfaee0?, 0xc0005ca2d0?)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:214 +0x19 fp=0xc0007db4e8 sp=0xc0007db4d0 pc=0x92f4b9
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc0005a5580)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144 +0xf0 fp=0xc0007db528 sp=0xc0007db4e8 pc=0x92f2b0
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.Pipeline.Do({{0xc0000cec00?, 0x0?, 0x0?}}, 0xc0003ee888?)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/pipeline.go:76 +0x45 fp=0xc0007db548 sp=0xc0007db528 pc=0x92ecc5
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.(*BlockBlobClient).CommitBlockList(0xc0000137e8, {0x1077320?, 0x18177e0?}, {{0x0, 0x0, 0x0}, {0xc0003ee888, 0x1, 0x1}, {0x0, ...}}, ...)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/generated/zz_blockblob_client.go:51 +0xc6 fp=0xc0007db668 sp=0xc0007db548 pc=0x96ed06
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*Client).CommitBlockList(0xc000013800, {0x1077320, 0x18177e0}, {0xc000a49070, 0x1, 0xe6?}, 0xc000683a98)
/nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/blockblob/client.go:285 +0x3bd fp=0xc0007db8e8 sp=0xc0007db668 pc=0x98c15d
main.(*OCSBlockBlob).commitBlockList(0xc000334288, 0xc0006d2908, 0xc00078e4e8)
/net/code/nbcm_builds/DD/20.5.0.1/dedupe/ocsd/ocsd/linuxR_x86/azure_blob_ops.go:1691 +0xba9 fp=0xc0007dbc88 sp=0xc0007db8e8 pc=0xadf8c9
main.commitBlockList(0xc0007b60a0, 0xc000631cf8)
/net/code/nbcm_builds/DD/20.5.0.1/dedupe/ocsd/ocsd/linuxR_x86/request_handle.go:931 +0x555 fp=0xc0007dbee0 sp=0xc0007dbc88 pc=0xaaea35
main.requestWorker(0xc0008a2cb0, 0xc000631cf8)
/net/code/nbcm_builds/DD/20.5.0.1/dedupe/ocsd/ocsd/linuxR_x86/main.go:48 +0x7dd fp=0xc0007dbfc0 sp=0xc0007dbee0 pc=0xb0951d
main.dispatchRequests.gowrap1()
/net/code/nbcm_builds/DD/20.5.0.1/dedupe/ocsd/ocsd/linuxR_x86/main.go:212 +0x25 fp=0xc0007dbfe0 sp=0xc0007dbfc0 pc=0xb0c665
runtime.goexit({})
/net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0007dbfe8 sp=0xc0007dbfe0 pc=0x487121
created by main.dispatchRequests in goroutine 2378
/net/code/nbcm_builds/DD/20.5.0.1/dedupe/ocsd/ocsd/linuxR_x86/main.go:212 +0x8c
Here is the stack from dlv (if that is more useful):
0 0x0000000000488961 in runtime.raise
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/sys_linux_amd64.s:154
1 0x000000000045fdc5 in runtime.dieFromSignal
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/signal_unix.go:942
2 0x0000000000460426 in runtime.sigfwdgo
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/signal_unix.go:1154
3 0x000000000045edc5 in runtime.sigtrampgo
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/signal_unix.go:432
4 0x0000000000488961 in runtime.raise
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/sys_linux_amd64.s:153
5 0x000000000045fdc5 in runtime.dieFromSignal
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/signal_unix.go:942
6 0x0000000000446e71 in runtime.crash
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/signal_unix.go:1031
7 0x0000000000446e71 in runtime.fatalpanic
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/panic.go:1290
8 0x000000000047e8e8 in runtime.gopanic
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/panic.go:804
9 0x0000000000444d94 in runtime.goPanicIndex
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/panic.go:115
10 0x000000000098425d in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.compareHeaders
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:162
11 0x0000000000984871 in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(*SharedKeyCredential).buildCanonicalizedHeader.func1
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:210
12 0x0000000000555a47 in sort.insertionSort_func
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/sort/zsortfunc.go:12
13 0x000000000055605b in sort.pdqsort_func
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/sort/zsortfunc.go:73
14 0x0000000000555545 in sort.Slice
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/sort/slice.go:29
15 0x000000000098465c in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(*SharedKeyCredential).buildCanonicalizedHeader
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:209
16 0x0000000000983df5 in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(*SharedKeyCredential).buildStringToSign
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:93
17 0x0000000000984fdd in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(*SharedKeyCredPolicy).Do
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:291
18 0x000000000092f2b0 in github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144
19 0x000000000095f565 in github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*retryPolicy).Do
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/runtime/policy_retry.go:142
20 0x000000000092f2b0 in github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144
21 0x0000000000960894 in github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.telemetryPolicy.Do
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/runtime/policy_telemetry.go:70
22 0x0000000000962151 in github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*telemetryPolicy).Do
at :1
23 0x000000000092f2b0 in github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144
24 0x000000000095d21c in github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.includeResponsePolicy
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/runtime/policy_include_response.go:19
25 0x000000000092f4b9 in github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.PolicyFunc.Do
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:214
26 0x000000000092f2b0 in github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144
27 0x000000000092ecc5 in github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.Pipeline.Do
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/pipeline.go:76
28 0x000000000096ed06 in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.(*BlockBlobClient).CommitBlockList
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/generated/zz_blockblob_client.go:51
29 0x000000000098c15d in github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*Client).CommitBlockList
at /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/blockblob/client.go:285
30 0x0000000000adf8c9 in main.(*OCSBlockBlob).commitBlockList
at /home/jmg/dedupe.mr/ocsd/ocsd/linuxR_x86/azure_blob_ops.go:1691
31 0x0000000000aaea35 in main.commitBlockList
at /home/jmg/dedupe.mr/ocsd/ocsd/linuxR_x86/request_handle.go:931
32 0x0000000000b0951d in main.requestWorker
at /home/jmg/dedupe.mr/ocsd/ocsd/linuxR_x86/main.go:48
33 0x0000000000b0c665 in main.dispatchRequests.gowrap1
at /home/jmg/dedupe.mr/ocsd/ocsd/linuxR_x86/main.go:212
34 0x0000000000487121 in runtime.goexit
at /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/asm_amd64.s:1700
As you can see, something called "compareHeaders" is generating an index out of range panic.
My assumption is that some argument we are passing to CommitBlockList is causing the problem, but I don't know what that might be.
Can someone familiar with the internals of the SDK tell me what options or other input to the call to CommitBlockList could be triggering this panic?
Any other suggestions as to how to debug this problem?
Thanks!
Joseph Milton Gilpin
Veritas
PS here is some output from dlv if that helps, including the args -v for the CommitBlockList function. Do you see something amiss in what is being passed to CommitBlockList?
Frame 29: /home/jmg/dedupe.mr/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/blockblob/client.go:285 (PC: 98c15d)
280: leaseAccess, modifiedAccess = exported.FormatBlobAccessConditions(options.AccessConditions)
281: cpkInfo = options.CPKInfo
282: cpkScope = options.CPKScopeInfo
283: }
284:
=> 285: resp, err := bb.generated().CommitBlockList(ctx, blockLookupList, commitOptions, headers, leaseAccess, cpkInfo, cpkScope, modifiedAccess)
286: return resp, err
287: }
288:
289: // GetBlockList returns the list of blocks that have been uploaded as part of a block blob using the specified block list filter.
290: // For more information, see https://docs.microsoft.com/rest/api/storageservices/get-block-list.
(dlv) args -v
bb = ("*github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.Client")(0xc000013800)
*github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.Client {
innerT: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.BlobClient {
internal: *(*"github.com/Azure/azure-sdk-for-go/sdk/azcore.Client")(0xc0007ba150),
endpoint: "https://bptadlsgen2.blob.core.windows.net:443/vrts-adlsgen2-bptl...+75 more",},
innerU: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.BlockBlobClient {
internal: *(*"github.com/Azure/azure-sdk-for-go/sdk/azcore.Client")(0xc0007ba150),
endpoint: "https://bptadlsgen2.blob.core.windows.net:443/vrts-adlsgen2-bptl...+75 more",},
sharedKey: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.SharedKeyCredential {
accountName: "bptadlsgen2",
accountKey: (*"sync/atomic.Value")(0xc0006c5e10),},}
ctx = (unreadable error loading module data: error while reading spliced memory at 0x171515161514193c: EOF)
base64BlockIDs = []string len: 1, cap: 0, [
"m99heWwDTuBtUVLxNKPAMQ==",
]
options = ("*github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.CommitBlockListOptions")(0xc000683a98)
*github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.CommitBlockListOptions {
Tags: map[string]string [],
Metadata: map[string]*string [
"ewkxczs": *"pRTgwRV",
"msld": *"KuFaQ",
"po": *"kgOwCMgD",
"potdh": *"plDhC",
],
RequestID: *string nil,
Tier: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.AccessTier nil,
Timeout: *int32 nil,
HTTPHeaders: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.BlobHTTPHeaders {
BlobCacheControl: *"max-age=86400",
BlobContentDisposition: *string nil,
BlobContentEncoding: *"zstd",
BlobContentLanguage: *"es-ES",
BlobContentMD5: []uint8 len: 16, cap: 18, [129,104,42,114,93,221,175,66,250,57,241,252,255,129,67,128],
BlobContentType: *"application/octet-stream",},
CPKInfo: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.CPKInfo nil,
CPKScopeInfo: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.CPKScopeInfo nil,
AccessConditions: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.BlobAccessConditions nil,
LegalHold: *bool nil,
ImmutabilityPolicyMode: *github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.ImmutabilityPolicySetting nil,
ImmutabilityPolicyExpiryTime: *time.Time nil,
TransactionalContentCRC64: []uint8 len: 0, cap: 0, nil,
TransactionalContentMD5: []uint8 len: 0, cap: 0, nil,}
~r0 = (unreadable empty OP stack)
~r1 = (unreadable empty OP stack)
(dlv)
Hello,
I am hoping someone can point us in the right direction for this problem.
We recently upgraded to the latest Microsoft go 1.23.2 and azure SDK v1.16.0. Previously we were using 1.22.5 and v1.14.0, so not very old.
After upgrading we have noticed that one of our automated tests has started failing about 20% of the time with a go panic. This panic is from an "index out of range" error deep in the stack upon calling CommitBlockList.
Here is the panic:
panic: runtime error: index out of range [12] with length 12
goroutine 2419 gp=0xc0001001c0 m=4 mp=0xc00005d508 [running]: panic({0xd9f180?, 0xc0007aa1f8?}) /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/panic.go:804 +0x168 fp=0xc0007da6b8 sp=0xc0007da608 pc=0x47e8e8 runtime.goPanicIndex(0xc, 0xc) /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/runtime/panic.go:115 +0x74 fp=0xc0007da6f8 sp=0xc0007da6b8 pc=0x444d94 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.compareHeaders({0xc0005185c0?, 0xc000682748?}, {0xc0005185f0?, 0x20?}, {0xc0005ca570?, 0x1?, 0xc000682840?}) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:162 +0x13d fp=0xc0007da718 sp=0xc0007da6f8 pc=0x98425d github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(SharedKeyCredential).buildCanonicalizedHeader.func1(0xc0000138a8?, 0x7f85146f8678?) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:210 +0x51 fp=0xc0007da760 sp=0xc0007da718 pc=0x984871 sort.insertionSort_func({0xc000682b38?, 0xc0004cce20?}, 0x0, 0xb) /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/sort/zsortfunc.go:12 +0xa7 fp=0xc0007da7a8 sp=0xc0007da760 pc=0x555a47 sort.pdqsort_func({0xc000682b38?, 0xc0004cce20?}, 0xc0000138a8?, 0x0?, 0x0?) /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/sort/zsortfunc.go:73 +0x31b fp=0xc0007da850 sp=0xc0007da7a8 pc=0x55605b sort.Slice({0xcc0f60?, 0xc0000138a8?}, 0xc000682b38) /net/code/extsrc/go/ms1.23.2-1/linuxR_x86/src/sort/slice.go:29 +0xc5 fp=0xc0007da8b8 sp=0xc0007da850 pc=0x555545 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(SharedKeyCredential).buildCanonicalizedHeader(0xcfafa0?, 0xc0005ca540) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:209 +0x3dc fp=0xc0007dabe0 sp=0xc0007da8b8 pc=0x98465c github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(SharedKeyCredential).buildStringToSign(0xc0006c5e00, 0xc0007a8f00) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:93 +0xa35 fp=0xc0007dadd0 sp=0xc0007dabe0 pc=0x983df5 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported.(SharedKeyCredPolicy).Do(0xc0003ee2b0, 0xc0005a5700) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.4.1/internal/exported/shared_key_credential.go:291 +0x21d fp=0xc0007daee8 sp=0xc0007dadd0 pc=0x984fdd github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(Request).Next(0xc000683050) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144 +0xf0 fp=0xc0007daf28 sp=0xc0007daee8 pc=0x92f2b0 github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(retryPolicy).Do(0x3?, 0xc0005a56c0) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/runtime/policy_retry.go:142 +0x5a5 fp=0xc0007db350 sp=0xc0007daf28 pc=0x95f565 github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(Request).Next(0xc0005a5680) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/internal/exported/request.go:144 +0xf0 fp=0xc0007db390 sp=0xc0007db350 pc=0x92f2b0 github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.telemetryPolicy.Do({{0xc0003eb6c0?, 0xd82f60?}}, 0xc0005a5680) /nbcm_builds/DD/20.5.0.1/dedupe/nbbuild/go-build/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.16.0/runtime/policy_telemetry.go:70 +0x1b4 fp=0xc0007db428 sp=0xc0007db390 pc=0x960894 github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(telemetryPolicy).Do(0x7f8514f7c0c8?, 0x40?)