search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.12k stars 3.76k forks source link

Problem with Get-AzAppConfigurationKeyValue when more that 100 records are present #23731

Open Keiyan opened 7 months ago

Keiyan commented 7 months ago

Description

When using Get-AzAppConfigurationKeyValue on an AppConfiguration with more than 100 entries, se are met with the following error, which seems to be due to a problem on the NextLink URL : Invalid URI: The format of the URI could not be determined.

All related information lies in attached logs.

Issue script & Debug output

[CmdletBeginProcessing]: Starting command
CmdletBeginProcessing: 
CmdletProcessRecordStart: 
CmdletGetPipeline: 
CmdletBeforeAPICall: 
URLCreated: /kv?api-version=1.0
RequestCreated: /kv?api-version=1.0
HeaderParametersAdded: 
============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://frdevcommonapp01config.azconfig.io/kv?api-version=1.0

Headers:
x-ms-unique-id                : 29
x-ms-client-request-id        : 0369ee5c-72b8-4d65-adb9-96e8ed0c546b
CommandName                   : Get-AzAppConfigurationKeyValue
FullCommandName               : Get-AzAppConfigurationKeyValue_Get
ParameterSetName              : __AllParameterSets
User-Agent                    : AzurePowershell/v0.0.0,PSVersion/v7.4.0,Az.AppConfigurationdata/1.3.0

Body:

BeforeCall: 
============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Date                          : Wed, 13 Dec 2023 14:56:23 GMT
Transfer-Encoding             : chunked
Connection                    : keep-alive
Link                          : </kv?api-version=1.0&after=RG9tYWluX29sZF9BY2Nlc3NSaWdodF9DbGllbnRBZFNlY3JldAo%3D>; rel="next"
Sync-Token                    : zAJw6V16=NDoxMyM5NTc4MTI5;sn=9578129
x-ms-request-id               : f5e8d60c-e363-40da-8ab3-665c54fb47c1
x-ms-correlation-request-id   : f5e8d60c-e363-40da-8ab3-665c54fb47c1
Strict-Transport-Security     : max-age=15724800; includeSubDomains
Access-Control-Allow-Origin   : *

Body:
{
  "items": [redacted],
  "@nextLink": "/kv?api-version=1.0&after=RG9tYWluX29sZF9BY2Nlc3NSaWdodF9DbGllbnRBZFNlY3JldAo%3D"
}

ResponseCreated: 
BeforeResponseDispatch: 

[redacted]
Finally: 
[CmdletProcessRecordAsyncEnd]: Finish HTTP process
CmdletProcessRecordAsyncEnd: 
[CmdletException]: Received Exception with message 'UriFormatException - Invalid URI: The format of the URI could not be determined. :    at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind, UriCreationOptions& creationOptions)
   at System.Uri..ctor(String uriString)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.onOk(HttpResponseMessage responseMessage, Task`1 response)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues_Call(HttpRequestMessage request, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues_Call(HttpRequestMessage request, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues(String syncToken, String acceptDatetime, String key, String label, String after, List`1 Select, String endpoint, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.ProcessRecordAsync()
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.ProcessRecordAsync()'
CmdletException: UriFormatException - Invalid URI: The format of the URI could not be determined. :    at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind, UriCreationOptions& creationOptions)
   at System.Uri..ctor(String uriString)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.onOk(HttpResponseMessage responseMessage, Task`1 response)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues_Call(HttpRequestMessage request, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues_Call(HttpRequestMessage request, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues(String syncToken, String acceptDatetime, String key, String label, String after, List`1 Select, String endpoint, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.ProcessRecordAsync()
   at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.ProcessRecordAsync()
Get-AzAppConfigurationKeyValue_Get: C:\Users\mickphilippon\OneDrive - KPMG\Documents\PowerShell\Modules\Az.AppConfiguration\1.3.0\AppConfigurationData.Autorest\exports\ProxyCmdletDefinitions.ps1:1735:9
Line |
1735 |          $steppablePipeline.Begin($PSCmdlet)
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Invalid URI: The format of the URI could not be determined.

CmdletProcessRecordEnd:

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.0
PSEdition                      Core
GitCommitId                    7.4.0
OS                             Microsoft Windows 10.0.19044
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.13.2                Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     1.3.0                 Az.AppConfiguration                 {Clear-AzAppConfigurationDeletedStore, Get-AzAppConfigurationDeletedStore, Get-AzAppConfigurationKey, Get-AzAppConfi…

Error output

3:59:22 PM - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.
3:59:22 PM - using account id 'redacted'...
3:59:22 PM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].

   HistoryId: 2

Message        : Invalid URI: The format of the URI could not be determined.
StackTrace     :    at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind, UriCreationOptions& creationOptions)
                    at System.Uri..ctor(String uriString)
                    at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.onOk(HttpResponseMessage responseMessage, Task`1 response)
                    at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues_Call(HttpRequestMessage request, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
                    at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues_Call(HttpRequestMessage request, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
                    at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.AppConfigurationdata.GetKeyValues(String syncToken, String acceptDatetime, String key, String label, String after, List`1 Select, String endpoint, Func`3 onOk, Func`3 onDefault, IEventListener eventListener, ISendAsync sender)
                    at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.ProcessRecordAsync()
                    at Microsoft.Azure.PowerShell.Cmdlets.AppConfigurationdata.Cmdlets.GetAzAppConfigurationKeyValue_Get.ProcessRecordAsync()
Exception      : System.UriFormatException
InvocationInfo : {Get-AzAppConfigurationKeyValue_Get}
Line           :         $steppablePipeline.Begin($PSCmdlet)

Position       : At C:\Users\mickphilippon\OneDrive - KPMG\Documents\PowerShell\Modules\Az.AppConfiguration\1.3.0\AppConfigurationData.Autorest\exports\ProxyCmdletDefinitions.ps1:1735 char:9
                 +         $steppablePipeline.Begin($PSCmdlet)
                 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 2

3:59:22 PM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
3:59:22 PM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
AzureQoSEvent:  Module: Az.Accounts:2.13.2; CommandName: Resolve-AzError; PSVersion: 7.4.0; IsSuccess: True; Duration: 00:00:00.0662331
3:59:22 PM - [ConfigManager] Got [True] from [EnableDataCollection], Module = [], Cmdlet = [].
3:59:23 PM - ResolveError end processing.
wyunchi-ms commented 7 months ago

Hi @Keiyan ,sorry for the issue. We will fix this in the future release. But you can workaround this by this script:

$store = Get-AzAppConfigurationStore -ResourceGroupName $rg -Name $name
$endpoint = $store.Endpoint
$uri = "$endpoint/kv?api-version=1.0"
$keyVaulePairList = @()
while ($true) {
    write-Host "Fetching data from $uri"
    $response = (Invoke-AzRest -Uri $uri -Method GET).Content | ConvertFrom-Json
    $keyVaulePairList += $response.items
    $nextLink = $response."@nextLink"
    if ($null -eq $nextLink) {
        break
    }
    $uri = $endpoint + $nextLink
}
arieljgm commented 4 months ago

I had the same problem, here I share a slightly improved simplified version to only obtain the KEY, but you can improve it or extend it to your liking.

$uri = "$Endpoint/kv?api-version=1.0"
$allKeys = New-Object System.Collections.Generic.List[String]
do {
    Write-Verbose "Fetching data from $uri"
    $response = (Invoke-AzRest -Uri $uri -Method GET).Content | ConvertFrom-Json
    foreach ($item in $response.items) {
        $allKeys.Add($item.key)
    }
    $uri = $Endpoint + $response."@nextLink"
} while ($null -ne $response."@nextLink")