search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.21k stars 3.82k forks source link

Get-AzHdInsightCluster `not a valid Base-64 string` error for ESP enabled cluster #10280

Closed stijzermans closed 4 years ago

stijzermans commented 4 years ago

Description

Get-AzHdInsightCluster throws an error when using this on a HDInsight cluster deployed with ESP enabled and after retrieving the cluster identity. Therefore, it's impossible to use Get-AzHdInsightCluster with this type of ESP clusters.

{
  "clusterIdentity.applicationId": "*****",
  "clusterIdentity.certificate": "*****",
  "clusterIdentity.aadTenantId": "*****",
  "clusterIdentity.resourceUri": "https://management.core.windows.net/",
  "clusterIdentity.certificatePassword": "*****"
}

Get-AzHdInsightCluster : The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
At line:1 char:12
+ $cluster = Get-AzHdInsightCluster -Debug -ResourceGroupName "hdirg ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : CloseError: (:) [Get-AzHDInsightCluster], FormatException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.HDInsight.GetAzureHDInsightCommand

Steps to reproduce

Deploy a cluster with the following template:

ARM template ``` { "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#", "contentVersion": "0.9.0.0", "parameters": { "SubscriptionId": { "type": "string" }, "clusterName": { "type": "string", "metadata": { "description": "The name of the HDInsight cluster to create." } }, "clusterLoginUserName": { "type": "string", "defaultValue": "admin", "metadata": { "description": "These credentials can be used to submit jobs to the cluster and to log into cluster dashboards." } }, "clusterLoginPassword": { "type": "securestring", "metadata": { "description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter." } }, "sshUserName": { "type": "string", "defaultValue": "sshuser", "metadata": { "description": "These credentials can be used to remotely access the cluster." } }, "sshPublicKey": { "type": "securestring" }, "clusterKind": { "type": "string", "defaultValue": "SPARK", "metadata": { "description": "The type of the HDInsight cluster to create." } }, "clusterKindVersion": { "type": "string", "defaultValue": "3.6", "metadata": { "description": "SPARK version." } }, "clusterVersion": { "type": "string", "defaultValue": "3.6", "metadata": { "description": "HDInsight cluster version." } }, "clusterWorkerNodeCount": { "type": "int", "defaultValue": 2, "metadata": { "description": "The number of nodes in the HDInsight cluster." } }, "headnodeVMSize": { "type": "string" }, "workernodeVMSize": { "type": "string" }, "clusterCertificateContents": { "type": "securestring" }, "clusterCertificatePassword": { "type": "securestring" }, "aadApplicationId": { "type": "string" }, "AdlsHostname": { "type": "string" }, "AdlsMountpoint": { "type": "string" }, "SQLServerName": { "type": "string" }, "SQLUserName": { "type": "string", "metadata": { "description": "These credentials can be used to remotely access hive." } }, "SQLPassword": { "type": "securestring" }, "metaDbName": { "type": "string" }, "rangerDbName": { "type": "string" }, "vnetName": { "type": "string" }, "vnetResourceGroup": { "type": "string" }, "subnetName": { "type": "string" }, "domainUsername": { "type": "string" }, "domain": { "type": "securestring" }, "clusterUsersGroups": { "type": "array" }, "addsId": { "type": "string" }, "ManagedIdentityId": { "type": "string" }, "billingTags": { "type": "object", "metadata": { "description": "The tags that will be set on the resources. Used for billing purposes." } } }, "variables": { "aadTenantId": "", "vnetId": "[resourceId(parameters('vnetResourceGroup'),'Microsoft.Network/virtualNetworks', parameters('vnetName'))]", "subnetRef": "[concat(variables('vnetId'), '/subnets/', parameters('subnetName'))]" }, "resources": [ { "apiVersion": "2015-03-01-preview", "name": "[parameters('clusterName')]", "type": "Microsoft.HDInsight/clusters", "location": "[resourceGroup().location]", "dependsOn": [], "properties": { "clusterVersion": "[parameters('clusterVersion')]", "osType": "Linux", "tier": "premium", "clusterDefinition": { "kind": "[parameters('clusterKind')]", "componentVersion": { "Spark": "[parameters('clusterKindVersion')]" }, "configurations": { "gateway": { "restAuthCredential.isEnabled": true, "restAuthCredential.username": "[parameters('clusterLoginUserName')]", "restAuthCredential.password": "[parameters('clusterLoginPassword')]" }, "core-site": { "fs.defaultFS": "adl://home", "dfs.adls.home.hostname": "[parameters('AdlsHostname')]", "dfs.adls.home.mountpoint": "[parameters('AdlsMountpoint')]" }, "hive-site": { "javax.jdo.option.ConnectionDriverName": "com.microsoft.sqlserver.jdbc.SQLServerDriver", "javax.jdo.option.ConnectionURL": "[concat('jdbc:sqlserver://', parameters('SQLServerName'), '.database.windows.net;database=', parameters('metaDbName'), ';encrypt=true;trustServerCertificate=true;create=false;loginTimeout=300')]", "javax.jdo.option.ConnectionUserName": "[parameters('SQLUserName')]", "javax.jdo.option.ConnectionPassword": "[parameters('SQLPassword')]" }, "hive-env": { "hive_database": "Existing MSSQL Server database with SQL authentication", "hive_database_name": "[parameters('metaDbName')]", "hive_database_type": "mssql", "hive_existing_mssql_server_database": "[parameters('metaDbName')]", "hive_existing_mssql_server_host": "[concat(parameters('SqlServerName'), '.database.windows.net')]", "hive_hostname": "[concat(parameters('SqlServerName'), '.database.windows.net')]" }, "admin-properties": { "audit_db_name": "[parameters('rangerDbName')]", "audit_db_user": "[parameters('SQLUserName')]", "audit_db_password": "[parameters('SQLPassword')]", "db_name": "[parameters('rangerDbName')]", "db_user": "[parameters('SQLUserName')]", "db_password": "[parameters('SQLPassword')]", "db_host": "[concat(parameters('SqlServerName'), '.database.windows.net')]", "db_root_user": "", "db_root_password": "" }, "ranger-admin-site": { "ranger.jpa.jdbc.url": "[concat('jdbc:sqlserver://', parameters('SQLServerName'), '.database.windows.net;database=', parameters('rangerDbName'))]" }, "ranger-env": { "ranger_privelege_user_jdbc_url": "[concat('jdbc:sqlserver://', parameters('SQLServerName'), '.database.windows.net;database=', parameters('rangerDbName'))]" }, "ranger-hive-security": { "ranger.plugin.hive.service.name": "[concat(parameters('clusterName'), '_hive')]" }, "ranger-yarn-security": { "ranger.plugin.yarn.service.name": "[concat(parameters('clusterName'), '_yarn')]" }, "clusterIdentity": { "clusterIdentity.applicationId": "[parameters('aadApplicationId')]", "clusterIdentity.certificate": "[parameters('clusterCertificateContents')]", "clusterIdentity.aadTenantId": "[concat('https://login.windows.net/', variables('aadTenantId'))]", "clusterIdentity.resourceUri": "https://management.core.windows.net/", "clusterIdentity.certificatePassword": "[parameters('clusterCertificatePassword')]" } } }, "computeProfile": { "roles": [ { "autoscale": null, "name": "headnode", "minInstanceCount": 1, "targetInstanceCount": 2, "hardwareProfile": { "vmSize": "[parameters('headnodeVMSize')]" }, "osProfile": { "linuxOperatingSystemProfile": { "username": "[parameters('sshUserName')]", "sshProfile": { "publicKeys": [ { "certificateData": "[parameters('sshPublicKey')]" } ] } } }, "virtualNetworkProfile": { "id": "[variables('vnetId')]", "subnet": "[variables('subnetRef')]" }, "scriptActions": [] }, { "autoscale": null, "name": "workernode", "targetInstanceCount": "[parameters('clusterWorkerNodeCount')]", "hardwareProfile": { "vmSize": "[parameters('workernodeVMSize')]" }, "osProfile": { "linuxOperatingSystemProfile": { "username": "[parameters('sshUserName')]", "sshProfile": { "publicKeys": [ { "certificateData": "[parameters('sshPublicKey')]" } ] } } }, "virtualNetworkProfile": { "id": "[variables('vnetId')]", "subnet": "[variables('subnetRef')]" }, "scriptActions": [] } ] }, "securityProfile": { "directoryType": "ActiveDirectory", "ldapsUrls": [ "[concat('ldaps://', parameters('domain'), ':636')]" ], "domainUsername": "[parameters('domainUsername')]", "domain": "[parameters('domain')]", "clusterUsersGroupDNs": "[parameters('clusterUsersGroups')]", "aaddsResourceId": "[parameters('addsId')]", "msiResourceId": "[parameters('ManagedIdentityId')]" } }, "identity": { "type": "UserAssigned", "userAssignedIdentities": { "[parameters('ManagedIdentityId')]": {} } }, "tags": "[parameters('billingTags')]" } ] } ```

Make sure to use the following command to generate base64 certificate.

# generate certificate in certstore
$cert = New-SelfSignedCertificate -DnsName $CertificateName -CertStoreLocation cert:\CurrentUser\My -KeySpec KeyExchange -NotAfter $certEndDate -NotBefore $certStartDate
$certThumbprint = $cert.Thumbprint
$cert = (Get-ChildItem -Path cert:\CurrentUser\My\$certThumbprint)
# generate PFX file
Export-PfxCertificate -Cert $cert -FilePath $certFilePath -Password $certPasswordSecureString

# cluster cert
$b64certificate = [System.Convert]::ToBase64String((Get-Content $certFilePath -Encoding Byte))

Environment data

Name                           Value
----                           -----
PSVersion                      6.2.2
PSEdition                      Core
GitCommitId                    6.2.2
OS                             Darwin 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:5…
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

   Directory: /Users/<user_folder>/.local/share/powershell/Modules

ModuleType Version    Name                                PSEdition ExportedCommands
---------- -------    ----                                --------- ----------------
Script     2.7.0      Az                                  Core,Desk 
Script     1.6.2      Az.Accounts                         Core,Desk {Disable-AzDataCollect…
Script     1.0.1      Az.Advisor                          Core,Desk {Get-AzAdvisorRecommen…
Script     1.0.2      Az.Aks                              Core,Desk {Get-AzAks, New-AzAks,…
Script     1.1.1      Az.AnalysisServices                 Core,Desk {Resume-AzAnalysisServ…
Script     1.3.1      Az.ApiManagement                    Core,Desk {Add-AzApiManagementAp…
Script     1.0.2      Az.ApplicationInsights              Core,Desk {Get-AzApplicationInsi…
Script     1.3.3      Az.Automation                       Core,Desk {Get-AzAutomationHybri…
Script     1.1.1      Az.Batch                            Core,Desk {Remove-AzBatchAccount…
Script     1.0.1      Az.Billing                          Core,Desk {Get-AzBillingInvoice,…
Script     1.3.1      Az.Cdn                              Core,Desk {Get-AzCdnProfile, Get…
Script     1.2.1      Az.CognitiveServices                Core,Desk {Get-AzCognitiveServic…
Script     2.6.0      Az.Compute                          Core,Desk {Remove-AzAvailability…
Script     1.0.1      Az.ContainerInstance                Core,Desk {New-AzContainerGroup,…
Script     1.1.0      Az.ContainerRegistry                Core,Desk {New-AzContainerRegist…
Script     1.3.0      Az.DataFactory                      Core,Desk {Set-AzDataFactoryV2, …
Script     1.0.1      Az.DataLakeAnalytics                Core,Desk {Get-AzDataLakeAnalyti…
Script     1.2.2      Az.DataLakeStore                    Core,Desk {Get-AzDataLakeStoreTr…
Script     1.0.1      Az.DeploymentManager                Core,Desk {Get-AzDeploymentManag…
Script     1.0.0      Az.DevTestLabs                      Core,Desk {Get-AzDtlAllowedVMSiz…
Script     1.1.1      Az.Dns                              Core,Desk {Get-AzDnsRecordSet, N…
Script     1.2.2      Az.EventGrid                        Core,Desk {New-AzEventGridTopic,…
Script     1.4.0      Az.EventHub                         Core,Desk {New-AzEventHubNamespa…
Script     1.1.1      Az.FrontDoor                        Core,Desk {New-AzFrontDoor, Get-…
Script     2.0.2      Az.HDInsight                        Core,Desk {Get-AzHDInsightJob, N…
Script     1.3.0      Az.IotHub                           Core,Desk {Add-AzIotHubKey, Get-…
Script     1.3.1      Az.KeyVault                         Core,Desk {Add-AzKeyVaultCertifi…
Script     1.3.1      Az.LogicApp                         Core,Desk {Get-AzIntegrationAcco…
Script     1.1.1      Az.MachineLearning                  Core,Desk {Move-AzMlCommitmentAs…
Script     1.0.1      Az.ManagedServices                  Core,Desk {Get-AzManagedServices…
Script     1.0.1      Az.MarketplaceOrdering              Core,Desk {Get-AzMarketplaceTerm…
Script     1.1.0      Az.Media                            Core,Desk {Sync-AzMediaServiceSt…
Script     1.3.0      Az.Monitor                          Core,Desk {Get-AzMetricDefinitio…
Script     1.14.0     Az.Network                          Core,Desk {Add-AzApplicationGate…
Script     1.1.0      Az.NotificationHubs                 Core,Desk {Get-AzNotificationHub…
Script     1.3.3      Az.OperationalInsights              Core,Desk {New-AzOperationalInsi…
Script     1.1.3      Az.PolicyInsights                   Core,Desk {Get-AzPolicyEvent, Ge…
Script     1.1.0      Az.PowerBIEmbedded                  Core,Desk {Remove-AzPowerBIWorks…
Script     1.4.5      Az.RecoveryServices                 Core,Desk {Get-AzRecoveryService…
Script     1.1.0      Az.RedisCache                       Core,Desk {Remove-AzRedisCachePa…
Script     1.0.2      Az.Relay                            Core,Desk {New-AzRelayNamespace,…
Script     1.7.0      Az.Resources                        Core,Desk {Get-AzProviderOperati…
Script     1.4.0      Az.ServiceBus                       Core,Desk {New-AzServiceBusNames…
Script     1.2.0      Az.ServiceFabric                    Core,Desk {Add-AzServiceFabricAp…
Script     1.1.0      Az.SignalR                          Core,Desk {New-AzSignalR, Get-Az…
Script     1.14.2     Az.Sql                              Core,Desk {Get-AzSqlDatabaseTran…
Script     1.7.0      Az.Storage                          Core,Desk {Get-AzStorageAccount,…
Script     1.2.0      Az.StorageSync                      Core,Desk {Invoke-AzStorageSyncC…
Script     1.0.0      Az.StreamAnalytics                  Core,Desk {Get-AzStreamAnalytics…
Script     1.0.2      Az.TrafficManager                   Core,Desk {Add-AzTrafficManagerC…
Script     1.4.2      Az.Websites                         Core,Desk {Get-AzAppServicePlan,…

    Directory: /usr/local/microsoft/powershell/6/Modules

ModuleType Version    Name                                PSEdition ExportedCommands
---------- -------    ----                                --------- ----------------
Manifest   1.2.3.0    Microsoft.PowerShell.Archive        Desk      {Compress-Archive, Exp…
Manifest   6.1.0.0    Microsoft.PowerShell.Host           Core      {Start-Transcript, Sto…
Manifest   6.1.0.0    Microsoft.PowerShell.Management     Core      {Add-Content, Clear-Co…
Manifest   6.1.0.0    Microsoft.PowerShell.Security       Core      {Get-Credential, Get-E…
Manifest   6.1.0.0    Microsoft.PowerShell.Utility        Core      {Export-Alias, Get-Ali…
Script     1.3.2      PackageManagement                   Desk      {Find-Package, Get-Pac…
Script     2.1.3      PowerShellGet                       Desk      {Find-Command, Find-DS…
Script     0.0        PSDesiredStateConfiguration         Desk      {ValidateNodeResources…
Script     2.0.0      PSReadLine                          Desk      {Get-PSReadLineKeyHand…
Binary     1.1.2      ThreadJob                           Desk      Start-ThreadJob

Debug output

DEBUG: 9:07:39 AM - GetAzureHDInsightCommand begin processing with ParameterSet '__AllParameterSets'.
DEBUG: 9:07:39 AM - using account id '<username>'...
DEBUG: [Common.Authentication]: Authenticating using Account: '<username>', environment: 'AzureCloud', tenant: '<tenant>'
DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0696480Z: 00000000-0000-0000-0000-000000000000 - LoggerBase.cs: Serializing token cache with 1 items.

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0702500Z: 00000000-0000-0000-0000-000000000000 - LoggerBase.cs: Serializing token cache with 1 items.

DEBUG: [Common.Authentication]: Authenticating using configuration values: Domain: '<tenant>', Endpoint: 'https://login.microsoftonline.com/', ClientId: '<client_id>', ClientRedirect: 'urn:ietf:wg:oauth:2.0:oob', ResourceClientUri: 'https://management.core.windows.net/', ValidateAuthority: 'True'
DEBUG: [Common.Authentication]: Acquiring token using context with Authority 'https://login.microsoftonline.com/<tenant>/', CorrelationId: '00000000-0000-0000-0000-000000000000', ValidateAuthority: 'True'
DEBUG: [Common.Authentication]: Acquiring token using AdalConfiguration with Domain: '<tenant>', AdEndpoint: 'https://login.microsoftonline.com/', ClientId: '<client_id>', ClientRedirectUri: urn:ietf:wg:oauth:2.0:oob
DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0708680Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: ADAL PCL.CoreCLR with assembly version '3.19.2.6005', file version '3.19.50302.0130' and informational version '2a8bec6c4c76d0c1ef819b55bdc3cda2d2605056' is running...

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0709530Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: ADAL PCL.CoreCLR with assembly version '3.19.2.6005', file version '3.19.50302.0130' and informational version '2a8bec6c4c76d0c1ef819b55bdc3cda2d2605056' is running...

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0710500Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: === Token Acquisition started: 
    CacheType: null
    Authentication Target: User
    , Authority Host: login.microsoftonline.com

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0711330Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: === Token Acquisition started:
    Authority: https://login.microsoftonline.com/<tenant>/
    Resource: https://management.core.windows.net/
    ClientId: <client_id>
    CacheType: null
    Authentication Target: User

DEBUG: [ADAL]: Verbose: 2019-10-14T07:07:39.0713040Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: Loading from cache.

DEBUG: [ADAL]: Verbose: 2019-10-14T07:07:39.0713710Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: Loading from cache.

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0722130Z: 00000000-0000-0000-0000-000000000000 - LoggerBase.cs: Deserialized 1 items to token cache.

DEBUG: [ADAL]: Verbose: 2019-10-14T07:07:39.0723550Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: Looking up cache for a token...

DEBUG: [ADAL]: Verbose: 2019-10-14T07:07:39.0724420Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: Looking up cache for a token...

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0725930Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: An item matching the requested resource was found in the cache

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0726770Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: An item matching the requested resource was found in the cache

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0728190Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: 37.28186975 minutes left until token in cache expires

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0728940Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: 37.28186975 minutes left until token in cache expires

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0729820Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: A matching item (access token or refresh token or both) was found in the cache

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0730570Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: A matching item (access token or refresh token or both) was found in the cache

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0732020Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: === Token Acquisition finished successfully. An access token was returned: Expiration Time: 10/14/19 7:44:55 AM +00:00

DEBUG: [ADAL]: Information: 2019-10-14T07:07:39.0732810Z: e86ecfd4-a6ba-4c4f-b35d-a7e5085c6b63 - LoggerBase.cs: === Token Acquisition finished successfully. An access token was returned: Expiration Time: 10/14/19 7:44:55 AM +00:00Access Token Hash: <token_hash>
     User id: <userid>

DEBUG: [Common.Authentication]: Received token with LoginType 'LiveId', Tenant: '<tenant>', UserId: '<username>'
DEBUG: [Common.Authentication]: Renewing Token with Type: 'Bearer', Expiry: '10/14/2019 07:44:55 +00:00', MultipleResource? 'True', Tenant: '<tenant>', UserId: '<username>'
DEBUG: [Common.Authentication]: User info for token DisplayId: '<username>', Name: <fullname>, IdProvider: 'https://sts.windows.net/<tenant>/', Uid: '<userid>'
DEBUG: [Common.Authentication]: Checking token expiration, token expires '10/14/2019 07:44:55 +00:00' Comparing to '10/14/2019 07:07:39 +00:00' With threshold '00:05:00', calculated time until token expiry: '00:37:16.9107230'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<hdi_rg>/providers/Microsoft.HDInsight/clusters/<cluster_name>?api-version=2015-03-01-preview

Headers:
User-Agent                    : ARM,SDK,v2.1.0

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-hdi-matched-rule         : ClusterResourcesAndSubResources
x-ms-hdi-routed-to            : RegionalRp
x-ms-request-id               : 21dc6524-0ab6-4802-a92d-9b6b32611704
x-ms-hdi-served-by            : <hdi_region>
x-ms-correlation-request-id   : 47b0d5f9-8cc8-4d5c-ad3b-0ea22ff1e3a2
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-ratelimit-remaining-subscription-reads: 11999
x-ms-routing-request-id       : WESTEUROPE:20191014T070741Z:47b0d5f9-8cc8-4d5c-ad3b-0ea22ff1e3a2
X-Content-Type-Options        : nosniff
Date                          : Mon, 14 Oct 2019 07:07:40 GMT

Body:
{
  .... cluster information ....
}

DEBUG: [Common.Authentication]: Renewing Token with Type: 'Bearer', Expiry: '10/14/2019 07:44:55 +00:00', MultipleResource? 'True', Tenant: '<tenant>', UserId: '<username>'
DEBUG: [Common.Authentication]: User info for token DisplayId: '<username>', Name: <fullname>, IdProvider: 'https://sts.windows.net/<tenant>/', Uid: '<userid>'
DEBUG: [Common.Authentication]: Checking token expiration, token expires '10/14/2019 07:44:55 +00:00' Comparing to '10/14/2019 07:07:40 +00:00' With threshold '00:05:00', calculated time until token expiry: '00:37:15.2851520'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<hdi_rg>/providers/Microsoft.HDInsight<adls_mountpath>configurations/core-site?api-version=2015-03-01-preview

Headers:
User-Agent                    : ARM,SDK,v2.1.0

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-hdi-matched-rule         : ClusterResourcesAndSubResources
x-ms-hdi-routed-to            : RegionalRp
x-ms-request-id               : 8cbef637-99e8-46e1-94df-3f9b85f7612c
x-ms-hdi-served-by            : <hdi_region>
x-ms-correlation-request-id   : 02234427-f1d7-4201-96d1-df8f056bc2ff
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-ratelimit-remaining-subscription-reads: 11998
x-ms-routing-request-id       : WESTEUROPE:20191014T070742Z:02234427-f1d7-4201-96d1-df8f056bc2ff
X-Content-Type-Options        : nosniff
Date                          : Mon, 14 Oct 2019 07:07:41 GMT

Body:
{
    .... adls information ....
}

DEBUG: [Common.Authentication]: Renewing Token with Type: 'Bearer', Expiry: '10/14/2019 07:44:55 +00:00', MultipleResource? 'True', Tenant: '<tenant>', UserId: '<username>'
DEBUG: [Common.Authentication]: User info for token DisplayId: '<username>', Name: <fullname>, IdProvider: 'https://sts.windows.net/<tenant>/', Uid: '<userid>'
DEBUG: [Common.Authentication]: Checking token expiration, token expires '10/14/2019 07:44:55 +00:00' Comparing to '10/14/2019 07:07:41 +00:00' With threshold '00:05:00', calculated time until token expiry: '00:37:14.3722050'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<hdi_rg>/providers/Microsoft.HDInsight<adls_mountpath>configurations/clusterIdentity?api-version=2015-03-01-preview

Headers:
User-Agent                    : ARM,SDK,v2.1.0

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
x-ms-hdi-matched-rule         : ClusterResourcesAndSubResources
x-ms-hdi-routed-to            : RegionalRp
x-ms-request-id               : c1bc39f4-9fd7-4bd2-a321-ae7c439d7ba2
x-ms-hdi-served-by            : <hdi_region>
x-ms-correlation-request-id   : 01d4d558-5a30-47c3-b05c-f39ebaa77185
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-ratelimit-remaining-subscription-reads: 11997
x-ms-routing-request-id       : WESTEUROPE:20191014T070743Z:01d4d558-5a30-47c3-b05c-f39ebaa77185
X-Content-Type-Options        : nosniff
Date                          : Mon, 14 Oct 2019 07:07:42 GMT

Body:
{
  "clusterIdentity.applicationId": "*****",
  "clusterIdentity.certificate": "*****",
  "clusterIdentity.aadTenantId": "*****",
  "clusterIdentity.resourceUri": "https://management.core.windows.net/",
  "clusterIdentity.certificatePassword": "*****"
}

Get-AzHdInsightCluster : The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
At line:1 char:12
+ $cluster = Get-AzHdInsightCluster -ResourceGroupName "<hdi_rg>" - ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : CloseError: (:) [Get-AzHDInsightCluster], FormatException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.HDInsight.GetAzureHDInsightCommand

DEBUG: AzureQoSEvent: CommandName - Get-AzHDInsightCluster; IsSuccess - False; Duration - 00:00:03.1797279; Exception - System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
   at System.Convert.FromBase64String(String s)
   at Microsoft.Azure.Commands.HDInsight.Models.ClusterConfigurationUtils.GetDefaultStorageAccountDetails(String version, IDictionary`2 coreSiteConfiguration, IDictionary`2 clusterIdentityConfiguration)
   at Microsoft.Azure.Commands.HDInsight.Models.AzureHDInsightCluster..ctor(Cluster cluster, IDictionary`2 clusterConfiguration, IDictionary`2 clusterIdentity)
   at Microsoft.Azure.Commands.HDInsight.GetAzureHDInsightCommand.<ExecuteCmdlet>b__8_0(Cluster entry)
   at System.Linq.Enumerable.SelectListIterator`2.ToList()
   at Microsoft.Azure.Commands.HDInsight.GetAzureHDInsightCommand.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 9:07:42 AM - GetAzureHDInsightCommand end processing.
DEBUG: 9:07:42 AM - GetAzureHDInsightCommand end processing.

Error output

   HistoryId: 10
Message        : The input is not a valid Base-64 string as it contains a non-base 64 
                 character, more than two padding characters, or an illegal character 
                 among the padding characters.
StackTrace     :    at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
                    at System.Convert.FromBase64String(String s)
                    at Microsoft.Azure.Commands.HDInsight.Models.ClusterConfigurationUtils.
                 GetDefaultStorageAccountDetails(String version, IDictionary`2 
                 coreSiteConfiguration, IDictionary`2 clusterIdentityConfiguration)
                    at Microsoft.Azure.Commands.HDInsight.Models.AzureHDInsightCluster..cto
                 r(Cluster cluster, IDictionary`2 clusterConfiguration, IDictionary`2 
                 clusterIdentity)
                    at Microsoft.Azure.Commands.HDInsight.GetAzureHDInsightCommand.<Execute
                 Cmdlet>b__8_0(Cluster entry)
                    at System.Linq.Enumerable.SelectListIterator`2.ToList()
                    at Microsoft.Azure.Commands.HDInsight.GetAzureHDInsightCommand.ExecuteC
                 mdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>
                 c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.Ex
                 ecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.Ex
                 ecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.Proce
                 ssRecord()
Exception      : System.FormatException
InvocationInfo : {Get-AzHDInsightCluster}
Line           : $cluster = Get-AzHdInsightCluster -ResourceGroupName "<hdi_rg>" 
                 -ClusterName "<hdi_name>"
Position       : At line:1 char:12
                 + $cluster = Get-AzHdInsightCluster -ResourceGroupName "<hdi_rg>" - 
                 ...
                 +            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 10

Note - I've redacted information regarding our clusters but believe this shows which errors occurs and how it occurs

stijzermans commented 4 years ago

Well, it seems my timing is a bit off. I saw a PR has been placed 2 hours ago for exactly this issue: https://github.com/Azure/azure-powershell/pull/10277

aim-for-better commented 4 years ago

Hi, you have mentioned that there is a pr to fix this. The PR is #10277

ghost commented 4 years ago

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.