search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.27k stars 3.87k forks source link

New-AzureRmAutomationModule fails when Storage Account Firewall is enabled #5885

Open arcotek-ltd opened 6 years ago

arcotek-ltd commented 6 years ago

Description

If the "Firewall" is enabled on the storage account holding the content (zip file of module), I get the following error:

Error importing the module MyModule. Import failed with the following error: Orchestrator.Shared.AsyncModuleImport.ModuleImportException: No content was read from the supplied ContentLink. [ContentLink.Uri=https://mysa.blob.core.windows.net/mycontainer/MyModule.zip]

The "Allow trusted Microsoft services to access this storage account" option is ticked, but unfortunately this doesn't do what one would hope.

If I enter the URL provided in the error, into a browser, I can access the file and the content is as expected.

I have tried adding all IP ranges that you publish for UK South and UK South 2, but this doesn't help.

Initially, I thought this was a DevTest issue, but I have whittled it down to the SA firewall feature, as mentioned here.

Script/Steps for Reproduction


$ContentLink = "https://mysa.blob.core.windows.net/mycontainer/MyModule.zip"

New-AzureRmAutomationModule `
        -ResourceGroupName "MyAARGName" `
        -AutomationAccountName "MyAAName" `
        -Name "MyModule" `
        -ContentLink $ContentLink `
        -Verbose

Please note, the error (above) comes from the portal. There is no way (that I can work out) to get the same error in PowerShell. All this PS cmdlet returns is "failed", same with Get-AzureRmAutomationModule

Module Version

PS R:\> Get-Module -Name AzureRM -ListAvailable

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                                             
---------- -------    ----                                ----------------                                                                                                                                                                                                             
Script     5.6.0      AzureRM

Environment Data

PS R:\> $PSVersionTable

Name                           Value                                                                                                                                                                                                                                                   
----                           -----                                                                                                                                                                                                                                                   
PSVersion                      5.1.14409.1012                                                                                                                                                                                                                                          
PSEdition                      Desktop                                                                                                                                                                                                                                                 
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                                                 
BuildVersion                   10.0.14409.1012                                                                                                                                                                                                                                         
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                                         
WSManStackVersion              3.0                                                                                                                                                                                                                                                     
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                                                     
SerializationVersion           1.1.0.1

Debug Output

PS R:\> $ContentLink = "https://mysa.blob.core.windows.net/mycontainer/MyModule.zip"

New-AzureRmAutomationModule `
        -ResourceGroupName "MyAARGName" `
        -AutomationAccountName "MyAAName" `
        -Name "MyModule" `
        -ContentLink $ContentLink `
        -Verbose `
        -Debug
DEBUG: 19:34:36 - NewAzureAutomationModule begin processing with ParameterSet '__AllParameterSets'.
DEBUG: 19:34:39 - using account id 'myuser@arcotek.co.uk'...
DEBUG: [Common.Authentication]: Authenticating using Account: 'myuser@arcotek.co.uk', environment: 'AzureCloud', tenant: 'a123456-b789-c101-d012-e123456789fa'
DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39:  - TokenCache: Serializing token cache with 2 items.

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39:  - TokenCache: Serializing token cache with 2 items.

DEBUG: [Common.Authentication]: Authenticating using configuration values: Domain: 'a123456-b789-c101-d012-e123456789fa', Endpoint: 'https://login.microsoftonline.com/', ClientId: '1950a258-227b-4e31-a9cf-717495945fc2', ClientRedirect: 'urn:ietf:wg:oauth:2.0:oob', ResourceClient
Uri: 'https://management.core.windows.net/', ValidateAuthrity: 'True'
DEBUG: [Common.Authentication]: Acquiring token using context with Authority 'https://login.microsoftonline.com/a123456-b789-c101-d012-e123456789fa/', CorrelationId: '00000000-0000-0000-0000-000000000000', ValidateAuthority: 'True'
DEBUG: [Common.Authentication]: Acquiring token using AdalConfiguration with Domain: 'a123456-b789-c101-d012-e123456789fa', AdEndpoint: 'https://login.microsoftonline.com/', ClientId: 'b234567-c789-d101-e012-f123456789ab', ClientRedirectUri: urn:ietf:wg:oauth:2.0:oob
DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39: c345678-d789-e101-f012-a123456789bc - AcquireTokenHandlerBase: === Token Acquisition started:
    Authority: https://login.microsoftonline.com/a123456-b789-c101-d012-e123456789fa/
    Resource: https://management.core.windows.net/
    ClientId: b234567-c789-d101-e012-f123456789ab
    CacheType: Microsoft.Azure.Commands.Common.Authentication.AuthenticationStoreTokenCache (2 items)
    Authentication Target: User

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 : 
DEBUG: 04/06/2018 18:34:39: c345678-d789-e101-f012-a123456789bc - TokenCache: Looking up cache for a token...

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39: c345678-d789-e101-f012-a123456789bc - TokenCache: An item matching the requested resource was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Verbose: 1 : 
DEBUG: 04/06/2018 18:34:39: c345678-d789-e101-f012-a123456789bc - TokenCache: 47.1654683416667 minutes left until token in cache expires

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39: c345678-d789-e101-f012-a123456789bc - TokenCache: A matching item (access token or refresh token or both) was found in the cache

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39: c345678-d789-e101-f012-a123456789bc - AcquireTokenHandlerBase: === Token Acquisition finished successfully. An access token was retuned:
    Access Token Hash: tRRT1LMjlzo1x2o5OGYU+YcsxfQbmJfurFQQJX3ZHR0=
    Refresh Token Hash: zml3qMhS5jorwE2q01GH+pibOY9db+/67PFLAiZ6/v0=
    Expiration Time: 04/06/2018 19:21:49 +00:00
    User Hash: IYpW8WL02laukR6oQHMcrkRbPaqdrHxVxWI5uzyJLc=

DEBUG: Microsoft.IdentityModel.Clients.ActiveDirectory Information: 2 : 
DEBUG: 04/06/2018 18:34:39:  - TokenCache: Serializing token cache with 2 items.

DEBUG: [Common.Authentication]: Received token with LoginType 'LiveId', Tenant: 'a123456-b789-c101-d012-e123456789fa', UserId: 'myuser@arcotek.co.uk'
DEBUG: [Common.Authentication]: Renewing Token with Type: 'Bearer', Expiry: '04/06/2018 19:21:49 +00:00', MultipleResource? 'True', Tenant: 'a123456-b789-c101-d012-e123456789fa', UserId: 'myuser@arcotek.co.uk'
DEBUG: [Common.Authentication]: User info for token DisplayId: 'myuser@arcotek.co.uk', Name: My Name, IdProvider: 'https://sts.windows.net/a123456-b789-c101-d012-e123456789fa/', Uid: 'e987654-f789-a101-b012-c123456789de'
DEBUG: [Common.Authentication]: Checking token expiration, token expires '04/06/2018 19:21:49 +00:00' Comparing to '04/06/2018 18:34:39 +00:00' With threshold '00:05:00', calculated time until token expiry: '00:47:09.9261003'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/d456789-e789-f101-a012-b123456789cd/resourceGroups/myAARGName/providers/Microsoft.Automation/automationAccounts/myAAName/modules/MyModule?api-version=2015-10-31

Headers:
Accept                        : application/json
x-ms-version                  : 2014-06-01

Body:
{
  "properties": {
    "contentLink": {
      "uri": "https://mysa.blob.core.windows.net/mycontainer/MyModule.zip"
    }
  },
  "name": "MyModule",
  "tags": {}
}

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Pragma                        : no-cache
x-ms-request-id               : a2aa1fe8-13b2-4e75-8610-20551537c6dc
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-ratelimit-remaining-subscription-writes: 1199
x-ms-correlation-request-id   : 26f96ec8-89d3-475c-a9ec-286eb237328a
x-ms-routing-request-id       : UKSOUTH:20180406T183440Z:26f96ec8-89d3-475c-a9ec-286eb237328a
X-Content-Type-Options        : nosniff
Cache-Control                 : no-cache
Date                          : Fri, 06 Apr 2018 18:34:40 GMT
Server                        : Microsoft-IIS/8.5
X-AspNet-Version              : 4.0.30319
X-Powered-By                  : ASP.NET

Body:
{
  "id": "/subscriptions/d456789-e789-f101-a012-b123456789cd/resourceGroups/myAARGName/providers/Microsoft.Automation/automationAccounts/myAAName/modules/MyModule",
  "name": "MyModule",
  "type": "Microsoft.Automation/AutomationAccounts/Modules",
  "location": "westeurope",
  "tags": {},
  "etag": null,
  "properties": {
    "isGlobal": false,
    "version": null,
    "sizeInBytes": 0,
    "activityCount": 0,
    "creationTime": "2018-04-06T01:26:17.95+01:00",
    "lastModifiedTime": "2018-04-06T19:34:40.55+01:00",
    "error": {
      "code": null,
      "message": null
    },
    "provisioningState": "Creating"
  }
}

DEBUG: [Common.Authentication]: Renewing Token with Type: 'Bearer', Expiry: '04/06/2018 19:21:49 +00:00', MultipleResource? 'True', Tenant: 'a123456-b789-c101-d012-e123456789fa', UserId: 'myuser@arcotek.co.uk'
DEBUG: [Common.Authentication]: User info for token DisplayId: 'myuser@arcotek.co.uk', Name: My Name, IdProvider: 'https://sts.windows.net/a123456-b789-c101-d012-e123456789fa/', Uid: 'e987654-f789-a101-b012-c123456789de'
DEBUG: [Common.Authentication]: Checking token expiration, token expires '04/06/2018 19:21:49 +00:00' Comparing to '04/06/2018 18:34:40 +00:00' With threshold '00:05:00', calculated time until token expiry: '00:47:08.4209498'
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/d456789-e789-f101-a012-b123456789cd/resourceGroups/myAARGName/providers/Microsoft.Automation/automationAccounts/myAAName/modules/MyModule?api-version=2015-10-31

Headers:
Accept                        : application/json
x-ms-version                  : 2014-06-01

Body:

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Pragma                        : no-cache
x-ms-request-id               : a2aa1fe8-13b2-4e75-8610-20551537c6dc
Strict-Transport-Security     : max-age=31536000; includeSubDomains
x-ms-ratelimit-remaining-subscription-reads: 14999
x-ms-correlation-request-id   : 0bc7b03a-bc3f-40fa-bccc-f6d810c0baf6
x-ms-routing-request-id       : UKSOUTH:20180406T183441Z:0bc7b03a-bc3f-40fa-bccc-f6d810c0baf6
X-Content-Type-Options        : nosniff
Cache-Control                 : no-cache
Date                          : Fri, 06 Apr 2018 18:34:40 GMT
Server                        : Microsoft-IIS/8.5
X-AspNet-Version              : 4.0.30319
X-Powered-By                  : ASP.NET

Body:
{
  "id": "/subscriptions/d456789-e789-f101-a012-b123456789cd/resourceGroups/myAARGName/providers/Microsoft.Automation/automationAccounts/myAAName/modules/MyModule",
  "name": "MyModule",
  "type": "Microsoft.Automation/AutomationAccounts/Modules",
  "location": "westeurope",
  "tags": {},
  "etag": null,
  "properties": {
    "isGlobal": false,
    "version": "1.0.7.5",
    "sizeInBytes": 123229,
    "activityCount": 17,
    "creationTime": "2018-04-06T01:26:17.95+01:00",
    "lastModifiedTime": "2018-04-06T19:34:40.55+01:00",
    "error": {
      "code": null,
      "message": ""
    },
    "provisioningState": "Creating"
  }
}

ResourceGroupName     : myAARGName
AutomationAccountName : myAAName
Name                  : MyModule
IsGlobal              : False
Version               : 1.0.7.5
SizeInBytes           : 123229
ActivityCount         : 17
CreationTime          : 06/04/2018 01:26:17 +01:00
LastModifiedTime      : 06/04/2018 19:34:40 +01:00
ProvisioningState     : Creating

DEBUG: AzureQoSEvent: CommandName - New-AzureRmAutomationModule; IsSuccess - True; Duration - 00:00:04.6219875; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 19:34:43 - NewAzureAutomationModule end processing.
DEBUG: 19:34:43 - NewAzureAutomationModule end processing.
arcotek-ltd commented 6 years ago

There is a feature request that will go some way to resolving this. Basically getting the "Allow trusted Microsoft services to access this storage account" checkbox to do what it should: https://feedback.azure.com/forums/34192--general-feedback/suggestions/33462937-whitelist-all-microsoft-services-in-storage-accoun

arcotek-ltd commented 6 years ago

This cmdlet Add-AzureRmStorageAccountNetworkRule must be very "Request-Intensive". Over the last two years of working solidly with ARM via REST and PS, I have never hit the "throttling request limit".

When using Remove-AzureRmStorageAccountNetworkRule or Add-AzureRmStorageAccountNetworkRule in a Foreach loop, they seems to pull the whole list if IP ranges , less the one removed each time. This might be why it's hitting the limit.

arcotek-ltd commented 6 years ago

OK, so I've found a work-around. As I said, I added all the UK South and UK South 2 datacenter IP ranges to the firewall, which didn't work. Whilst preparing the debug output, I noticed the location was westeurope and not UK. I've now added the West Europe IP ranges and it works*.

I had to remove the UK South and UK South 2 IPs first (which took a long time as I hit the throttling request limit issue - mentioned above) and I still could not add the last 10 published IP ranges as I hit the maximum IP ranges allowed (100). There are currently 110 IP ranges in West Europe.

Update*

It works for West Europe, but not UK South. I'm testing against different SAs - one in a PAYG sub and the other is in a DevTest sub. The debug output is from the PAYG subscription. who's AA is in West Europe.

Re-running New-AzureRmAutomationModule against the DevTest AA, the debug output confirms the location as uksouth, however, even after adding all the IP ranges you publish, it still doesn't work. The theory has been proved, so either there is an issue with DevTest or you're IP list is missing the required IP range, which I assume is uksouth.management.azure.com

arcotek-ltd commented 6 years ago

Just added IP address for uksouth.management.azure.com to the firewall. No cigar.

markcowl commented 6 years ago

@vrdmr can you take a look?

arcotek-ltd commented 6 years ago

@vrdmr What does the "automation-elephant" tag mean please?

vrdmr commented 6 years ago

@arcotek-ltd I assigned it to the internal team in automation which takes look at the Modules.

sourabhguha commented 6 years ago

This is a feature request for the Azure Storage Account. Please open the issue with them.

arcotek-ltd commented 6 years ago

@sourabhguha, How, exactly, is this a feature request? Enabling the firewall breaks the cmdlet, The features already exist!!! This cmdlet and the SA firewall simply don't work together, as I have gone to great lengths to prove to you.

You seem to be saying that you are not prepared to "fix" an existing feature, compounded by the fact that your list of published UK South IP address is incomplete.

If you are not prepared to fix this BUG, then I will have to move my storage account to a region that does work with the SA firewall. Again, in my testing, I have proven that this cmdlet does work in West Europe.

sourabhguha commented 6 years ago

Let me elaborate - The setting in the storage account has the following option "Allow trusted Microsoft services to access this storage account". This option does not seem to recognize cmdlet calls coming from Azure Automation which is hosted in Microsoft datacenters. This would be a issue request to the Azure Storage team to ensure that calls coming from Azure Automation is recognized as trusted Microsoft service.

csand-msft commented 5 years ago

We are investigating what is involved with making Automation a trusted service with Storage.

chixcancode commented 5 years ago

@csand-msft for now, what would be the work-around? We have a customer with storage service endpoints enabled and we need our DSC in Azure Automation to access files in blob storage.

csand-msft commented 5 years ago

The Automation team is working on implementing a Service Tag for the Automation services. This will minimize the work to include Automation endpoints in security rules. The ETA is around Sept. 2019. We will announce when the service tag is ready for use.

https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags

lavermil commented 4 years ago

As far as I can find NSGs are not applicable to Storage Accounts. The only firewall option is IP addresses. Am I missing something?