search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.97k stars 2.95k forks source link

`az storage account network-rule add` is exiting too early? #18415

Open yellowhat opened 3 years ago

yellowhat commented 3 years ago

Describe the bug

Hi, I would like to create a storage account with a network rule to allow only an ip address:

RG=zz
SA=ewq${RANDOM}qwe
CNT=test
LOCATION=westeurope

az group create \
    --name "$RG" \
    --location "$LOCATION"

az storage account create \
    --name "$SA" \
    --resource-group "$RG" \
    --access-tier Hot \
    --sku Standard_LRS \
    --default-action Deny \
    --https-only true \
    --min-tls-version TLS1_2

az storage account network-rule add \
    --account-name "$SA" \
    --resource-group "$RG" \
    --ip-address "$(curl -s ifconfig.me)"

# sleep 15

az storage container create \
    --name "$CNT" \
    --account-name "$SA"

Errors:

Client-Request-ID=629bdd78-c8f5-11eb-a95a-00155dee1d3d Retry policy did not allow for a retry: Server-Timestamp=Wed, 09 Jun 2021 07:36:19 GMT, Server-Request-ID=61861b58-501e-004d-1902-5d8ebb000000, HTTP status code=403, Exception=This request is not authorized to perform this operation. ErrorCode: AuthorizationFailure<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.RequestId:61861b58-501e-004d-1902-5d8ebb000000Time:2021-06-09T07:36:19.1222232Z</Message></Error>.

The request may be blocked by network rules of storage account. Please check network rule set using 'az storage account show -n accountname --query networkRuleSet'.
If you want to change the default action to apply when no rule matches, please use 'az storage account update'.

If I uncomment sleep 15 it does work.

Expected Behavior

No error reported

Environment Summary

Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.33
Python 3.9.5
Installer: 

azure-cli 2.24.2

Extensions:
blueprint 0.3.0

Thanks

yonzhan commented 3 years ago

storage

Juliehzl commented 3 years ago

Hi @yellowhat, I have tried you scenario and I cannot reproduce your issue. The network access rule will take effect until you remove it from your storage account. You could check if the ip rule is added successfully with az storage account network-rule list.

yellowhat commented 3 years ago

Dear @Juliehzl, I have just tried with the provided code in the first post and gave me the same error. (az version 2.26.0). As you suggested a az storage account network-rule list just after the creation did the trick (maybe forced a refresh?)

Thanks

yellowhat commented 3 years ago

I have just tried to following:

RG=zzz
SA=ewq${RANDOM}qwe
CNT=test
LOCATION=westeurope

az group create \
    --name "$RG" \
    --location "$LOCATION"

az storage account create \
    --name "$SA" \
    --resource-group "$RG" \
    --access-tier Hot \
    --sku Standard_LRS \
    --default-action Deny \
    --https-only true \
    --min-tls-version TLS1_2

az storage account network-rule add \
    --account-name "$SA" \
    --resource-group "$RG" \
    --ip-address "$(curl -s ifconfig.me)"

az storage account network-rule list \
    --account-name "$SA" \
    --resource-group "$RG"

# sleep 15

az storage container create \
    --name "$CNT" \
    --account-name "$SA"

I am getting:

Client-Request-ID=277dee3c-e887-11eb-9dea-00155d6cc2fe Retry policy did not allow for a retry: Server-Timestamp=Mon, 19 Jul 2021 11:47:54 GMT, Server-Request-ID=5cf2ea68-601e-00f1-4b93-7cf6c5000000, HTTP status code=403, Exception=This request is not authorized to perform this operation. ErrorCode: AuthorizationFailure<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.RequestId:5cf2ea68-601e-00f1-4b93-7cf6c5000000

Time:2021-07-19T11:47:55.1749837Z</Message></Error>.

The request may be blocked by network rules of storage account. Please check network rule set using 'az storage account show -n accountname --query networkRuleSet'.
If you want to change the default action to apply when no rule matches, please use 'az storage account update'.

Adding the sleep 15, made it work.

May I ask to reopen this issue?

Juliehzl commented 3 years ago

I have just tried to following:

RG=zzz
SA=ewq${RANDOM}qwe
CNT=test
LOCATION=westeurope

az group create \
    --name "$RG" \
    --location "$LOCATION"

az storage account create \
    --name "$SA" \
    --resource-group "$RG" \
    --access-tier Hot \
    --sku Standard_LRS \
    --default-action Deny \
    --https-only true \
    --min-tls-version TLS1_2

az storage account network-rule add \
    --account-name "$SA" \
    --resource-group "$RG" \
    --ip-address "$(curl -s ifconfig.me)"

az storage account network-rule list \
    --account-name "$SA" \
    --resource-group "$RG"

# sleep 15

az storage container create \
    --name "$CNT" \
    --account-name "$SA"

I am getting:

Client-Request-ID=277dee3c-e887-11eb-9dea-00155d6cc2fe Retry policy did not allow for a retry: Server-Timestamp=Mon, 19 Jul 2021 11:47:54 GMT, Server-Request-ID=5cf2ea68-601e-00f1-4b93-7cf6c5000000, HTTP status code=403, Exception=This request is not authorized to perform this operation. ErrorCode: AuthorizationFailure<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.RequestId:5cf2ea68-601e-00f1-4b93-7cf6c5000000

Time:2021-07-19T11:47:55.1749837Z</Message></Error>.

The request may be blocked by network rules of storage account. Please check network rule set using 'az storage account show -n accountname --query networkRuleSet'.
If you want to change the default action to apply when no rule matches, please use 'az storage account update'.

Adding the sleep 15, made it work.

May I ask to reopen this issue?

I can repro your scenario with all the same settings with you. But Azure CLI just sends request to server, and for such kind of issue, we need to see if it will take some time to deploy network rules in service side.

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details
## Describe the bug Hi, I would like to create a storage account with a network rule to allow only an ip address: ``` RG=zz SA=ewq${RANDOM}qwe CNT=test LOCATION=westeurope az group create \ --name "$RG" \ --location "$LOCATION" az storage account create \ --name "$SA" \ --resource-group "$RG" \ --access-tier Hot \ --sku Standard_LRS \ --default-action Deny \ --https-only true \ --min-tls-version TLS1_2 az storage account network-rule add \ --account-name "$SA" \ --resource-group "$RG" \ --ip-address "$(curl -s ifconfig.me)" # sleep 15 az storage container create \ --name "$CNT" \ --account-name "$SA" ``` **Errors:** ``` Client-Request-ID=629bdd78-c8f5-11eb-a95a-00155dee1d3d Retry policy did not allow for a retry: Server-Timestamp=Wed, 09 Jun 2021 07:36:19 GMT, Server-Request-ID=61861b58-501e-004d-1902-5d8ebb000000, HTTP status code=403, Exception=This request is not authorized to perform this operation. ErrorCode: AuthorizationFailureAuthorizationFailureThis request is not authorized to perform this operation.RequestId:61861b58-501e-004d-1902-5d8ebb000000Time:2021-06-09T07:36:19.1222232Z. The request may be blocked by network rules of storage account. Please check network rule set using 'az storage account show -n accountname --query networkRuleSet'. If you want to change the default action to apply when no rule matches, please use 'az storage account update'. ``` If I uncomment `sleep 15` it does work. ## Expected Behavior No error reported ## Environment Summary ``` Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.33 Python 3.9.5 Installer: azure-cli 2.24.2 Extensions: blueprint 0.3.0 ``` Thanks
Author: yellowhat
Assignees: Juliehzl
Labels: `OKR3.2 Candidate`, `Service Attention`, `Storage`, `feature-request`
Milestone: Jul 2021 (2021-08-03)
yellowhat commented 3 years ago

Are there any update on this?

Thanks

evelyn-ys commented 3 years ago

Wait for service side to explain

yellowhat commented 3 years ago

Are there any update on this?

Thanks

yellowhat commented 2 years ago

Is there a ticket that can be followed? Thanks

PatrickMessierGoC commented 2 years ago

Same issue here. Unable to use the following command sequence in a pipeline in a predictable way. What's going on?

  az storage account network-rule add \
    --account-name "${parameters[--storage-account-name]}"\
    --action "Allow" \
    --ip-address "${build_agent_public_ip}" \
    --output none

  blob_container_exists="$(az storage container exists \
    --account-key "${parameters[--storage-account-key]}" \
    --account-name "${parameters[--storage-account-name]}" \
    --name "${blob_service_container_name}" \
    --output tsv
  )"

Error

ERROR: Client-Request-ID=d0a364ae-dd27-11ec-9474-57cdb6597cbe Retry policy did not allow for a retry: Server-Timestamp=Thu, 26 May 2022 19:12:41 GMT, Server-Request-ID=f469d2ca-701e-0003-0a34-71159f000000, HTTP status code=403, Exception=This request is not authorized to perform this operation. ErrorCode: AuthorizationFailure<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.RequestId:f469d2ca-701e-0003-0a34-71159f000000Time:2022-05-26T19:12:42.3231221Z</Message></Error>.
ERROR: 
The request may be blocked by network rules of storage account. Please check network rule set using 'az storage account show -n accountname --query networkRuleSet'.
If you want to change the default action to apply when no rule matches, please use 'az storage account update'.
markdebruijne commented 9 months ago

Also (still) having this issue, with a 20 seconds sleep before executing the next pipeline task

az storage account network-rule add --account-name ${{ parameters.storageAccountName }} --ip-address $(hostedAgentOutboundIpAddress) --action Allow

versions

azure-cli                         2.46.0 *
core                              2.46.0 *
telemetry                          1.0.8
PatrickMessierGoC commented 9 months ago

My experimentation showed me that it doesn't work for good after a given period. In fact, it works randomly from the time the az storage account network-rule add command return.

I ultimately opened a case with Microsoft Azure about this issue last year. I was trying to open up the storage account's firewall in order to allow an Azure DevOps Build Agent to interact with the storage account content. I've been told that this use case was not supported by Microsoft.

My understanding - this has yet to be proven - is that traffic to Storage Account from a Build Agent may be routed through various internal service endpoints and Internet gateways on Azure side. Thus, the IP rule added to the Storage Account's firewall based on a dynamically retrieved client IP addres end up only working when the traffic happens to go through the same route.

markdebruijne commented 9 months ago

I was trying to open up the storage account's firewall in order to allow an Azure DevOps Build Agent to interact with the storage account content. [..] Thus, the IP rule added to the Storage Account's firewall based on a dynamically retrieved client IP addres end up only working when the traffic happens to go through the same route.

That is exactly our use case as well.

  1. We are determining the public outbound IP-address of the microsoft hosted build agent 
    $outboundIP = (Invoke-RestMethod http://ipinfo.io/json).ip
Write-Host "This CI/CD hosted agent has public IP : $outboundIP"      
Write-Host "##vso[task.setvariable variable=hostedAgentOutboundIpAddress]$outboundIP"
  2. Add network rule for that determined IP
  3. Execute the intended storage accounts operations
  4. Remove network rule for that determined IP

My experimentation showed me that it doesn't work for good after a given period Ours as well. It is just buggy. Sometimes it just misfires and the added network rule isn't reflected up to three minutes after. And if it works, it works straight away

Added an inbetween step to verify & wait on the change to be become effective

- task: AzureCLI@2
  displayName: Verify firewall adjustment for ${{ parameters.storageAccountName }}
  inputs:
    azureSubscription: ${{ parameters.azureServiceConnectionAzureInfra }}
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
      Write-Host "Wait 20 seconds to wait on updated IP restrictions to become effective"
      Start-Sleep -Seconds 20
      Write-Host "Verify whether IP-address $(hostedAgentOutboundIpAddress) is already allowed in the storage account firewall."
      $matchedIpRules = az storage account network-rule list --account-name ${{ parameters.storageAccountName }} --output json --query "ipRules[?action == 'Allow'] | [?ipAddressOrRange == '$(hostedAgentOutboundIpAddress)']" | ConvertFrom-Json
      if($null -eq $matchedIpRules -or $matchedIpRules.count -ne 1) {
        Write-Host "##vso[task.logissue type=error]Intended IP-address $(hostedAgentOutboundIpAddress) not yet allowed in the storage account firewall.";
        EXIT 1
      }
  retryCountOnTaskFailure: 6 # task might fail if the firewall adjustment (previous task) isn't effective yet. https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-195-update#automatic-retries-for-a-taskVeri

However some times the az storage account network-rule list even reports the temporarily IP-address as whitelisted, but the next task (yes, same pipeline execution job) executing the storage account operation is failing due to the IP being block still 🤷

markdebruijne commented 8 months ago

@yonzhan It is read as bug/incident? As I see it tagged as backlog (and not something more "incident-ish")

markdebruijne commented 7 months ago

@calvinhzy do you have an update perhaps?

markdebruijne commented 7 months ago

I came across this blog post, sharing it here as also others might didn't find it yet: Azure DevOps Build Agent not able to access Azure Storage TLTR (as I understand it)