search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4.01k stars 2.99k forks source link

Not possible to store flow logs in a storage in different subscription using cli #21724

Open Candelit opened 2 years ago

Candelit commented 2 years ago

Hi. I'm building CLI code to create our network as code, using cli as our default. Now, I have been struggeling all day to get it to work.

This is, we have a centralized storage and log analytics workspace in a separate subscription that we want to use for every NSG on the network.

Tis article from 2018 states that it is possible, it is if done using the portal. https://azure.microsoft.com/es-es/blog/new-azure-network-watcher-integrations-and-network-security-group-flow-logging-updates/

But, using CLI and az network watcher flow-log create it is not possible. Now I finally got this:

(FlowLogTargetResourceIsInDifferentSubscription) Target resource /subscriptions/123e3418-XXXXXXXXXX--XXXX-a1d9a0dc/resourceGroups/rg-dev-infra/providers/Microsoft.Network/networkSecurityGroups/nsg-vn-devtest-sn-internal-default should be in the same subscription as flow log resource d52ec4e6-XXXX-XXXXXXXXXX-cf3. Code: FlowLogTargetResourceIsInDifferentSubscription Message: Target resource /subscriptions/123e3418-XXXXXXXXXX--XXXX-a1d9a0dc/resourceGroups/rg-dev-infra/providers/Microsoft.Network/networkSecurityGroups/nsg-vn-devtest-sn-internal-default should be in the same subscription as flow log resource d52ec4e6-3349-XXXX-XXXXXXXXXX-cf3.

This totally contradicts what the article says and also what can be done using the portal.

The code I use: NSG_ID=$(az network nsg show --resource-group $RESOURCE_GROUP --name $NSG --query "id" --output tsv) FLOWLOG_STORAGE_ID=$(az storage account show --subscription $FLOWLOG_SUBSCRIPTION --resource-group $FLOWLOG_RESOURCE_GROUP --name $FLOWLOG_STORAGE --query "id" --output tsv) WORKSPACE_ID=$(az monitor log-analytics workspace show --subscription $FLOWLOG_SUBSCRIPTION --resource-group $FLOWLOG_RESOURCE_GROUP --workspace-name $FLOWLOG_WORKSPACE --query "id" --output tsv)

az network watcher flow-log create \ --location $LOCATION \ --name ${NSG}-flowlog \ --nsg $NSG_ID \ --subscription $FLOWLOG_SUBSCRIPTION \ --storage-account $FLOWLOG_STORAGE_ID --workspace $WORKSPACE_ID \ --log-version 2 \ --retention 30

Thanks // Thomas

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

yonzhan commented 2 years ago

network

YuanyuanNi commented 2 years ago

Hi: @Candelit As the error info shows that CLI does not support the resources from different subscriptions now, "nsg-vn-devtest-sn-internal-default should be in the same subscription as flow log resource d52ec4e6-3349-XXXX-XXXXXXXXXX-cf3". Best wishes.

Candelit commented 2 years ago

Hi: @Candelit As the error info shows that CLI does not support the resources from different subscriptions now, "nsg-vn-devtest-sn-internal-default should be in the same subscription as flow log resource d52ec4e6-3349-XXXX-XXXXXXXXXX-cf3". Best wishes.

Hi, an accurate observation. But why should CLI support less than what can be done in the portal or using PowerShell? There is not any article that states this, I had to spend a lot of time trying before I got to this error message, maybe I should just drop it there and let everyone else make the same journey, but I'm still hopeful, that Microsoft can fix it for us as it is possible in both the portal and PowerShell... In a 'real' environement, it is in my experience not often that these types of resources are in the same subscription. You would if nothing else want separation of the network logs for forensics...

ghost commented 2 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details
Hi. I'm building CLI code to create our network as code, using cli as our default. Now, I have been struggeling all day to get it to work. This is, we have a centralized storage and log analytics workspace in a separate subscription that we want to use for every NSG on the network. Tis article from 2018 states that it is possible, it is if done using the portal. https://azure.microsoft.com/es-es/blog/new-azure-network-watcher-integrations-and-network-security-group-flow-logging-updates/ But, using CLI and az network watcher flow-log create it is not possible. Now I finally got this: (FlowLogTargetResourceIsInDifferentSubscription) Target resource /subscriptions/123e3418-XXXXXXXXXX--XXXX-a1d9a0dc/resourceGroups/rg-dev-infra/providers/Microsoft.Network/networkSecurityGroups/nsg-vn-devtest-sn-internal-default should be in the same subscription as flow log resource d52ec4e6-XXXX-XXXXXXXXXX-cf3. Code: FlowLogTargetResourceIsInDifferentSubscription Message: Target resource /subscriptions/123e3418-XXXXXXXXXX--XXXX-a1d9a0dc/resourceGroups/rg-dev-infra/providers/Microsoft.Network/networkSecurityGroups/nsg-vn-devtest-sn-internal-default should be in the same subscription as flow log resource d52ec4e6-3349-XXXX-XXXXXXXXXX-cf3. This totally contradicts what the article says and also what can be done using the portal. The code I use: NSG_ID=$(az network nsg show --resource-group $RESOURCE_GROUP --name $NSG --query "id" --output tsv) FLOWLOG_STORAGE_ID=$(az storage account show --subscription $FLOWLOG_SUBSCRIPTION --resource-group $FLOWLOG_RESOURCE_GROUP --name $FLOWLOG_STORAGE --query "id" --output tsv) WORKSPACE_ID=$(az monitor log-analytics workspace show --subscription $FLOWLOG_SUBSCRIPTION --resource-group $FLOWLOG_RESOURCE_GROUP --workspace-name $FLOWLOG_WORKSPACE --query "id" --output tsv) az network watcher flow-log create \ --location $LOCATION \ --name ${NSG}-flowlog \ --nsg $NSG_ID \ --subscription $FLOWLOG_SUBSCRIPTION \ --storage-account $FLOWLOG_STORAGE_ID --workspace $WORKSPACE_ID \ --log-version 2 \ --retention 30 Thanks // Thomas --- #### Document Details ⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.* * ID: 132ba089-fae4-ddf8-1119-4e83c6be94f5 * Version Independent ID: 31f4477d-92a5-7126-89df-44d9ea397e81 * Content: [az network watcher flow-log](https://docs.microsoft.com/en-us/cli/azure/network/watcher/flow-log?view=azure-cli-latest) * Content Source: [latest/docs-ref-autogen/network/watcher/flow-log.yml](https://github.com/MicrosoftDocs/azure-docs-cli/blob/main/latest/docs-ref-autogen/network/watcher/flow-log.yml) * Service: **virtual-network** * GitHub Login: @rloutlaw * Microsoft Alias: **routlaw**
Author: Candelit
Assignees: necusjz, kairu-ms
Labels: `Network`, `Service Attention`, `customer-reported`, `Auto-Assign`, `Traiged`
Milestone: Backlog
YuanyuanNi commented 2 years ago

![image](https://user-images.githubusercontent.com/95396631/163325321-3f76314f-6c6a-439b-b231-501aa2ae3a76.png Hi, @Candelit we have reproduced this issue and routing this to service team, thank you for your reporting. Best wishes.

JamieTaffurelli commented 1 year ago

Hi, is there any update on this? I am facing the same issue in Terraform for both the azurerm and azapi providers.

Candelit commented 1 year ago

I can now report that it works fine when deployed using bicep and ARM templates.

selengokkaya commented 1 year ago

Hi,

Is there any update on this? I am still facing the same issue in Terraform. We have log analytics workspace in a separate subscription than NSGs and when deploying NSG Flow logs I get this error even the workspace and resource group exist.

Error: log analytics workspaces "log-xxxxxxxx" (Resource Group "rg-yyyyyyyy") was not found

When I deploy NSG flow logs to a log analytics workspace that is in the same subscription it succeeds.

zshao9 commented 11 months ago

See similar errors using Terraform azurerm provider:

│ Error: creating "Flow Log: (Name \"nsg-flowlogsdbx_workspace_nsg_eastusprod\" / Network Watcher Name \"NetworkWatcher_aaaaaaa\" / Resource Group \"gggggggggggg\")": network.FlowLogsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="FlowLogTargetResourceIsInDifferentSubscription" Message="Target resource /subscriptions/xxxxxxxxxx/resourceGroups/dbx_workspace_eastusprod_rg/providers/Microsoft.Network/networkSecurityGroups/dbx_workspace_nsg_eastusprod should be in the same subscription as flow log resource yyyyyyyyyy." Details=[]
Harsha-CS commented 9 months ago

@zshao9 , @efd7887 I'm from the NSG flow log product group. The behavior mentioned in product documentation is accurate - User can create an NSG flow log and select a storage account in in a subscription that is different from the subscription of NSG. However, user cannot create an NSG flow log where the flow log resource is in a subscription different from the NSG itself. This is likely the case of your error, further details below.

Considerations for NSG flow logs Storage account Location: The storage account must be in the same region as the network security group. Subscription: The storage account must be in the same subscription of the network security group or in a subscription associated with the same Microsoft Entra tenant of the network security group's subscription. Performance tier: The storage account must be standard. Premium storage accounts aren't supported. Self-managed key rotation: If you change or rotate the access keys to your storage account, NSG flow logs stop working. To fix this problem, you must disable and then re-enable NSG flow logs.

Ref: https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview?WT.mc_id=AZ-MVP-5005118#considerations-for-nsg-flow-logs

Here is the validated creation that succeeded:

  1. Set context to same as NSG subscription Set-AzContext -Subscription ********-****-****-****-********9379

  2. Create NSG flow log New-AzNetworkWatcherFlowLog -Location eastus2euap -Name enabledtestfl -TargetResourceId "/subscriptions/********-****-****-****-********9379/resourceGroups/****test3/providers/Microsoft.Network/networkSecurityGroups/enabledCatetgoriesTest-nsg" -StorageId "/subscriptions/********-****-****-****-********5a16/resourceGroups/****test2/providers/Microsoft.Storage/storageAccounts/**********" -Enabled $true

  3. Success message

    ResourceGroupName Name     Location    ProvisioningState   TargetResourceId                     TargetResourceGuid                   StorageId
----------------- ----     --------    ----------------- ----------------                        ------------------                   ---------                  
             enabledtestfl eastus2euap Succeeded         /subscriptions/********-****-****-****-********9379/resourceGroups/****test3/providers/Microsoft.Network/networkSecurityGroups/enabledCatetgoriesTest-nsg ********-****-****-****-********a963 /subscriptions/********-****…

Validation with failure:

  1. Set context to different subscription other than NSG subscription, in this case we are using the subscription of storage account as an example Set-AzContext -Subscription ********-****-****-****-********5a16

  2. Create NSG flow log, notice the command is same as before, the only difference being the subscription context New-AzNetworkWatcherFlowLog -Location eastus2euap -Name enabledtestfl -TargetResourceId "/subscriptions/********-****-****-****-********9379/resourceGroups/****test3/providers/Microsoft.Network/networkSecurityGroups/enabledCatetgoriesTest-nsg" -StorageId "/subscriptions/********-****-****-****-********5a16/resourceGroups/****test2/providers/Microsoft.Storage/storageAccounts/**********" -Enabled $true

  3. Failure message

    New-AzNetworkWatcherFlowLog: Operation returned an invalid status code 'BadRequest'
StatusCode: 400
ReasonPhrase: 
ErrorCode: FlowLogTargetResourceIsInDifferentSubscription
ErrorMessage: Target resource /subscriptions/********-****-****-****-********9379/resourceGroups/****test3/providers/Microsoft.Network/networkSecurityGroups/enabledCatetgoriesTest-nsg should be in the same subscription as flow log resource ********-****-****-****-************.