search

Azure / AzOps

AzOps is a PowerShell module which deploys (Push) ARM Resource Templates & Bicep files at all Azure scope levels and exports (Pull) ARM resource hierarchy.
https://aka.ms/AzOps
MIT License
371 stars 158 forks source link

Deletion of bicep file with parameter file - JSON serialization #849

Closed mathwro closed 3 months ago

mathwro commented 5 months ago

Describe the bug

When i'm deleting a bicep file and it's matching json parameter file from our management group structure, the validation steps are failing due to failing to serialize the json of the parameter file.

https://github.com/Azure/AzOps/blob/3a77d8cd73216f41480113470f27cc2133f21471/src/functions/Invoke-AzOpsPush.ps1#L400

Steps to reproduce

  1. Add bicep file and parameter file to management group structure (Example, network.bicep & network.parameters.json)
  2. Merge to main and deploy
  3. Now delete both files

Screenshots

image

daltondhcp commented 5 months ago

Thanks for reporting! Quick note: deleting custom templates only removes them from the repo, not Azure (see Issue #785 for details).

I couldn't replicate the error with a valid parameters.json. For me, it only occurs with an invalid JSON, like missing a closing brace.

Can you run Get-Content network.parameters.json | ConvertFrom-Json -AsHashTable locally successfully? If so, sharing your template and parameter file would help us debug further. Thanks! image

mathwro commented 5 months ago

I've tried to replicate the commands done, just locally and i dont get any serialization errors when performed locally with powershell 7.4.

image

Here is parameter file with a identical structure.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.3",
  "parameters": {
    "environment": {
      "value": "pt"
    },
    "application": {
      "value": "demo"
    },
    "partition": {
      "value": "swc"
    },
    "identifier": {
      "value": "001"
    },
    "networkConfiguration": {
      "value": {
        "type": "internal",
        "vnetAddressPrefix": [
          "10.0.0.0/21"
        ],
        "peeredVnetIpPrefixes": [
          "10.0.0.0/23"
        ],
        "dhcpOptions": {
          "dnsServers": [
            "1.1.1.1"
          ]
        },
        "subnets": [
          {
            "subnet": "10.0.128.0/22",
            "udr": "udr01",
            "serviceEndpoints": [],
            "delegations": [
              {
                "name": "Microsoft.ContainerService/managedClusters",
                "properties": {
                  "serviceName": "Microsoft.ContainerService/managedClusters"
                }
              }
            ],
            "nsgRules": [
            ]

          },
          {
            "subnet": "10.0.0.0/23",
            "udr": "udr01",
            "serviceEndpoints": [],
            "delegations": [
              {
                "name": "Microsoft.ContainerService/managedClusters",
                "properties": {
                  "serviceName": "Microsoft.ContainerService/managedClusters"
                }
              }
            ],
            "nsgRules": []              
          },
          {
            "subnet": "10.192.132.0/24",
            "udr": "udr01",
            "serviceEndpoints": [],
            "delegations": [
              {
                "name": "Microsoft.ContainerService/managedClusters",
                "properties": {
                  "serviceName": "Microsoft.ContainerService/managedClusters"
                }
              }
            ],
            "nsgRules": []
          },
          {
            // "snet-lb",
            "subnet": "10.0.0.0/26",
            "udr": "udr-lndz-to-conn",
            "serviceEndpoints": [],
            "nsgRules": []
          },
          {
            "subnet": "10.0.0.128/27",
            "udr": "udr01",
            "serviceEndpoints": [
              {
                "service": "Microsoft.ContainerRegistry"
              }
            ],
            "nsgRules": [
              {
                "name": "OUT",
                "properties": {
                  "protocol": "Tcp",
                  "sourcePortRange": "*",
                  "destinationPortRange": "443",
                  "sourceAddressPrefix": "10.0.0.128/27", 
                  "destinationAddressPrefix": "10.0.0.0/29",
                  "access": "Allow",
                  "priority": 1101,
                  "direction": "Outbound"
                }
              }
            ]
          },
          {
            "subnet": "10.0.0.64/26",
            "udr": "udr01",
            // "delegations": [],
            "serviceEndpoints": [
              {
              "service": "Microsoft.ContainerRegistry"
              }
            ],
            "nsgRules": [
              {
                "name": "OUT-443",
                "properties": {
                  "protocol": "Tcp",
                  "sourcePortRange": "*",
                  "destinationPortRange": "443",
                  "sourceAddressPrefix": "10.0.0.64/26",
                  "destinationAddressPrefix": "10.0.0.0/29",
                  "access": "Allow",
                  "priority": 1101,
                  "direction": "Outbound"
                }
              }
            ]
          },
          {
            "subnet": "10.192.1.160/27",
            "udr": "udr01",
            "serviceEndpoints": [
              {
                "service": "Microsoft.ContainerRegistry"
              }
            ],
            "nsgRules": [
              {
                "name": "OUT-443F",
                "properties": {
                  "protocol": "Tcp",
                  "sourcePortRange": "*",
                  "destinationPortRange": "443",
                  "sourceAddressPrefix": "10.192.1.160/27",
                  "destinationAddressPrefix": "10.192.0.0/29",
                  "access": "Allow",
                  "priority": 1101,
                  "direction": "Outbound"
                }
              }
            ]
          },
          {
            "subnet": "10.0.0.0/29",
            "udr": "udr01",
            // "delegations": [],
            // "serviceEndpoints": [
            //   {
            //     "service": "Microsoft.KeyVault"
            //   }
            // ],
            "nsgRules": []
          },
          {
            "subnet": "10.0.0.208/29",
            "udr": "udr01",
            // "delegations": [],
            // "serviceEndpoints": [
            //   {
            //     "service": "Microsoft.KeyVault"
            //   }
            // ],
            "nsgRules": []
          },
          {
            // "snet-pe",
            "subnet": "10.0.0.216/29",
            "nsgRules": [],
            "udr": "udr01",
            // "delegations": [],
            "nsgRules": []
          },
          {
            // "snet-pe",
            "subnet": "10.0.0.200/29",
            "nsgRules": [],
            "udr": "udr01",
            // "delegations": [],
            "nsgRules": []
          }
        ]
      }
    },
    "routeTables": {
      "value": [
        {
          "routeTablePrefix": "udr01",
          "routes": [
            {
              "name": "On-Prem",
              "properties": {
                "addressPrefix": "0.0.0.0/0",
                "nextHopType": "VirtualAppliance",
                "nextHopIpAddress": "10.0.2.4"
              }
            }
          ]
        }
      ]
    },
    "hubVnetName": {
      "value": "vnet001"
    },
    "hubVnetRG": {
      "value": "rg-005"
    },
    "hubSubscriptionId": {
      "value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
  }
}
daltondhcp commented 5 months ago

I am unable to repro with just your parameter file (no errors when deleting), can you please share the bicep deployment template as well?

mathwro commented 5 months ago

A bit delayed but finally got some time to clean it up properly.

I've verified again, and are still not seeing any local errors regardless of what i'm doing.

param networkConfiguration object
param application string
param partition string
param rgLocation string = 'West Europe'
param environment string
param identifier string
param routeTables array
param hubVnetName string
param hubVnetRg string = resourceGroup().name
param hubSubscriptionId string

var subnetNamePrefix = 'snet-${application}-${partition}-'
var networkSecurityGroupNamePrefix = 'nsg-${subnetNamePrefix}'
var intraSubnetRules = [for (item, i) in networkConfiguration.subnets: {
  rules: [
    {
      name: 'AllowCurrentSubnet'
      properties: {
        protocol: '*'
        sourcePortRange: '*'
        sourceAddressPrefix: item.subnet
        destinationAddressPrefix: '*'
        destinationPortRange: '*'
        access: 'Allow'
        priority: 4095
        direction: 'Inbound'
      }
    }
    {
      name: 'DenyInternalVnetInbound'
      properties: {
        protocol: '*'
        sourcePortRange: '*'
        sourceAddressPrefixes: union(networkConfiguration.vnetAddressPrefix, networkConfiguration.peeredVnetIpPrefixes)
        destinationAddressPrefix: '*'
        destinationPortRange: '*'
        access: 'Deny'
        priority: 4096
        direction: 'Inbound'
      }
    }
  ]
}]

resource networkSecurityGroups 'Microsoft.Network/networkSecurityGroups@2022-01-01' = [for (item, i) in networkConfiguration.subnets: {
  name: '${networkSecurityGroupNamePrefix}${padLeft((i + 1), 3, '0')}'
  location: rgLocation
  tags: {
    networkZone: networkConfiguration.type
  }
  properties: {
    securityRules: concat(item.nsgRules, intraSubnetRules[i].rules)
  }
}]
{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.3",
  "parameters": {
    "environment": {
      "value": "pt"
    },
    "application": {
      "value": "demo"
    },
    "partition": {
      "value": "swc"
    },
    "identifier": {
      "value": "001"
    },
    "networkConfiguration": {
      "value": {
        "type": "internal",
        "vnetAddressPrefix": [
          "10.0.0.0/21"
        ],
        "peeredVnetIpPrefixes": [
          "10.0.10.0/23"
        ],
        "dhcpOptions": {
          "dnsServers": [
            "1.1.1.1"
          ]
        },
        "subnets": [
          {
            "subnet": "10.0.128.0/22",
            "udr": "udr01",
            "serviceEndpoints": [],
            "delegations": [
              {
                "name": "Microsoft.ContainerService/managedClusters",
                "properties": {
                  "serviceName": "Microsoft.ContainerService/managedClusters"
                }
              }
            ],
            "nsgRules": [
            ]

          },
          {
            "subnet": "10.0.0.0/23",
            "udr": "udr01",
            "serviceEndpoints": [],
            "delegations": [
              {
                "name": "Microsoft.ContainerService/managedClusters",
                "properties": {
                  "serviceName": "Microsoft.ContainerService/managedClusters"
                }
              }
            ],
            "nsgRules": []              
          },
          {
            "subnet": "10.192.132.0/24",
            "udr": "udr01",
            "serviceEndpoints": [],
            "delegations": [
              {
                "name": "Microsoft.ContainerService/managedClusters",
                "properties": {
                  "serviceName": "Microsoft.ContainerService/managedClusters"
                }
              }
            ],
            "nsgRules": []
          },
          {
            // "snet-lb",
            "subnet": "10.0.0.0/26",
            "udr": "udr-lndz-to-conn",
            "serviceEndpoints": [],
            "nsgRules": []
          },
          {
            "subnet": "10.0.0.128/27",
            "udr": "udr01",
            "serviceEndpoints": [
              {
                "service": "Microsoft.ContainerRegistry"
              }
            ],
            "nsgRules": [
              {
                "name": "OUT",
                "properties": {
                  "protocol": "Tcp",
                  "sourcePortRange": "*",
                  "destinationPortRange": "443",
                  "sourceAddressPrefix": "10.0.0.128/27", 
                  "destinationAddressPrefix": "10.0.0.0/29",
                  "access": "Allow",
                  "priority": 1101,
                  "direction": "Outbound"
                }
              }
            ]
          },
          {
            "subnet": "10.0.0.64/26",
            "udr": "udr01",
            // "delegations": [],
            "serviceEndpoints": [
              {
              "service": "Microsoft.ContainerRegistry"
              }
            ],
            "nsgRules": [
              {
                "name": "OUT-443",
                "properties": {
                  "protocol": "Tcp",
                  "sourcePortRange": "*",
                  "destinationPortRange": "443",
                  "sourceAddressPrefix": "10.0.0.64/26",
                  "destinationAddressPrefix": "10.0.0.0/29",
                  "access": "Allow",
                  "priority": 1101,
                  "direction": "Outbound"
                }
              }
            ]
          },
          {
            "subnet": "10.192.1.160/27",
            "udr": "udr01",
            "serviceEndpoints": [
              {
                "service": "Microsoft.ContainerRegistry"
              }
            ],
            "nsgRules": [
              {
                "name": "OUT-443F",
                "properties": {
                  "protocol": "Tcp",
                  "sourcePortRange": "*",
                  "destinationPortRange": "443",
                  "sourceAddressPrefix": "10.192.1.160/27",
                  "destinationAddressPrefix": "10.192.0.0/29",
                  "access": "Allow",
                  "priority": 1101,
                  "direction": "Outbound"
                }
              }
            ]
          },
          {
            "subnet": "10.0.0.0/29",
            "udr": "udr01",
            // "delegations": [],
            // "serviceEndpoints": [
            //   {
            //     "service": "Microsoft.KeyVault"
            //   }
            // ],
            "nsgRules": []
          },
          {
            "subnet": "10.0.0.208/29",
            "udr": "udr01",
            // "delegations": [],
            // "serviceEndpoints": [
            //   {
            //     "service": "Microsoft.KeyVault"
            //   }
            // ],
            "nsgRules": []
          },
          {
            // "snet-pe-prometheus",
            "subnet": "10.0.0.216/29",
            "nsgRules": [],
            "udr": "udr01",
            // "delegations": [],
            "nsgRules": []
          },
          {
            // "snet-pe-aksapi",
            "subnet": "10.0.0.200/29",
            "nsgRules": [],
            "udr": "udr01",
            // "delegations": [],
            "nsgRules": []
          }
        ]
      }
    },
    "routeTables": {
      "value": [
        {
          "routeTablePrefix": "udr01",
          "routes": [
            {
              "name": "On-Prem",
              "properties": {
                "addressPrefix": "0.0.0.0/0",
                "nextHopType": "VirtualAppliance",
                "nextHopIpAddress": "10.0.2.4"
              }
            }
          ]
        }
      ]
    },
    "hubVnetName": {
      "value": "vnet001"
    },
    "hubVnetRG": {
      "value": "rg-005"
    },
    "hubSubscriptionId": {
      "value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    }
  }
}
Jefajers commented 5 months ago

Hi thanks for supplying the network.bicep template as well.

We are however not able to reproduce any issue during processing of neither the network.bicep or network.parameters.json.

As you mention that you are not able to get any local errors with the supplied templates in this issue.

Are you able to reproduce the error with network.bicep or network.parameters.json files attached to this issue using the module?. If you are then we could setup a call to better understand the circumstances (versions, settings, environment structure, etc) involved to attempt identifying what is happening.

Current module logic is expected to skip the .bicep file entirely during deletion processing and when processing the parameters.json file it would identify its not of a supported resource deletion type and then skip it as well.

With the final result: files are able to be deleted and the module has not performed any action towards Azure in regards to the potential file content.

mathwro commented 5 months ago

Hi, let me follow up with my customer if possible to set up a call for this and get back to you.

From what we are seeing it doesn't seem to skip the parameter file. It does indeed skip the bicep file, but it still attempts to serialize the JSON parameter file before determining any actions for it.

I'm unaware of the full context of what is going on, but i was hoping it might be possible to skip any steps involving the parameter file, if the bicep/arm template it's targeted for were deleted as well

Jefajers commented 5 months ago

Thanks let us know how it goes, and regarding "It does indeed skip the bicep file, but it still attempts to serialize the JSON parameter file before determining any actions for it.". The serialize step is indeed attempted and is required to parse the content inside the file before deciding on whether to act or not.

mathwro commented 4 months ago

We are happy to arrange for a meeting if possible @Jefajers

We have recently seen a similar issue again and would like to have this resolved somehow

Jefajers commented 4 months ago

Hi @mathwro, thanks for the supplied templates I have been able to reproduce the issue and identified a module logic issue which is only used during deletion to overcome the fact that we don't have the files in the actual branch.

The logic in this scenario creates an erroneous bicep file which triggers these conversion errors down the line that we see in this thread. In the up coming AzOps release 2.6.0 this logic as been updated and I am not able to reproduce the issue.

mathwro commented 3 months ago

Sounds great. I'll be looking forward to seeing the changes in effect