In azure.azure_firewall table seeing few keys are missing for json object when passing it directly using transform.FromField

[kaidaguerre]

Slack conversation thread https://turbothq.slack.com/archives/C01AC8JQNHH/p1610017247038000

        {
            Name:        "ip_configurations",
            Description: "A collection of IP configuration of the Azure Firewall resource",
            Type:        proto.ColumnType_JSON,
            Transform:   transform.FromField("AzureFirewall.AzureFirewallPropertiesFormat.IPConfigurations"),
        },

Test column using transform

            {
                Name:        "test_column",
                Description: "A list of AKAs (also-known-as) that uniquely identify this resource",
                Type:        proto.ColumnType_JSON,
                Transform:   transform.From(testFunction),
            },

Here in field test_column we are getting all the properties but not in the field ip_configurations missing keys for ip_configurations

    "PrivateIPAddress": "10.0.0.4",
    "ProvisioningState": "Succeeded",

steampipe query --output json "select name, test_column, ip_configurations from azure.azure_firewall"

[
 {
  "ip_configurations": [
   {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/azureFirewalls/test_firewall/azureFirewallIpConfigurations/test_pi",
    "name": "test_pi",
    "properties": {
     "publicIPAddress": {
      "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/publicIPAddresses/test_pi"
     },
     "subnet": {
      "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/virtualNetworks/testvn/subnets/AzureFirewallSubnet"
     }
    }
   }
  ],
  "name": "test_firewall",
  "test_column": [
   {
    "PrivateIPAddress": "10.0.0.4",
    "ProvisioningState": "Succeeded",
    "PublicIPAddress": {
     "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/publicIPAddresses/test_pi"
    },
    "Subnet": {
     "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/virtualNetworks/testvn/subnets/AzureFirewallSubnet"
    }
   }
  ]
 },
]

Test transform function

func testFunction(ctx context.Context, d *transform.TransformData) (interface{}, error) {
    data := d.HydrateItem.(firewallInfo)

    var output []map[string]interface{}
    for _, firewall := range *data.AzureFirewall.AzureFirewallPropertiesFormat.IPConfigurations {
        output = append(output, map[string]interface{}{
            "PrivateIPAddress":  firewall.AzureFirewallIPConfigurationPropertiesFormat.PrivateIPAddress,
            "Subnet":            firewall.AzureFirewallIPConfigurationPropertiesFormat.Subnet,
            "PublicIPAddress":   firewall.AzureFirewallIPConfigurationPropertiesFormat.PublicIPAddress,
            "ProvisioningState": firewall.AzureFirewallIPConfigurationPropertiesFormat.ProvisioningState,
        })
    }
    return output, nil
}

Struct for IPConfigurations

type AzureFirewallIPConfigurationPropertiesFormat struct {
    // PrivateIPAddress - READ-ONLY; The Firewall Internal Load Balancer IP to be used as the next hop in User Defined Routes.
    PrivateIPAddress *string `json:"privateIPAddress,omitempty"`
    // Subnet - Reference to the subnet resource. This resource must be named 'AzureFirewallSubnet' or 'AzureFirewallManagementSubnet'.
    Subnet *SubResource `json:"subnet,omitempty"`
    // PublicIPAddress - Reference to the PublicIP resource. This field is a mandatory input if subnet is not null.
    PublicIPAddress *SubResource `json:"publicIPAddress,omitempty"`
    // ProvisioningState - READ-ONLY; The provisioning state of the Azure firewall IP configuration resource. Possible values include: 'Succeeded', 'Updating', 'Deleting', 'Failed'
    ProvisioningState ProvisioningState `json:"provisioningState,omitempty"`
}

Branch name azure-firewall-test on https://github.com/turbotio/steampipe-plugin-azure repo

[tyagiparth]

This seems to be a bug with Azure SDK where we have a custom marshaler for the AzureFirewallIPConfigurationPropertiesFormat struct which does not marshal the PrivateIPAddress and ProvisioningState properties. The function is:

// MarshalJSON is the custom marshaler for AzureFirewallIPConfigurationPropertiesFormat.
func (aficpf AzureFirewallIPConfigurationPropertiesFormat) MarshalJSON() ([]byte, error) {
    objectMap := make(map[string]interface{})
    if aficpf.Subnet != nil {
        objectMap["subnet"] = aficpf.Subnet
    }
    if aficpf.PublicIPAddress != nil {
        objectMap["publicIPAddress"] = aficpf.PublicIPAddress
    }
    return json.Marshal(objectMap)
}

The workaround for this would be something we are already doing with a transform function.