search

Azure / bicep

Bicep is a declarative language for describing and deploying Azure resources
MIT License
3.25k stars 754 forks source link

Strong typing for parameters and outputs #4158

Closed rynowak closed 1 year ago

rynowak commented 3 years ago

Is your feature request related to a problem? Please describe.

For parameters and outputs the set of possible types that code can declare is very limited (primitives, array, and object). When the type of a module parameter is object you don't get the same type safety guarantees that are normally possible in bicep. Having the ability to specify a more specific type than object would add type-safety for complex parameters and outputs in modules.

Here's a motivating example:

// A rule looks like ....
// {
//   name: 'rule'
//   properties: {
//     frontendPort: 5000
//     protocol: 'Tcp'
//.    ...
//   }
// }

param rule object

resource balancer 'Microsoft.Network/loadBalancers@2020-06-01' = {
  name: 'cool-load-balancer'
  properties: {
    loadBalancingRules: [
      rule
    ]
  }
}

Load balancers can be large and complex. If you want to parameterize a load balancer with modules, you have the choice to either forego type checking, or write a flat list of parameters that get combined into an object.

note: There is already a proposal for strongly-typed resources as parameters/outputs. This is about objects that are not resources.

Describe the solution you'd like

I'd like the ability to specify or infer a more specific type so that the parameters and outputs can be type-checked. I've discussed a few different options with the team, and want to gather more feedback. It's possible that more than one of these solutions are desirable given that they optimize for different scenarios.

Proposal: Refer to types by name

The simplest idea to understand is being able to specify the type using the named definitions that the compiler already knows. These are based on the OpenAPI descriptions used to generate Bicep's type definitions. In this example the type of rule is determined by looking up the provided type string against the resource type definitions.

param rule type 'Microsoft.Network/loadBalancers@2020-06-01#LoadBalancerRule'

resource balancer 'Microsoft.Network/loadBalancers@2020-06-01' = {
  name: 'cool-load-balancer'
  properties: {
    loadBalancingRules: [
      rule
    ]
  }
}

The part after the # is the type name, which is looked up in the context of the provided resource type. Failure to find the specified type would be a warning the failure to find a declared resource type.

To make this work we'd want to provide completion for the part after #. So the user can get completion for the whole string in 3 parts.

note: the syntax shown here is chosen to be similar/consistent with this proposal for type-specifiers for resources.

Proposal: Add a typeof specifier

This is similar to typeof type operator in TypeScript. In this example that type of the rule parameter is specify by type checking the provided expression.

param rule typeof balancer.properties.loadBalancingRules[0]

resource balancer 'Microsoft.Network/loadBalancers@2020-06-01' = {
  name: 'cool-load-balancer'
  properties: {
    loadBalancingRules: [
      rule
    ]
  }
}

The expression passed to typeof could be any expression (in theory) and could be combined with other type-specifier constructs as necessary. The expression is not evaluated, it is only type-checked.

note: the TypeScript typeof type operator is limited to property access and indexing. We probably need a way to specify that we want the element type of an array. Typescript uses typeof MyArray[number], which would seem foreign in bicep. In this case I filled that in as [*] but it needs more thought.

Proposal: Type inference via target-typing

This is similar to type inference in where it applies in TypeScript or some functional languages. Similar to (but more complex) target-typed new() in C#. In this example the type rule has its type inferred based on where it appears.

param rule auto

resource balancer 'Microsoft.Network/loadBalancers@2020-06-01' = {
  name: 'cool-load-balancer'
  properties: {
    loadBalancingRules: [
      rule
    ]
  }
}

note: This really optimizes for the use-case where a parameter is used once, or used in the same context. Lots of complex scenarios are possible when type inference gets involved.

KennethMLdk commented 3 years ago

I would definitely go for proposal "Refer to types by name". The param type and resource type is similar and is IMHO easier to use. Auto always makes me uneasy

miqm commented 3 years ago

898 #622 #3723 referencing some issues that touched or discussed this topic earlier.

miqm commented 3 years ago

Here are some of my thoughts on this. All options could coexist, I don’t see any issues with ability to exactly type type or use typeof or auto.

As for the problem with union types I’d limit it to objects and enums only and throw error if user assigns other type.

second concern is what about arrays? I feel that lots of use cases would be to use for loop on parameter to create resources with different values of same property.

I’d also leave object/array type and define what object type is being expected after or use decorator for it.

Also, with option 1 I feel we might have a discussion on how to keep in sync or simplify writing resource types, but eventually if we implement some type aliases they could be used here as well.

StijnDevogel commented 3 years ago

second concern is what about arrays? I feel that lots of use cases would be to use for loop on parameter to create resources with different values of same property.

I’d also leave object/array type and define what object type is being expected after or use decorator for it. I came also here to address this.

In our company we're use a web application gateway module that has parameter arrays for each specific property (httpListeners, gatewayIPConfigurations, ...)

So I'm not quite sure how you could discover what type is inside the array. But definitely consider this one in your discussions

rouke-broersma commented 3 years ago

In the Bicep community call there was an ask for example of how customers currently use object. Our main reason to use object is to provide a simple interface to the module user for which properties they should supply, where we provide sane defaults for everything not supplied. If we use any of the proposed solutions we would push the burden of understanding the properties to supply to the module onto the consumer. We can no longer clearly define the interface for our module which negates a large benefit of modules.

For our use cases custom defined types or type inference based on property usage instead of parameter usage would suit but the proposed solutions would not.

With inference based on property usage I mean that one object auto param or variable can infer the distinct property types (recursively) based on the location where the property is used.

Example:

// vnetInput
// {
//   name string
//   range string
//   subnets array auto
      // subnets
      // {
      //   name string
      //   range string
      // }
// } 
param vnetInput auto

resource vnet 'Microsoft.Network/virtualNetworks@2020-11-01' {
  name: name
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        range
      ]
    }
    subnets: [for subnet in subnets: {
      name: subnet.name
      properties: {
        addressPrefix: subnet.range
      }
    }]
  }
}
WithHolm commented 2 years ago

I'm not sure if it's way out of scope, but could it be an answer to implement some kind of json-schema-like language as a part of the input validation for objects? I'm thinking both that you can write it in a bicep way into your file, but also that you can defer to a external source for the actual validation (like $schema and $ref in json-schema) to 'save lines' in you definition.

Seeing as the actual output of bicep is json, and json already have pretty extensive tooling for definitions, then why not build upon that?

examples:

@schema({
  properties:{
    one:{
      type:string
      enum:[
        'value1'
        'value2'
      ]
    }
    two:{
      type:integer
      minimum:1
      maximum:2
    }
  }
  required:[
    'one'
  ]

})
param first object = {
  one:'value1'
  two:9
}

you could have it be able to reference exisiting external schemas:

//defer to schema.json 'inputname' property
@schema('./schema.json#/inputname')
param otherfileJsonSchema object

//defer to json schema file, on the internet
@schema('http://../schema.json')
param remotefileSchema object

//defer to schema.bicep 'inputname' parameter validation.
@schema('./schema.bicep#/inputname')
param otherfileBicepSchema object

or possibly for simplicity provide it with a example of input you want, and a schema+example would be generated:

@example({
    name:'someitem'
    range:'somerange'
    subnets:[
      {
        name:'somename'
        range:'somerange'
      }
    ]
})
param example object

as for resource input?

@resource('Microsoft.DocumentDB/databaseAccounts@2021-10-15')
param resource object

i don't know if this is something you have already thought about and dismissed in another discussion, but if its not, please concider it.

alex-frankel commented 2 years ago

@WithHolm - this ask seems similar/the same as what is described in #3723 and is related to this issue as well. Generally speaking, we need to continue to expand ways of getting value out of the type system. As you mention, the foundation is already there to do all sorts of type validation, but we need to provide a way of describing your own.

VanessaRussell commented 2 years ago

Adding that this should work for variables as well since they haven't been mentioned within this issue yet. I landed on this issue when I wasn't allowed to strongly type the SiteProperties that I'm reusing between function and function slot:

Desired - type checking:

var functionProperties SiteProperties = {
  serverFarmId: appServicePlan.id
  httpsOnly: true
  siteConfig: {
    webSocketsEnabled: false
    use32BitWorkerProcess: false
  }
}

resource functionApp 'Microsoft.Web/sites@2021-03-01' = {
  name: functionAppName
  location: targetResourceGroup
  kind: 'functionapp'
  identity: {
    type: 'SystemAssigned'
  }
  properties: functionProperties
}

resource functionAppSlot 'Microsoft.Web/sites/slots@2021-03-01' = {
  parent: functionApp
  name: functionAppSlotName
  location: targetResourceGroup
  kind: 'functionapp'
  identity: {
    type: 'SystemAssigned'
  }
  properties: functionProperties
}

Actual - no type checking 😢

var functionProperties = {
  serverFarmId: appServicePlan.id
  httpsOnly: true
  siteConfig: {
    webSocketsEnabled: false
    use32BitWorkerProcess: false
  }
}

resource functionApp 'Microsoft.Web/sites@2021-03-01' = {
  name: functionAppName
  location: targetResourceGroup
  kind: 'functionapp'
  identity: {
    type: 'SystemAssigned'
  }
  properties: functionProperties
}

resource functionAppSlot 'Microsoft.Web/sites/slots@2021-03-01' = {
  parent: functionApp
  name: functionAppSlotName
  location: targetResourceGroup
  kind: 'functionapp'
  identity: {
    type: 'SystemAssigned'
  }
  properties: functionProperties
}
ChristopherGLewis commented 2 years ago

Adding some notes from #6624

I like the simplicity of referring to the types using the symbolic name. I like typeof rather than 'type' or 'of type' and I like referencing the symbolic names of defined resources. This should allow types to propagate up from modules.

In addition to the coding experience, where ever possible the bicep build command should implement what it can in the ARM template - description, validations, enums and ranges are what I would expect.

param storageAcctName string
param azRegion string

param skuName string typeof objStorAcct.skuName.name = 'Standard_LRS'
param accessTier string typeof objStorAcct.properties.accessTier = 'Hot'
param kind string typeof objStorAcct.kind = 'StorageV2'

resource objStorAcct 'Microsoft.Storage/storageAccounts@2021-01-01' = {
   name: storageAcctName
   location: azRegion
   properties: {
      accessTier:  accessTier 
   }
   sku: {
      name: skuName 
   }
   kind: kind 
}

I'd like to throw in the possibility of using a type decorator, which might be easier to parse, although it makes the code larger

@type(objStorAcct.skuName.name)
param skuName string = 'Standard_LRS'
@type(objStorAcct.properties.accessTier )
param accessTier string = 'Hot'
@type(objStorAcct.kind )
param kind string = 'StorageV2'

Vars and objects are key to this - build-up of objects is getting pretty common in Bicep files, although its going to get complicated:

param storageAcctName string
param azRegion string

param skuName string typeof objStorAcct.skuName.name = 'Standard_LRS'
param accessTier string typeof objStorAcct.properties.accessTier = 'Hot'
param kind string typeof objStorAcct.kind = 'StorageV2'

//var with nested types.  The accessTier property would have to validate the accessTier param
var props typeof objStorAcct.properties = {
     accessTier:  accessTier    
}

resource objStorAcct 'Microsoft.Storage/storageAccounts@2021-01-01' = {
   name: storageAcctName
   location: azRegion
   properties: props 
   sku: {
      name: skuName 
   }
   kind: kind 
}

I think global types would be helpful: 

param azRegion string typeof azure.region

Finally, I think auto types might be hard to understand and @schema decorator kind of defeats the purpose.

moattarwork commented 2 years ago

I like the idea of auto but I think the string typeof xxx is far from being useful. What is wrong with extending the types instead. The annotation for allowed values can be replaced with the concept of typing and support of enumerations.

The only case would be the use of typeof which can be a build-in function to resolve the type from string or literal specification of the type:

typeof(objStorAcct.properties)
or 
typeof('Microsoft.Storage/storageAccounts@2021-01-01#xxx')
akata72 commented 2 years ago

Hi, is there any progress or decisions/design regarding this topic. It would be a gamechanger to have this implemented.

alex-frankel commented 2 years ago

We are definitely going to implement something to enable strong typing -- both completely custom and with the ability to reference subschemas from resource types (or possibly elsewhere) through a typeof kind of functionality.

Don't have an ETA as we are still not closed on design, but we recognize that this is a serious limitation in the consumption of modules -- particularly from an external registry!

itpropro commented 2 years ago

Just adding some suggestions from #7499

Describe the solution you'd like I can see multiple ways to do this.

  1. Use the same way as the resource, probably a JSON or Bicep schema file, although even for resources the description or types are often different between ARM/Bicep intellisense and the MS Docs.
  2. Provide some way to nest parameters with decorators for objects. Here with a new key parameter type:
@allowed([
  @allowed([
    2
    4
    8
    16
    32
    64
  ])
  @description('Optional. The scale up/out capacity, representing server\'s compute units.')
  key capacity int = 4

  @allowed([
    'Gen5'
  ])
  @description('Optional. The family of hardware.')
  key skuFamily string = 'Gen5'
])
@description('Optional. The SKU (pricing tier) of the server.')
param sku object = {}
  1. Provide the ability to construct complex parameter objects instead of directly nesting them:
@allowed([
  2
  4
  8
  16
  32
  64
])
@description('Optional. The scale up/out capacity, representing server\'s compute units.')
key/subparam capacity int = 4

@allowed([
  'Gen5'
])
@description('Optional. The family of hardware.')
key/subparam skuFamily string = 'Gen5'

@allowed([
  capacity
  skuFamily
])
@description('Optional. The SKU (pricing tier) of the server.')
param sku object = {}
stan-sz commented 2 years ago

Suggestion: for object params, the schema for it could be a json schema kept in a separate file. Then with the machinery loadJsonContent it could be embedded and used at runtime to validate the input.

Jaykul commented 2 years ago

On top of the constraining inputs, let's not forget intellisense on properties of outputs.

In one example we have a "names" module that helps us follow best practices for resource naming and in particular, a standard set of abbreviations for resources. It basically takes parameters like the base name or "workload", the environment, and region (some of which can be calculated from resource group or subscription names) ... image

The simple version of this was a module with an output for each resource type, so you could use: names.outputs.servicebus and you'd get completion for all the outputs -- but there's a limit on the number of outputs! Now we need to return an object with dozens of properties, and we need intellisense (and linting) to work on the properties of that object so that people can pick the right one, rather than needing to type it correctly from memory (even "serviceBus" vs "servicebus" trips people up).

Obviously this "type" information doesn't necessarily need to make it into json at all -- the only thing that's really necessary is auto completion in VS Code.

ChristopherGLewis commented 2 years ago

Adding DataCollectionRules as a resource that typically uses a JSON value. The MS recommended template looks like this:

param dcrName string  //this should start with MSVMI-
param azRegion string = resourceGroup().location

@allowed([ 'Linux', 'Windows' ])
param kind string

param dataSources object = {}
param destinations object = {}
param dataFlows array = []

resource dataCollectionRule 'Microsoft.Insights/dataCollectionRules@2021-09-01-preview' = {
  name: dcrName
  location: azRegion
  kind: kind
  properties: {
    dataSources: dataSources
    destinations: destinations
    dataFlows: dataFlows
  }
}

with the dataSources, destinations and dataFlows passed in using JSON objects:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "ruleName": {
            "value": "dcr-windowsvmbasic"
        },
        "location": {
            "value": "northcentralus"
        },
        "dataSources": {
            "value": {
                "windowsEventLogs": [
                    {
                        "streams": [
                            "Microsoft-Event"
                        ],
                        "scheduledTransferPeriod": "PT5M",
                        "xPathQueries": [
                            "Application!*[System[(Level=1 or Level=2)]]",
                            "Security!*[System[(band(Keywords,4503599627370496))]]",
                            "System!*[System[(Level=1 or Level=2)]]"
                        ],
                        "name": "eventLogsDataSource"
                    }
                ],
                "performanceCounters": [
                    {
                        "counterSpecifiers": [
                            "\\Processor Information(_Total)\\% Processor Time",
                            "\\Processor Information(_Total)\\% Privileged Time",
                            "\\Processor Information(_Total)\\% User Time",
                            "\\Processor Information(_Total)\\Processor Frequency",
                            "\\System\\Processes",
                            "\\Process(_Total)\\Thread Count",
                            "\\Process(_Total)\\Handle Count",
                            "\\System\\System Up Time",
                            "\\System\\Context Switches/sec",
                            "\\System\\Processor Queue Length",
                            "\\Memory\\% Committed Bytes In Use",
                            "\\Memory\\Available Bytes",
                            "\\Memory\\Committed Bytes",
                            "\\Memory\\Cache Bytes",
                            "\\Memory\\Pool Paged Bytes",
                            "\\Memory\\Pool Nonpaged Bytes",
                            "\\Memory\\Pages/sec",
                            "\\Memory\\Page Faults/sec",
                            "\\Process(_Total)\\Working Set",
                            "\\Process(_Total)\\Working Set - Private",
                            "\\LogicalDisk(_Total)\\% Disk Time",
                            "\\LogicalDisk(_Total)\\% Disk Read Time",
                            "\\LogicalDisk(_Total)\\% Disk Write Time",
                            "\\LogicalDisk(_Total)\\% Idle Time",
                            "\\LogicalDisk(_Total)\\Disk Bytes/sec",
                            "\\LogicalDisk(_Total)\\Disk Read Bytes/sec",
                            "\\LogicalDisk(_Total)\\Disk Write Bytes/sec",
                            "\\LogicalDisk(_Total)\\Disk Transfers/sec",
                            "\\LogicalDisk(_Total)\\Disk Reads/sec",
                            "\\LogicalDisk(_Total)\\Disk Writes/sec",
                            "\\LogicalDisk(_Total)\\Avg. Disk sec/Transfer",
                            "\\LogicalDisk(_Total)\\Avg. Disk sec/Read",
                            "\\LogicalDisk(_Total)\\Avg. Disk sec/Write",
                            "\\LogicalDisk(_Total)\\Avg. Disk Queue Length",
                            "\\LogicalDisk(_Total)\\Avg. Disk Read Queue Length",
                            "\\LogicalDisk(_Total)\\Avg. Disk Write Queue Length",
                            "\\LogicalDisk(_Total)\\% Free Space",
                            "\\LogicalDisk(_Total)\\Free Megabytes",
                            "\\Network Interface(*)\\Bytes Total/sec",
                            "\\Network Interface(*)\\Bytes Sent/sec",
                            "\\Network Interface(*)\\Bytes Received/sec",
                            "\\Network Interface(*)\\Packets/sec",
                            "\\Network Interface(*)\\Packets Sent/sec",
                            "\\Network Interface(*)\\Packets Received/sec",
                            "\\Network Interface(*)\\Packets Outbound Errors",
                            "\\Network Interface(*)\\Packets Received Errors"
                        ],
                        "samplingFrequencyInSeconds": 10,
                        "streams": [
                            "Microsoft-InsightsMetrics"
                        ],
                        "scheduledTransferPeriod": "PT1M",
                        "name": "perfCounterDataSource10"
                    }
                ]
            }
        },
        "dataFlows": {
            "value": [
                {
                    "streams": [
                        "Microsoft-InsightsMetrics"
                    ],
                    "destinations": [
                        "azureMonitorMetrics-default"
                    ]
                },
                {
                    "streams": [
                        "Microsoft-Event"
                    ],
                    "destinations": [
                        "la--1715273091"
                    ]
                }
            ]
        },
        "destinations": {
            "value": {
                "azureMonitorMetrics": {
                    "name": "azureMonitorMetrics-default"
                },
                "logAnalytics": [
                    {
                        "name": "la--1715273091",
                        "workspaceId": "b6fbfb28-25f0-437d-bbc6-e0868ce3c2e3",
                        "workspaceResourceId": "/subscriptions/98c0ce08-01d3-4027-a248-6d62fff92884/resourceGroups/rg-p-zusnc1-azuremonitor/providers/Microsoft.OperationalInsights/workspaces/log-p-zusnc1-azuremonitor"
                    }
                ]
            }
        },
        "tagsArray": {
            "value": {}
        },
        "platformType": {
            "value": "Windows"
        }
    }
}
WhitWaldo commented 2 years ago

My issue on this was linked here and closed, but I'd like to resurface my own take on how this should be approached. I would like to see strong-typing introduced more like how interfaces are used in TypeScript - namely, they describe the shape of an object and members can have attributes attached, but they otherwise have no impact on runtime; they are relevant only during development via Intellisense and build-time via the compiler, but then the compiler only maintains the status quo at runtime: validations on weakly-typed objects and arrays.

I would ideally like to see the typing of objects not strictly tied to any given resource because 1) the resources are versioned and may change, potentially necessitating significant changes for a version bump and 2) not all objects passed throughout are strictly bound to a single resource. Especially when coupled with lambda functions to map out specific properties, I'd like to have greater flexibility into what's being passed into a module so I might bind the one object to more than just a parent module.

I just wrote up a fuller example in my other issue, but I'd like to be able to do something like the following. First, describe an interface for an authorization rule in an Event Hub using the validation attributes to limit downstream usage:

interface authorizationRule = {
  name: string
  @minlength(1)
  @minlength(3)
  @allowed([
    'Listen'
    'Send'
    'Manage'
  ])
  rights: array
}

Note that nothing about this references an Event Hub - I know it's used with the Event Hub only because it's embodied in that module and accepted as an input parameter (below), but it's thus not dependent on any resource versions nor do I lose out on any of the inline validation already available in Bicep.

In my module, I'd simply specify this as the type of the array I expect and could just as easily decorate the array parameter with a @minlength(1) as desired.

@param EventHubName string
@param AuthorizationRules authorizationRule[] = [] //Indicates that members of the array should be typed with the interface - alternatively specified as array<authorizationRule>

//...

module EventHubAuthorizationRule './authorization-rule.bicep' = [for (authorizationRules, i) in AuthorizationRules: {
  name: 'eh-${EventHubName}-AuthRule-${authorizationRule.name}'
  params: {
    Name: authorizationRule.name
    Rights: AuthorizationRule.rights
  }
}]

And because, like TypeScript, the interface wouldn't persist to runtime, I'd expect the Bicep compiler to output exactly the weakly typed output it does today:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.10.61.36676",
      "templateHash": "1281152232305116607"
    }
  },
  "parameters": {
    "AuthorizationRules": {
      "type": "array"
    },
    "EventHubName": {
      "type": "string",
      "metadata": {
        "description": "The name of the Event Hub resource"
      }
    },
//...

To reference the interface between files, I'd propose it simply be visible like any param out of a Bicep module. From 

//interfaces.bicep

interface authorizationRule = {
  name: string
  @minlength(1)
  @minlength(3)
  @allowed([
    'Listen'
    'Send'
    'Manage'
  ])
  rights: array
}
//main.bicep
var authRule as authorizationRule from './interfaces.bicep' = {
  name: 'Management'
  rights: [
    'Manage'
  ]
}

module EventHubAuthRules './eventhub-authrules.bicep' = {
  name: 'eh-authrules'
  params: {
    EventHubName: eventHubName
    AuthorizationRules: [
      authRule
    ]
  }
}

But my biggest issue with #4158 is that it limits complex values only to a single resource type. My proposal is broader - if you want to limit to a single resource, go for it, but if you want to describe a complex type that's broad enough to describe every resource of a whole resource group, go for it (especially if made up of many smaller, more easily maintained interfaces).

rouke-broersma commented 2 years ago

I'm @WhitWaldo, I want to be able to specify my own structure for the objects I pass as params, independent of an actual resource type, while enforcing the types at least compile time.

I also want to pass whole resources by ref, but that's not related to its typings but rather for this like rbac Authorization and such.

dazinator commented 1 year ago

Just chiming in to add my expectations which is most likely similar to others.

When declaring parameters I want to declare strongly typed objects or arrays so that in the consuming bicep file, intellisense can actually be useful and suggest / generate the value that needs to be passed if need be.

For example consider typing the following into a bicep file, when consuming the module that takes an object and object[] parameter - see comments:

module thing 'thing.bicep' = {
  name: 'thing'
  params: {
    someTypedObject: {
       // This can be generated for me based on the objects type. It knows which properties are `required` here
       someMandatoryFlag:  // fill this in
    }
    someTypedObjectArray: [
     {
        // This array can be generated for me with 1 object - based on the `min length` decorator for this object array param of 1, as well as the objec created thanks to the object type and required properties etc. 
       someMandatoryFlag:  // fill this in
    }
   ]     

  }
}

Update: I'd also expect any solution to support custom type definitions, not just pre-existing type definitions from ARM. For exampe, my modules abstract away one or many ARM resources, its natural for my modules to want to expose custom parameter types as they are operating at a different level of granularity than the individual ARM resources - exposing built-in types is useful, but custom types is going to be more powerful for abstracting things correctly.

alex-frankel commented 1 year ago

On today's community call, we will be demoing a preview of custom types support that technically has already shipped (though we haven't published anything about it yet). We will also be sharing a gist with some basic documentation to get you started with the feature.

@adrianhall already wrote up a nice post about it: https://adrianhall.github.io/azure/2022/11/28/bicep-type-checking/

cc @jeskew as FYI

rouke-broersma commented 1 year ago

On today's community call, we will be demoing a preview of custom types support that technically has already shipped (though we haven't published anything about it yet). We will also be sharing a gist with some basic documentation to get you started with the feature.

@adrianhall already wrote up a nice post about it: https://adrianhall.github.io/azure/2022/11/28/bicep-type-checking/

cc @jeskew as FYI

This looks excellent, I am super excited!

miqm commented 1 year ago

@jeskew - one more thing to consider while improving strong typing - would be nice to have ability to define a dictionary type, where key is string and the value is a custom type. then we can use items to iterate over the object and use key for a name symbol in constructed output and value to be used by resources.

jeskew commented 1 year ago

@miqm Dictionary types are planned for an upcoming release (likely 0.14). There’s a syntax proposal in https://github.com/Azure/bicep/issues/9228

dazinator commented 1 year ago

I am trying to define a type in one module bicep file, that references a type declared in another module bicep file. VS isn't recognising it:

/modules/foo.bicep

type Foo = {      
  name: string
}

/recipes/bar.bicep

type Bar = {      
  first: Foo
  another: string
}

In the above example the Bar type does not compile as the type Foo cannot be found. Do i need to import this type somehow?

jeskew commented 1 year ago

@dazinator Types have to be defined in the template where they're used, but work on type sharing is tracked in https://github.com/Azure/bicep/issues/9311

dazinator commented 1 year ago

@jeskew

After enabling custom types and getting everything to compile, when actually deploying to azure, I get these new errors:

{'code': 'MultipleErrorsOccurred', 'message': 'Multiple error occurred: BadRequest,BadRequest,BadRequest. Please see details.'}

Inner Errors:
{'code': 'InvalidTemplate', 'target': '/subscriptions/******/resourceGroups/my-test/providers/Microsoft.Resources/deployments/vnet', 'message': "Deployment template validation failed: 'The resource 'Microsoft.Network/networkSecurityGroups/nsg-uniun-dev-we-01' at line '1' and column '5055' is defined multiple times in a template. Please see https://aka.ms/arm-template/#resources for usage details.'.", 'additionalInfo': [{'type': 'TemplateViolation', 'info': {'lineNumber': 1, 'linePosition': 5055, 'path': 'properties.template.resources.newNsg'}}]}

Inner Errors:
{'code': 'InvalidTemplate', 'target': '/subscriptions/******/resourceGroups/my-test/providers/Microsoft.Resources/deployments/appGatewayPublicIp', 'message': "Deployment template validation failed: 'The resource 'Microsoft.Network/publicIPAddresses/pip-uniun-agw-dev-we-01' at line '1' and column '1542' is defined multiple times in a template. Please see https://aka.ms/arm-template/#resources for usage details.'.", 'additionalInfo': [{'type': 'TemplateViolation', 'info': {'lineNumber': 1, 'linePosition': 1542, 'path': 'properties.template.resources.existingPublicIp'}}]}

Inner Errors:
{'code': 'InvalidTemplate', 'target': '/subscriptions/******/resourceGroups/my-test/providers/Microsoft.Resources/deployments/loadBalancerPublicIp', 'message': "Deployment template validation failed: 'The resource 'Microsoft.Network/publicIPAddresses/pip-uniun-lbe-dev-we-01' at line '1' and column '1546' is defined multiple times in a template. Please see https://aka.ms/arm-template/#resources for usage details.'.", 'additionalInfo': [{'type': 'TemplateViolation', 'info': {'lineNumber': 1, 'linePosition': 1546, 'path': 'properties.template.resources.existingPublicIp'}}]}

Undoing my refactoring of custom types resolves this issue again. I can't see any duplicate definition of the resources mentioned..

Marchelune commented 1 year ago

Is there an update on this? The user-defined types are very nice for params, but sometimes I'd be happy to leverage the existing resource type definitions. In my current scenario, I want to define an application gateway and I want to define all its complex sub-properties (frontend, backend pools, http probes etc.) in different modules - everything in the same module extremely difficult to read.

It seems sub-optimal to redefine custom types for those properties (even if/when we're able to import/export them): for example, the bicep compiler knows that frontendIPConfigurations is of type ApplicationGatewayFrontendIPConfiguration[], so the information is already there. If the type is tied to the API version of the resource, that would also help to see/raise alerts on API version model changes.

param gatewayName string

@minLength(1)
param frontendHttpListeners array

@minLength(1)
param frontendIPConfigurations array
// ideally
// param frontendIPConfigurations ApplicationGatewayFrontendIPConfiguration[]

resource appGateway 'Microsoft.Network/applicationGateways@2022-09-01' = {
    name: gatewayName
    location: resourceGroup().location
    properties: {  

      frontendIPConfigurations: frontendIPConfigurations      
      httpListeners: frontendHttpListeners
      // ... etc
  }
}
WhitWaldo commented 1 year ago

@Marchelune That might be more of a separate issue than one strictly having to do with custom types. Correct me if I'm wrong, but I seem to recall that Application Gateway is much like a Virtual Network in that all the child resources must be present when the parent is created (necessitating that single large module) since an update to the parent from multiple files (modules) isn't feasible. While importing/exporting custom types to shuffle data around the requisite modules would be helpful, I'm pretty sure that effort would be blocked more by the "cannot update some types, must do full deployments at once" issue.

WithHolm commented 1 year ago

@WhitWaldo you are pretty bang on, AGW is not put together in a modular way, so if you need to update one part of it, you need to do a full "replace" (ie PUT complete object) to do any updates.
However it doesn't take away from his request to be able to use the types inside agw in order to determine parameters. this could even be achieved via a type definition where you reference the correct resource even:

//nothing here is actual, just a example of how you could reference a 'built in' type
type AgwFrontend ref('Microsoft.Network/applicationGateways@2022-11-01',ApplicationGatewayFrontendIPConfiguration)
param frontend_clean AgwFrontend[]

param frontend_dirty {
        id: 'string'
        name: 'string'
        properties: {
          privateIPAddress: 'string'
          privateIPAllocationMethod: 'string'
          privateLinkConfiguration: {
            id: 'string'
          }
          publicIPAddress: {
            id: 'string'
          }
          subnet: {
            id: 'string'
          }
        }
      }[]
Marchelune commented 1 year ago

@WhitWaldo I agree that it is not really custom types, given that those type already exist. But it fits the "strong typing parameters and output" IMO because I "only" need strongly typed objects rather than "any" objects. In fact I am already using that solution:

//======
// application-gateway.bicep

param gatewayName string

@minLength(1)
param frontendHttpListeners array

@minLength(1)
param frontendIPConfigurations array

resource appGateway 'Microsoft.Network/applicationGateways@2022-09-01' = {
    name: gatewayName
    location: resourceGroup().location
    properties: {  

      frontendIPConfigurations: frontendIPConfigurations      
      httpListeners: frontendHttpListeners
      // ... etc
  }
}

//======
// application-gateway.main.bicep

module routingRules './helpers/gateway-frontend-routing-rules-config.bicep' = {
  name: '${gatewayName}-routing-rules'
  params: {
    foo: 'whatever params'
    // etc...
  }
}

module gatewayModule './application-gateway.bicep' = {
  name: 'app-gateway-${deploymentSuffix}'
  params: {
    gatewayName: gatewayName    
    requestRoutingRules: routingRules.outputs.appGatewayRoutingRules
    rewriteRuleSets: routingRules.outputs.appGatewayRewriteRuleSets
    // etc....
  }
}

//======
// helpers/gateway-frontend-routing-rules-config.bicep

param foo string

output appGatewayRewriteRuleSets array = []
output appGatewayRoutingRules array = [for i in range(1,10): {
  id: 'rule-${foo}-${i}'
  // etc, you can have whatever complexity here
}]

The "helper" module has no actual resources, it's really just a function that returns arrays/objects. You are right that it is deployed all at once - but that would still be the case here, whilst allowing cleaner code.

At least on my side, with 6/7 listener/rule/backend combinations that must be parametrised per environment, the app gateway configuration via bicep becomes a barely maintainable 600 lines file. Extracting everything to a helpers makes it much clearer, but without strong types in those helpers, I am at risk of typos/missing properties (a risk that already exists when using intermediary var in single file, but made prominent when the config is over several files).

WhitWaldo commented 1 year ago

When I spin up an App Gateway today, I'm mostly using custom types as a stand-in so I can get validation that I'm at least typing the names of the parameter values correctly, verify on the caller side that I'm passing the right stuff in where I intend and get strong typing for use in lambda and other functions.

For example, we lack an import/export at the moment, so I've got these types at the start of my AppGw module:

type ipAddressProperties = {
  @description('The zones in which the IP address should be deployed')
  ipZones: string[]
  @description('Whether or not the IP address can be static (true) or not (false)')
  isStaticIp: bool
  @description('Whether the IP address is a standard (true) or basic (false) SKU')
  isStandardIpSku: bool
  @description('Whether or not the IP address is public (true) or not (false)')
  isPublicIp: bool
}

@description('Used to identify the details of a specific certificate')
type certificateMapping = {
  @description('The subject value of the certificate')
  subject: string
  @description('The name of the secret in the Key Vault')
  secretName: string
  @description('The thumbprint of the certificate')
  thumbprint: string
  @description('The identifier of the Key Vault instance')
  keyVault: keyVaultIdentifier
}

@description('Used to identify a Key Vault and where it\'s deployed to')
type keyVaultIdentifier = {
  @description('The name of the Key Vault')
  name: string
  @description('The ID of the subscription the Key Vault is associated with')
  subscriptionId: string
  @description('The name of the resource group the Key Vault is associated with')
  resourceGroupName: string
}

type uaiDetail = {
  @description('The name of the user-assigned identity that will execute the deployment script')
  name: string
  @description('The ID of the subscription that the user-assigned identity is associated with that will execute the deployment script')
  subscriptionId: string
  @description('The name of the resource group that the user-assigned identity is associated with that will execute the deployment script')
  resourceGroupName: string
}

And those are then followed by a wall of parameters that input those and still other parameters (including, note the use of the keyVaultIdentifier type in the certificateMapping so I can identify the key vault I'm pulling each from in case they differ:

@description('The IP address aspects to assign to the Application Gateway')
param IpAddressProperties ipAddressProperties

@description('The various subjects for which certificates should be secured for both Front Door, Application Gateway and for deployment to the SF cluster')
param CertificateSubjects certificateMapping[]

@description('The details of the user-assigned identity that will execute the deployment script')
param UaiDetails uaiDetail

And later on, continued use of the values out of these custom types:

//Maintains a reference to the certificate in the Key Vault for this subject
resource Secret 'Microsoft.KeyVault/vaults/secrets@2022-07-01' existing = [for (c, i) in CertificateSubjects: {
  name: '${c.keyVault.name}/${c.secretName}'
  scope: resourceGroup(c.keyVault.subscriptionId, c.keyVault.resourceGroupName)
}]

var endpointSecretName = [for (endpoint, i) in AllEndpoints: {
  secretName: first(map(filter(CertificateSubjects, cert => startsWith(cert.secretName, '*.') ? endsWith(endpoint.targetDomain, replace(cert.secretName, '*.', '')) : endpoint == cert.subject), c => c.secretName))
}]

Then, later on in the block of AppGw resource logic, I can trivially reference:

//...
    sslCertificates: [for (c, i) in CertificateSubjects: {
      name: c.secretName
      properties: {
         keyVaultSecretId: Secret[i].id
      }
    }]

...among all the other types.

I agree - it'd be great if I could use a parent/child syntax of some sort to at least artificially move up and down the resource dependency tree (asked about this in a similar vein for load balancer a while ago in #724 ) with some way of having Bicep work out how to chain it all together for a single PUT request at deployment, but simplify the creation of these elaborate resources, but again.. I think that's a separate issue to this one.

WhitWaldo commented 1 year ago

@Marchelune It occurs to me based on re-reading your last comment that you might not be aware of the preview custom types feature that I use above: https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/user-defined-data-types

There's no import/export support yet (back and forth about it in #9311 ) but based on the last community call, that'll likely be showing up soon.

Marchelune commented 1 year ago

@WhitWaldo thanks for your extensive feedback! I think we are very much aligned on that, in fact I did try user-defined types, but faced 2 issues:

That said, I do agree that user-defined types are a good intermediary solution, although I'd still personally prefer, as suggested in this proposal from @rynowak, something like

param frontendIPConfigurations type 'Microsoft.Network/applicationGateways@2022-09-01#ApplicationGatewayFrontendIPConfiguration'[]
alex-frankel commented 1 year ago

@marchelune -- I believe we will get to what you are looking for with the planned typeof operator discussed in #9229