jkotalik commented 2 years ago

Adding `providers`

This is a proposal to add providers to Microsoft.Resources/deployments as a way to specify deployment-time configuration for imports statements.

Problem today

Bicep extensibility allows for deploying resources outside of the ARM control plane. For example, one can deploy a Kubernetes resource today to a AKS cluster purely by defining the Kubernetes resources in bicep: https://github.com/Azure/bicep/tree/main/docs/examples_extensibility/aks.

resource backDeploy 'apps/Deployment@v1' = {
  metadata: {
    name: backName
  }
  ...

Extensible resources are imported into a bicep file via the import keyword. This allows configuring options like namespace and kubeConfig on the import statement to be used when deploying Kubernetes resources.

@secure()
param kubeConfig string

import kubernetes as k8s {
    kubeConfig: kubeConfig
    namespace: 'default'
}

Though passing a parameter will work for this scenario, let's say we start using module(s) to deploy resources. This will add a lot of complexity to the bicep file, as we need to pass this "ambient" context around everywhere we want to import kubernetes.

@secure()
param kubeConfig string

import kubernetes as k8s {
    kubeConfig: kubeConfig
    namespace: 'default'
}

module mymodule 'module.bicep' {
  name: 'mymodule'
  params: {
    kubeConfig: kubeConfig
  }
}

// module.bicep
import kubernetes as k8s {
    kubeConfig: kubeConfig
    ...

This problem will expand rapidly as more parameters are added to an import statement.

For comparison, let's look at the az provider itself. The call to import az into the bicep file is:

import az from az

Where calls to get information like the resourceGroup or subscriptionId are made:

resource todo 'SOME_RESOURCE' = {
  name: 'todo'
  tags: [
    az.resourceGroup().id
  ]
}

So logical questions would be:

How does the az provider know what resourceGroup is and what to use?
Why does the az provider have this capability and not the kubernetes provider?

For an ARM deployment, the resourceGroup is specified at deployment-time and is ambiently available when resources are being deployed as part of ARM. That is, when ARM resources are being deployed, if the scope is correct, the value for resourceGroup is always available.

So for the Kubernetes extensibility provider, and eventually other providers, what if we wanted to ambiently have the kubeConfig available when deploying Kubernetes resources? Today, we only can pass the kubeConfig as a parameter to the import statement, unlike the az provider where you can specify the resourceGroup and subscriptionId as ambient configuration on a deployment and set as part of az deployment command.

Proposal

To be able to specify configuration at deployment time, I'd like to propose a new section to the Microsoft.Resources/deployments schema.

{
  "type": "Microsoft.Resources/deployments",
  "apiVersion": "2021-04-01",
  "name": "string",
  "location": "string",
  "tags": {
    "tagName1": "tagValue1",
    "tagName2": "tagValue2"
  },
  "scope": "string",
  "properties": {
    "debugSetting": {
      "detailLevel": "string"
    },
    "expressionEvaluationOptions": {
      "scope": "string"
    },
    "mode": "string",
    "onErrorDeployment": {
      "deploymentName": "string",
      "type": "string"
    },
    "parameters": {},
    "parametersLink": {
      "contentVersion": "string",
      "uri": "string"
    },
    "providers": { <-  THIS IS THE CHANGE *****
      "providerName": {
         "type": “string”,
         // Additional configuration here.
      }
    },
    "template": {},
    "templateLink": {
      "contentVersion": "string",
      "id": "string",
      "queryString": "string",
      "relativePath": "string",
      "uri": "string"
    }
  },
  "resourceGroup": "string",
  "subscriptionId": "string"
}

Providers are a section to specify configuration at deployment time. This would start being used by all import statements in bicep. The additional required properties on the az import is out of scope of this initial proposal for providers, but long term having the ability to specify the resource group and subscription on the import via providers would be great.

Additionally, the bicep definition for a module would also need to update to support providers as well:

module mymodule 'module.bicep' = {
  name: 'mymodule'
  providers: {
      ...
  }
}

Providers would require specifying the type of provider. The rest of it would be a collection of properties which are specific to the provider. For example, the az provider would allow specifying the scope of a deployment (which could either be the subscription, resourceGroup, etc).

This information will be ambiently available for each resource being deployed inside of the deployment engine. For example, each time a Kubernetes resource is deployed, the kubeConfig will be available and sent as part of the call to the Extensibility Provider API. This part would require changes in the ARM Deployment Engine to make sure provider configuration is propagated correctly to Extensibility Providers.

What this looks like with `kubernetes`

Here is what the kubernetes provider would look like where we want to specify the kubeConfig at deployment time:

{
    "providers": {
        "k8s": {
            "kubeConfig": "some-string",
            "type": "Kubernetes"
        }
    }
}

module mymodule 'module.bicep' = {
  name: 'mymodule'
  providers: {
    k8s: {
      scope: 'some-string',
      type: 'Kubernetes'
    }
  }
}

Usage in bicep:

@secure()

import kubernetes as k8s

resource backDeploy 'apps/Deployment@v1' = {
  ...
}

ARM json representation:

{
  ...
  "parameters": {
    ...
  },

  "providers": {
    "k8s": {
      "kubeConfig": "some-string",
      "namespace": "default",
      "type": "Kubernetes"    
    }
  },
  "imports": {
    "k8s": {
      "provider": "Kubernetes",
      "version": "1.0",
      "config": {
        "namespace": "default"
      }
    }
  },
  "resources": {
    "stg": {
      "import": "k8s",
      "type": "apps/Deployment@v1",
      ...
    }
  }
}

What this could look like with `az`

Here is what the az provider would look like. I'm not locked on the idea of scope here, potentially this could be some sort of split between resource group, subscription id, tenant id, etc.

{
    "providers": {
        "az": {
            "scope": "/subscription/some-id/resourcegroup/my-rg",
            "type": "AzureResourceManager"
        }
    }
}

module container 'module.bicep' = {
  name: 'nginx'
  providers: {
    az: {
      scope: '/subscription/some-id/resourcegroup/my-rg', // Note, this value is used for the `resourceGroup()` and `subscription()` functions
      type: 'AzureResourceManager'
    }
  }
}

Usage in bicep:

import az as az

var uniqueStorageName = '${storagePrefix}${uniqueString(az.resourceGroup().id)}'

resource stg 'Microsoft.Storage/storageAccounts@2021-04-01' = {
  name: uniqueStorageName
  location: location
  sku: {
    name: storageSKU
  }
  kind: 'StorageV2'
  properties: {
    supportsHttpsTrafficOnly: true
  }
}

ARM json representation:

{
  ...
  "parameters": {
    ...
  },
  "variables": {
    "uniqueStorageName": "[format('{0}{1}', parameters('storagePrefix'), uniqueString(az.resourceGroup().id))]"
  },
  "providers": {
    "az": {
      "scope": "/subscription/some-id/resourcegroup/my-rg",
      "type": "AzureResourceManager"    
    }
  },
  "imports": {
    "az": {
      "provider": "AzureResourceManager",
      "version": "1.0"
    }
  },
  "resources": {
    "stg": {
      "import": "az",
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2021-04-01",
      "name": "[variables('uniqueStorageName')]",
      "location": "[parameters('location')]",
      "sku": {
        "name": "[parameters('storageSKU')]"
      },
      "kind": "StorageV2",
      "properties": {
        "supportsHttpsTrafficOnly": true
      }
    }
  }
}

How does configuration flow between modules

Modules provide interesting scenarios around how provider configuration should be propagated. There is prior art here though, which we probably need to follow. Specifically, the way resourceGroup and subscription flows today we will need to match for providers. These today flow across modules automatically, which means provider configuration is ambiently flowed to match this behavior.

For example, if we specify the kubeConfig for the k8s provider at a top level, it will be propagated to all submodules automatically. Ex: import k8s from kubernetes in a submodule would automatically have the kubeConfig set as well.

Can providers collide with parameters?

Today, if someone were to specify an import and param in the same file, the symbolic names would conflict causing a compilation error.

If someone were to specify a param in a submodule that would conflict with the provider symbolic name, this causes a conflict. There could be multiple different solutions here, but one solution would be that the submodule would explicitly need to require specifying the provider instead and the provider wouldn't be ambiently flowed.

Specifying via CLI

As part of the az CLI, we would add the ability to specify provider arguments similar to how parameters are specified via CLI. For now, this will only allow for json input rather than allowing for individual parameter values. We can expand this to support individual parameter values in the future.

az deployment group create --providers '{\"k8s\": {\"kubeConfig\": \"some-string\"}}'

az deployment group create --providers providers.json

Specifying via config file

The providers json input would look similar to the inputs prior, but can be written to a separate file as input:

{
  "$schema": "https://schema.management.azure.com/schemas/<SOME_VERSION>/deploymentProviders.json#",
  "contentVersion": "1.0.0.0",
  "providers": {
    "k8s": {
      "kubeConfig": "some-string",
      "type": "Kubernetes"
    }
  }
}

The JSON schema for the file would look like:

{
    "id": "https://schema.management.azure.com/schemas/2019-04-01/deploymentProviders.json#",
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Providers",
    "description": "An Azure deployment providers file",
    "type": "object",
    "properties": {
        "$schema": {
            "type": "string"
        },
        "contentVersion": {
            "type": "string",
            "pattern": "(^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+$)",
            "description": "A 4 number format for the version number of this provider file. For example, 1.0.0.0"
        },
        "providers": {
            "type": "object",
            "additionalProperties": {
                "$ref": "#/definitions/provider"
            }
        }
    },
    "additionalProperties": false,
    "required": [
        "$schema",
        "contentVersion",
        "providers"
    ],
    "definitions": {
        "providers": {
            "type": "object",
            "properties": {
                "provider": {
                    "type": "object",
                    "additionalProperties": {
                        "$ref": "#/definitions/provider"
                    }
                }
            },
            "additionalProperties": false,
        },
         "provider": {
            "type": "object",
            "properties": {
                "type": {
                    "type": "string",
                    "description": "The type of provider"
                },
            },
            "additionalProperties": {
                "type": ["object", "string"]
            }
        }
    }
}

Additionally, providers could also be appended to the schema for parameters, such that both providers and parameters could be specified in the same file.

{
  "$schema": "https://schema.management.azure.com/schemas/<SOME_VERSION>/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "providers": {
    "k8s": {
      "kubeConfig": "some-string",
      "type": "Kubernetes"
    }
  },
  "parameters": {
     .
  }
}

Error cases

A few common mistakes and how their errors will propagate.

Mistake: A field in my provider section is missing. Ex, I forgot the kubeConfig field when invoking az deployment. Result: On deployment, when deploying the resource that depends on kubeConfig, the deployment will fail.

Mistake: A provider is configured for an import that doesn’t exist. Result: On deployment, the deployment engine will see that a provider supplied doesn’t have a corresponding import, and will fail the deployment.

I've received feedback from the ARM team around how we can make these errors happen earlier. https://github.com/Azure/bicep/issues/1278 may be a follow up to consider.

How does this work with existing ways of specifying resourceGroup and subscription?

If and when we introduce the ability to specify a provider for az, there are now three different places where a resourceGroup and subscription can be specified.

Top level on the Microsoft.Resources/deployments
On the az provider
Part of the request path string

Here is the proposal on ordering:

If subscriptionId, resourceGroup, and/or scope are set, use that information in the deployment.
Otherwise, if a provider is set for an import, use that information.
Otherwise, default to information that is part of the URL.

Is this `globals`?

So a keen eye here would notice the proposal for providers matches closely to what behavior a global variable would have: https://github.com/Azure/bicep/issues/1121.

This is fairly intentional and may need to be thought about before implementing this feature. In the long term, if we wanted to add globals to bicep, the providers feature would make more sense to live in a section called globals inside of the Microsoft.Resources/deployments resource. The ask for globals may be the better path here; however that would be larger than the scope of this issue, albeit implementation very similar.

@alex-frankel @majastrz for follow up here. I think https://github.com/Azure/bicep/issues/1121 would be a launch pad for this and if we should pursue that proposal as well.

Functions on providers/imports

Another follow up to this issue would be to allow extensibility providers to define provider methods on themselves. For example, if I wanted to query the namespace of the provider, I could call k8s.namespace().

See https://github.com/Azure/bicep/issues/6652 for follow up. I think conceptually allowing for a function to get the current value of a provider similar to params would be the path of action.

Other options evaluated

I evaluated an option where instead of specifying a separate section for providers, it would be combine with the parameters section. This ended up being very tricky to reason about in bicep code and generally @alex-frankel @majastrz both agreed a separate section would be better.

Recommended operations

I think tackling both adding the providers section and updating the import az pieces can be done as separate work items. The updates to import az are tied to https://github.com/Azure/bicep/issues/6641 as well.

Open questions

What else should live on the az provider? Right now it is just the scope which can derive the subscription and resourceGroup. What about tenantId? Location?
Should we pursue globals here? Do we want to avoid designing a feature that fits one use when in reality it should support more use cases?

jkotalik commented 2 years ago

From the community call, https://github.com/Azure/bicep/issues/1121 seems to be very favorably looked at and may be something design in its entirety.

jkotalik commented 2 years ago

I think there are a few options here:

Work through the providers proposal designing around the fact that we want to add globals in the future (so naming, branding, etc will be based on globals). Then, initially only add support for providers in the globals section.
Add globals support and the entire E2E around it, then add support for providers after.

From my viewpoint, I'd strongly prefer the first option as it unblocks my team's scenarios and still leaves room for designing globals.

slavizh commented 2 years ago

For me this proposal seems ok. I cannot fully understand the need of this for k8s extensibility as my k8s knowledge is limited and we do not have k8s bicep extensibility yet to try it but this could work on passing some values between modules. Most notably:

tags
resources (resource IDs)

With this proposal and the future extensibility I wonder if may start to hit some of the most notable ARM template limits like:

800 resources/deployments
template/deployment/parameters size limits

If I am not mistaken somewhere there was a proposal for a breaking change as well for targetScope. If there is a breaking change for this would be nice if there are a few versions that allows you to have the old way and the new way at the same time. That way you can migrate over time without having to be stuck at particular version.

anthony-c-martin commented 2 years ago

Just wanted to put https://gist.github.com/anthony-c-martin/9d289ccb219aa4ebc934693a10f5c339 on your radar - my rough notes on solving a similar problem:

Generalizing the concept of 'declaring a scope' and 'satisfying a scope' for extensible resources.
Having minimal boilerplate to pass imports between modules.

slavizh commented 2 years ago

Can we get some mockup examples of how imports are used inside the module. For example kubeconfig

jkotalik commented 2 years ago

@anthony-c-martin yeah it does seem very similar to this proposal, glad we are thinking alike here. I think the main difference is being able to specify information on an import outside of a template (ex at deploy time like a parameter). But effectively the concepts you had for passing info into a module would be extended outside of the module to the deployment template, so I think it's a given.

shenglol commented 2 years ago

We had a discussion on this and came to the conclusion that it's worth adding a new property imports (instead of providers) to both Microsoft.Resources/deployments and ARM template for specifying deployment-time configuration for providers.

What this looks like with Kubernetes

k8sDeploy.bicep

import 'kubernetes@v1'
// The import statement has two side effects:
// 1. Implicitly declares a symbol "kubernetes" which can be accessed in the module.
//    The symbol name can be overridden using the 'as' keyword: import 'kubernetes@v1' as k8s.
// 2. When instantiating the module from a parent module, the configuration for 'kubernetes@v1' must be specified.
//    The configuration has the following type:
//    {
//      kubeConfig: secureString
//    }

resource myService 'core/Service@v1' = {
  metadata: {
    name: 'myService'
  }
  spec: {
    ports: [8888]
    // ...
  }
}

// Property properties can be accessed via the implicitly created symbol 'kubernetes'.
output providerVersion = kubernetes.version

main.bicep

module aksDeploy '...' = { /* ... */ }

module k8sDeploy 'k8sDeploy.bicep' = {
  imports: {
    kubernetes: {
      kubeConfig: aksDeploy.outputs.kubeConfig
    }
  }
}

Compiled ARM template:

{
  // ...
  "resources": {
    "aksDeploy": { /* ... */ },
    "k8sDeploy": {
      "type": "Microsoft.Resources/deployments",
      "properties": {
        "imports": {
          "kubernetes": {
            "config": {
              "value": "[reference('aksDeploy').outputs.kubeConfig]"
            }
          }
        },
        "template": {
          "imports": {
            "kubernetes": {
               "provider": "Kubernetes",
               "version": "v1",
               "config": {
                 "kubeConfig": {
                   "type": "secureString"
                 }
               }
            }
          },
          "resources": {
            "myService": {
              "import": "k8s",
              "type": "core/Service@v1",
              "properties": {
                "metadata": {
                  "name": "myService"
                },
                "spec": {
                  "ports": [8888]
                  // ...
                }
              }
            }
          },
          "outputs": {
             // A new template function "imports" will need to be added to access provider properties
            "providerVersion": "[imports('k8s').version]"
          }
        }
      }
    }
  }
}

What this could look like with `az`

child.bicep

import 'az@v1'
// or
// import 'az@v1' with {
//   targetScope: 'resourceGroup'
// }

main.bicep

import 'az@v1' with {
 targetScope: 'subscription'
}

resource rg 'Microsoft.Resources/resourceGroup@2020-01-01' = { ... }

module child 'child.bicep' = {
  name: 'child'
  imports: {
    az: {
      scope: rg
    }
  }
  // The top level scope property can still be used
  scope: rg
}

Compiled ARM template

imports configuration for az will not be emitted, since az is a first party resource provider, and the deployment engine knows how to handle Azure resources.

shenglol commented 2 years ago

Should this be global?

No. Globals is convenient, but it can be evil on the other hand if not used correctly. We've decided not to make imports global.

Topics to discuss:

Default import

Should az be imported by default?
If so, what version should be imported?
Should it be imported by the Bicep compiler?
Should it be part of a Bicep configuration file?

Deployment experience

How should import configuration be specified when running deployment commands? By adding a new imports parameter?
Currently, deployment commands at any scope (rg, sub, mg, and tenant) can be used to deploy non-azure resources. Should we create a unified deployment command for deploying both azure and non-azure resources?
Can imports configuration be specified via Bicep parameters file (https://github.com/Azure/bicep/issues/7301)? Should we revisit the design of Bicep parameters file?

slavizh commented 2 years ago

Should az be imported by default? - No, but it should be required to define it. That way it becomes part of standard writing of templates and anyone can choose the version they are comfortable, no matter the version of bicep.exe If so, what version should be imported? - N/A Should it be imported by the Bicep compiler? - No Should it be part of a Bicep configuration file? - No

How should import configuration be specified when running deployment commands? By adding a new imports parameter? - This could be additional option but I think it is better if these things are controlled completely by the Bicep template. Currently, deployment commands at any scope (rg, sub, mg, and tenant) can be used to deploy non-azure resources. Should we create a unified deployment command for deploying both azure and non-azure resources? - Yes Can imports be specified via Bicep parameters file ( https://github.com/Azure/bicep/issues/7301)? Should we revisit the design of Bicep parameters file? - I think it is better not as it will be weird experience. It will be like you can run a program A but it is up to you to specify the nugget package versions that will be used by that program.

shenglol commented 2 years ago

@slavizh , sorry, I made a mistake on the last discussion item. By specifying "imports via Bicep parameters file" I meant specifying import configuration values (I've revised that). The idea is that import configurations are very similar to parameters, and users may need to provide certain configuration values at deploy time, so we will need a way to set the values in a parameters file.

jeskew commented 2 years ago

I agree with @slavizh on not importing az by default on principle, but that would mean every existing Bicep template would need to be rewritten. Perhaps only imposing this requirement on users who have adopted imports would be a good compromise. If no import statements appear in a template, the compiler could assume it is dealing with a "legacy" template and automatically import az; if any import statements are used, then no providers would be imported by default.

slavizh commented 2 years ago

@jeskew some templates will have to be re-written anyway due to change how scope is defined.

slavizh commented 2 years ago

@shenglol I guess it is fine to have imports in parameters file but I would like to see to specify schema for parameters file so even if you have parameter like object that has 5 levels for example to have intellisense for the lowest level properties. So may be import will be more suitable for that.

jeskew commented 2 years ago

@jeskew some templates will have to be re-written anyway due to change how scope is defined.

We could choose to do that, but I would advocate for allowing use of the targetScope keyword in files that do not use the import keyword. We can add a deprecation notice and discuss eventually phasing out the default az import and support for the targetScope keyword (e.g., in the 1.0 release), but I don't think it would be very user friendly to abruptly shift to requiring an import 'az@v1' statement in all templates written before providers were introduced.

elygre commented 2 years ago

Re-listening to the community call, I would also like to advocate for a solution where "no import statement" means "automatic import of az", while "any import statement" means "import only what is specified".

ghost commented 1 year ago

Hi jkotalik, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! :smile: :mechanical_arm:

Azure / bicep

Proposal: add `providers` to Microsoft.Resources/deployments #6653

Adding `providers`

Problem today

Proposal

What this looks like with `kubernetes`

What this could look like with `az`

How does configuration flow between modules

Can providers collide with parameters?

Specifying via CLI

Specifying via config file

Error cases

How does this work with existing ways of specifying resourceGroup and subscription?

Is this `globals`?

Functions on providers/imports

Other options evaluated

Recommended operations

Open questions

What this looks like with Kubernetes

k8sDeploy.bicep

main.bicep

Compiled ARM template:

What this could look like with `az`

child.bicep

main.bicep

Compiled ARM template

Should this be global?

Topics to discuss:

Default import

Deployment experience

Azure / bicep

Proposal: add `providers` to Microsoft.Resources/deployments #6653

Adding providers

Problem today

Proposal

What this looks like with kubernetes

What this could look like with az

How does configuration flow between modules

Can providers collide with parameters?

Specifying via CLI

Specifying via config file

Error cases

How does this work with existing ways of specifying resourceGroup and subscription?

Is this globals?

Functions on providers/imports

Other options evaluated

Recommended operations

Open questions

What this looks like with Kubernetes

k8sDeploy.bicep

main.bicep

Compiled ARM template:

What this could look like with az

child.bicep

main.bicep

Compiled ARM template

Should this be global?

Topics to discuss:

Default import

Deployment experience

Adding `providers`

What this looks like with `kubernetes`

What this could look like with `az`

Is this `globals`?

What this could look like with `az`