Closed jkotalik closed 1 year ago
From the community call, https://github.com/Azure/bicep/issues/1121 seems to be very favorably looked at and may be something design in its entirety.
I think there are a few options here:
From my viewpoint, I'd strongly prefer the first option as it unblocks my team's scenarios and still leaves room for designing globals.
For me this proposal seems ok. I cannot fully understand the need of this for k8s extensibility as my k8s knowledge is limited and we do not have k8s bicep extensibility yet to try it but this could work on passing some values between modules. Most notably:
With this proposal and the future extensibility I wonder if may start to hit some of the most notable ARM template limits like:
If I am not mistaken somewhere there was a proposal for a breaking change as well for targetScope. If there is a breaking change for this would be nice if there are a few versions that allows you to have the old way and the new way at the same time. That way you can migrate over time without having to be stuck at particular version.
Just wanted to put https://gist.github.com/anthony-c-martin/9d289ccb219aa4ebc934693a10f5c339 on your radar - my rough notes on solving a similar problem:
Can we get some mockup examples of how imports are used inside the module. For example kubeconfig
@anthony-c-martin yeah it does seem very similar to this proposal, glad we are thinking alike here. I think the main difference is being able to specify information on an import outside of a template (ex at deploy time like a parameter). But effectively the concepts you had for passing info into a module would be extended outside of the module to the deployment template, so I think it's a given.
We had a discussion on this and came to the conclusion that it's worth adding a new property imports
(instead of providers
) to both Microsoft.Resources/deployments
and ARM template for specifying deployment-time configuration for providers.
import 'kubernetes@v1'
// The import statement has two side effects:
// 1. Implicitly declares a symbol "kubernetes" which can be accessed in the module.
// The symbol name can be overridden using the 'as' keyword: import 'kubernetes@v1' as k8s.
// 2. When instantiating the module from a parent module, the configuration for 'kubernetes@v1' must be specified.
// The configuration has the following type:
// {
// kubeConfig: secureString
// }
resource myService 'core/Service@v1' = {
metadata: {
name: 'myService'
}
spec: {
ports: [8888]
// ...
}
}
// Property properties can be accessed via the implicitly created symbol 'kubernetes'.
output providerVersion = kubernetes.version
module aksDeploy '...' = { /* ... */ }
module k8sDeploy 'k8sDeploy.bicep' = {
imports: {
kubernetes: {
kubeConfig: aksDeploy.outputs.kubeConfig
}
}
}
{
// ...
"resources": {
"aksDeploy": { /* ... */ },
"k8sDeploy": {
"type": "Microsoft.Resources/deployments",
"properties": {
"imports": {
"kubernetes": {
"config": {
"value": "[reference('aksDeploy').outputs.kubeConfig]"
}
}
},
"template": {
"imports": {
"kubernetes": {
"provider": "Kubernetes",
"version": "v1",
"config": {
"kubeConfig": {
"type": "secureString"
}
}
}
},
"resources": {
"myService": {
"import": "k8s",
"type": "core/Service@v1",
"properties": {
"metadata": {
"name": "myService"
},
"spec": {
"ports": [8888]
// ...
}
}
}
},
"outputs": {
// A new template function "imports" will need to be added to access provider properties
"providerVersion": "[imports('k8s').version]"
}
}
}
}
}
}
az
import 'az@v1'
// or
// import 'az@v1' with {
// targetScope: 'resourceGroup'
// }
import 'az@v1' with {
targetScope: 'subscription'
}
resource rg 'Microsoft.Resources/resourceGroup@2020-01-01' = { ... }
module child 'child.bicep' = {
name: 'child'
imports: {
az: {
scope: rg
}
}
// The top level scope property can still be used
scope: rg
}
imports
configuration for az
will not be emitted, since az
is a first party resource provider, and the deployment engine knows how to handle Azure resources.
No. Globals is convenient, but it can be evil on the other hand if not used correctly. We've decided not to make imports
global.
az
be imported by default?imports
parameter?imports
configuration be specified via Bicep parameters file (https://github.com/Azure/bicep/issues/7301)? Should we revisit the design of Bicep parameters file?Should az be imported by default? - No, but it should be required to define it. That way it becomes part of standard writing of templates and anyone can choose the version they are comfortable, no matter the version of bicep.exe If so, what version should be imported? - N/A Should it be imported by the Bicep compiler? - No Should it be part of a Bicep configuration file? - No
How should import configuration be specified when running deployment commands? By adding a new imports parameter? - This could be additional option but I think it is better if these things are controlled completely by the Bicep template. Currently, deployment commands at any scope (rg, sub, mg, and tenant) can be used to deploy non-azure resources. Should we create a unified deployment command for deploying both azure and non-azure resources? - Yes Can imports be specified via Bicep parameters file ( https://github.com/Azure/bicep/issues/7301)? Should we revisit the design of Bicep parameters file? - I think it is better not as it will be weird experience. It will be like you can run a program A but it is up to you to specify the nugget package versions that will be used by that program.
@slavizh , sorry, I made a mistake on the last discussion item. By specifying "imports
via Bicep parameters file" I meant specifying import configuration values (I've revised that). The idea is that import configurations are very similar to parameters, and users may need to provide certain configuration values at deploy time, so we will need a way to set the values in a parameters file.
I agree with @slavizh on not importing az by default on principle, but that would mean every existing Bicep template would need to be rewritten. Perhaps only imposing this requirement on users who have adopted imports would be a good compromise. If no import
statements appear in a template, the compiler could assume it is dealing with a "legacy" template and automatically import az; if any import
statements are used, then no providers would be imported by default.
@jeskew some templates will have to be re-written anyway due to change how scope is defined.
@shenglol I guess it is fine to have imports in parameters file but I would like to see to specify schema for parameters file so even if you have parameter like object that has 5 levels for example to have intellisense for the lowest level properties. So may be import will be more suitable for that.
@jeskew some templates will have to be re-written anyway due to change how scope is defined.
We could choose to do that, but I would advocate for allowing use of the targetScope
keyword in files that do not use the import
keyword. We can add a deprecation notice and discuss eventually phasing out the default az
import and support for the targetScope
keyword (e.g., in the 1.0 release), but I don't think it would be very user friendly to abruptly shift to requiring an import 'az@v1'
statement in all templates written before providers were introduced.
Re-listening to the community call, I would also like to advocate for a solution where "no import statement" means "automatic import of az", while "any import statement" means "import only what is specified".
Hi jkotalik, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! :smile: :mechanical_arm:
Adding
providers
This is a proposal to add
providers
to Microsoft.Resources/deployments as a way to specify deployment-time configuration for imports statements.Problem today
Bicep extensibility allows for deploying resources outside of the ARM control plane. For example, one can deploy a
Kubernetes
resource today to a AKS cluster purely by defining the Kubernetes resources in bicep: https://github.com/Azure/bicep/tree/main/docs/examples_extensibility/aks.Extensible resources are imported into a bicep file via the
import
keyword. This allows configuring options likenamespace
andkubeConfig
on the import statement to be used when deploying Kubernetes resources.Though passing a parameter will work for this scenario, let's say we start using
module
(s) to deploy resources. This will add a lot of complexity to the bicep file, as we need to pass this "ambient" context around everywhere we want to importkubernetes
.This problem will expand rapidly as more parameters are added to an import statement.
For comparison, let's look at the
az
provider itself. The call toimport az
into the bicep file is:Where calls to get information like the
resourceGroup
orsubscriptionId
are made:So logical questions would be:
az
provider know whatresourceGroup
is and what to use?az
provider have this capability and not thekubernetes
provider?For an ARM deployment, the
resourceGroup
is specified at deployment-time and is ambiently available when resources are being deployed as part of ARM. That is, when ARM resources are being deployed, if the scope is correct, the value forresourceGroup
is always available.So for the
Kubernetes
extensibility provider, and eventually other providers, what if we wanted to ambiently have thekubeConfig
available when deploying Kubernetes resources? Today, we only can pass thekubeConfig
as a parameter to theimport
statement, unlike theaz
provider where you can specify theresourceGroup
andsubscriptionId
as ambient configuration on a deployment and set as part ofaz deployment
command.Proposal
To be able to specify configuration at deployment time, I'd like to propose a new section to the
Microsoft.Resources/deployments
schema.Providers are a section to specify configuration at deployment time. This would start being used by all import statements in bicep. The additional required properties on the
az
import is out of scope of this initial proposal for providers, but long term having the ability to specify the resource group and subscription on the import via providers would be great.Additionally, the bicep definition for a module would also need to update to support providers as well:
Providers would require specifying the type of provider. The rest of it would be a collection of properties which are specific to the provider. For example, the
az
provider would allow specifying thescope
of a deployment (which could either be the subscription, resourceGroup, etc).This information will be ambiently available for each resource being deployed inside of the deployment engine. For example, each time a
Kubernetes
resource is deployed, thekubeConfig
will be available and sent as part of the call to the Extensibility Provider API. This part would require changes in the ARM Deployment Engine to make sure provider configuration is propagated correctly to Extensibility Providers.What this looks like with
kubernetes
Here is what the
kubernetes
provider would look like where we want to specify the kubeConfig at deployment time:Usage in bicep:
ARM json representation:
What this could look like with
az
Here is what the
az
provider would look like. I'm not locked on the idea of scope here, potentially this could be some sort of split between resource group, subscription id, tenant id, etc.Usage in bicep:
ARM json representation:
How does configuration flow between modules
Modules provide interesting scenarios around how provider configuration should be propagated. There is prior art here though, which we probably need to follow. Specifically, the way resourceGroup and subscription flows today we will need to match for providers. These today flow across modules automatically, which means provider configuration is ambiently flowed to match this behavior.
For example, if we specify the kubeConfig for the
k8s
provider at a top level, it will be propagated to all submodules automatically. Ex:import k8s from kubernetes
in a submodule would automatically have the kubeConfig set as well.Can providers collide with parameters?
Today, if someone were to specify an import and param in the same file, the symbolic names would conflict causing a compilation error.
If someone were to specify a
param
in a submodule that would conflict with the provider symbolic name, this causes a conflict. There could be multiple different solutions here, but one solution would be that the submodule would explicitly need to require specifying the provider instead and the provider wouldn't be ambiently flowed.Specifying via CLI
As part of the
az
CLI, we would add the ability to specify provider arguments similar to how parameters are specified via CLI. For now, this will only allow for json input rather than allowing for individual parameter values. We can expand this to support individual parameter values in the future.Specifying via config file
The providers json input would look similar to the inputs prior, but can be written to a separate file as input:
The JSON schema for the file would look like:
Additionally, providers could also be appended to the schema for parameters, such that both providers and parameters could be specified in the same file.
Error cases
A few common mistakes and how their errors will propagate.
Mistake: A field in my provider section is missing. Ex, I forgot the kubeConfig field when invoking
az deployment
. Result: On deployment, when deploying the resource that depends on kubeConfig, the deployment will fail.Mistake: A provider is configured for an import that doesn’t exist. Result: On deployment, the deployment engine will see that a provider supplied doesn’t have a corresponding import, and will fail the deployment.
I've received feedback from the ARM team around how we can make these errors happen earlier. https://github.com/Azure/bicep/issues/1278 may be a follow up to consider.
How does this work with existing ways of specifying resourceGroup and subscription?
If and when we introduce the ability to specify a provider for
az
, there are now three different places where a resourceGroup and subscription can be specified.Here is the proposal on ordering:
Is this
globals
?So a keen eye here would notice the proposal for
providers
matches closely to what behavior aglobal
variable would have: https://github.com/Azure/bicep/issues/1121.This is fairly intentional and may need to be thought about before implementing this feature. In the long term, if we wanted to add
globals
to bicep, the providers feature would make more sense to live in a section calledglobals
inside of the Microsoft.Resources/deployments resource. The ask for globals may be the better path here; however that would be larger than the scope of this issue, albeit implementation very similar.@alex-frankel @majastrz for follow up here. I think https://github.com/Azure/bicep/issues/1121 would be a launch pad for this and if we should pursue that proposal as well.
Functions on providers/imports
Another follow up to this issue would be to allow extensibility providers to define provider methods on themselves. For example, if I wanted to query the namespace of the provider, I could call
k8s.namespace()
.See https://github.com/Azure/bicep/issues/6652 for follow up. I think conceptually allowing for a function to get the current value of a provider similar to params would be the path of action.
Other options evaluated
I evaluated an option where instead of specifying a separate section for providers, it would be combine with the
parameters
section. This ended up being very tricky to reason about inbicep
code and generally @alex-frankel @majastrz both agreed a separate section would be better.Recommended operations
I think tackling both adding the providers section and updating the
import az
pieces can be done as separate work items. The updates toimport az
are tied to https://github.com/Azure/bicep/issues/6641 as well.Open questions
az
provider? Right now it is just thescope
which can derive the subscription and resourceGroup. What about tenantId? Location?globals
here? Do we want to avoid designing a feature that fits one use when in reality it should support more use cases?