Azure / bicep

Bicep is a declarative language for describing and deploying Azure resources
MIT License
3.22k stars 746 forks source link

loadTextContent() - support variable or parameter as input #3816

Open slavizh opened 3 years ago

slavizh commented 3 years ago

Is your feature request related to a problem? Please describe.

I would like for loadTextContent() to support variable or parameter as input parameter. What I would like to offer is for the end user to specify text from file or to use the default file available. Currently if I use the below syntax I get error: The value must be a compile-time constant.bicep(BCP032)

I know that function is being used at compile time so this will probably require some other approach and even changes in template spec for supporting it there as well.

var alertRules = {
  memoryLow: {
    description: empty(bastionHostMonitoring.alertRules.memoryLow.descriptionFilePath) ? loadTextContent('alert-descriptions/memory-low.txt') : loadTextContent(bastionHostMonitoring.alertRules.memoryLow.descriptionFilePath)
  }
}
alex-frankel commented 3 years ago

We cannot do this with a param since we won't know the value at compile time. In other words, if you were to run bicep build with a param argument, we'd have no idea which file to "load".

In theory, this should be possible with variables, but my understanding is it requires more of #444 implemented

mblant commented 2 years ago

Very interested in this thread.

The ability to load another file inside a var would open a load of options around not hardcoded 10s or 100s are variables to reference existing file to load data

var manifest = json(loadTextContent('./data/manifest.json'))

var itemsArray = [ for item in manifest : {
  data: json(loadTextContent('./data/${item.dataFile}'))
  parameters: json(loadTextContent('./data/${item.parametersFile}'))
}

In theory, this should all be available at compile time as all files are static and nothing is dynamically generated. The manifest object is be generated. So, the itemsArray should be compiled off manifest as well at compile time. Depending on how the complier interrupts order of operation, right?

alex-frankel commented 2 years ago

3607 seems related to this @mblant, but it is not the same ask

mblant commented 2 years ago

I removed the string interpellation and just put the file path in the manifest file and it worked. So, I then assumed it is the interpellation causing the compile time error.

But then, after three or four runs in my test environment, it’s back to the compile time error “The value must be a compile-time constant. bicep(BCP032)” Not sure what is up in the compiler that it let it run several times (and it actually worked!!) only to go back to throwing the error with no changes? Happy to get some more data, if someone can give a pointer on where to start.

az bicep version Bicep CLI version 0.4.1124 (66c84c8ee5)

mblant commented 2 years ago

Interesting. No errors if there is only a single element in the manifest.json. Add additional elements and back to failure.

leechan-bicep commented 2 years ago

Hi, I have the same error when providing more than one file location getting the "The value must be a compile-time constant." error.Please let us know how to fix this issue

var stringArray = [ './policies/policy_1.json' ]

var policies = [for i in range(0, length(stringArray)): json(loadTextContent(stringArray[i]))]

alex-frankel commented 2 years ago

Repro for triage:

works:

var stringArray = [
  './policies/policy_1.json'
]

var policies = [for i in range(0, length(stringArray)): json(loadTextContent(stringArray[i]))]

doesn't work:

var stringArray = [
  './policies/policy_1.json'
  './policies/policy_2.json'
]

var policies = [for i in range(0, length(stringArray)): json(loadTextContent(stringArray[i]))]
alex-frankel commented 2 years ago

The reason this works is because the type of a single-item string array gets simplified down to a string literal, which the function can handle. It's not something we intentionally designed and in fact we should probably block this scenario. We are going to take a look at validating this better.

alex-frankel commented 2 years ago

@miqm will break out this last comment and track the fix with a separate issue

miqm commented 2 years ago

Anyway, back to the original proposal - using parameters as arguments for loadTextContent is not possible. Parameters are evaluated during the deployment (runtime), while file loading in loadTextContent is done during compilation (bicep build command). the azure cli does the compilation on the fly before the actual deployment occurs, hence the impression that it could be done.

The only way I can think of to provide this functionality is to introduce a compile-time parameters (similar to C# symbols - https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/preprocessor-directives#conditional-compilation). AFAIR there are some other issues that refers to introducing similar functionality (i.e. #3109, #1761). It might be worth to think about it. but IMHO it will not be as fast as you would like :)

cpinotossi commented 2 years ago

Hello, would be great if we would support this and make it more general. So it will also support similar functions like loadFileAsBase64. It would give more flexibility when working with modules.

miqm commented 2 years ago

Would having function that loads json from file be a solution? Mind, that vars can be runtime values, while load* functions can accept only compile-time values as arguments.

wsucoug69 commented 2 years ago

We have the same requirement\need, we are creating role definitions and looking to pass in the json file containing the permissions.

garrettsingletary commented 2 years ago

+1 to this. Have the same need for populating permissions from json.

charotAmine commented 2 years ago

Hello @leechan-bicep , @alex-frankel

A workaround for this :

Hi, I have the same error when providing more than one file location getting the "The value must be a compile-time constant." error.Please let us know how to fix this issue

var stringArray = [ './policies/policy_1.json' ]

var policies = [for i in range(0, length(stringArray)): json(loadTextContent(stringArray[i]))]

May be : var stringArray = [ loadTextContent('./policy-1.json') loadTextContent('./policy-2.json') loadTextContent('./policy-3.json') ]

var policies = [for i in range(0, length(stringArray)): json(stringArray[i])] Since in anyways, we should put a constant path.

SeSeicht commented 2 years ago

hey, I tried @charotAmine code, but it didn't worked for me. But I created a similar way to solve this: @alex-frankel goes in the same direction like your code snippet

var stringArray = [
  json(loadTextContent('./file1.json'))
  json(loadTextContent('./file2json'))
  json(loadTextContent('./file3.json'))
  ]

module m_pol_def 'br/modules:microsoft.authorization.policydefinitions:1.0' = [for onePol in stringArray: { 
  name: onePol.name
    params: {...}
}]
calebherbison commented 2 years ago

+1 for using this to load local files for openapi specs

krowlandson commented 2 years ago

@alex-frankel

I have a similar situation which appears to be a cross between this issue and #3607.

In my case, I am looking to load multiple "resource configuration" files. These files may vary from deployment to deployment but will be re-usable. Depending on the specific scenario, I would like to point my Bicep configuration at a single JSON file containing a list of required "resource configuration" files, and then loop over that to load in the "resource configuration" files.

I will always know the location of the first file, and the point of this is to abstract the list of other sources from the template to simplify code maintenance and D.R.Y.

Take an example such as the following:

configLocations.Scenario1.json example

[
    "config/resource1settings.json",
    "config/resource2settings.json",
    "config/resource3settings.json",
    "config/resource4settings.json",
    "config/resource5settings.json",
    "config/resource6settings.json"
]

config/resource1settings.json example

{
  "setting1": "value",
  "setting2": 100,
  "setting3": true,
  "setting4": []
}

sample.bicep example

// The following var is used to load the list of resource configs to import
var listResourceConfigs = loadJsonContent('configLocations.Scenario1.json')

// The following var is used to load the resource configs
var resourceConfigs = [for config in listResourceConfigs: loadJsonContent(config)]

This results in the error The value must be a compile-time constant.bicep(BCP032):

Bicep error: The value must be a compile-time constant.bicep(BCP032)

As such, it would be really useful to have a way to do the following for compile-time operations:

  1. Declare constants (only visible in Bicep, maybe using a decorator format such as @constant(key='value'))
  2. Support for loops (again, possibly using a different format to declare the loops so they are explicitly identified as used at compile-time)

In my scenario, explicitly declaring the list of "resource configuration" files into the bicep file as proposed in the above "work-arounds" would defeat the point of this abstraction.

Thank you

ezYakaEagle442 commented 2 years ago

I have a similar requirement:

I try to configure an Azure Container App setting a pram to set the Resources CPU&RAM. I hit The value must be a compile-time constant.bicep(BCP032):

How to implement this please ?


@allowed([
  json('0.25')
  json('0.5')
  json('0.75')
  json('1.0')  
  json('1.25')
  json('1.5')
  json('1.75')
  json('2.0')    
])
@description('The container Resources CPU')
param containerResourcesCpu object = json('0.25')

@allowed([
  json('0.5')
  json('1.0')  
  json('1.5')
  json('2.0')    
  json('2.5')
  json('3.0')  
  json('3.5')
  json('4.0')    
])
@description('The container Resources Memory')
param containerResourcesMemory object = json('0.5')

resource ContainerApp 'Microsoft.App/containerApps@2022-03-01' = {
  properties: {
    template: {
      containers: [
        {
          resources: {
            cpu: containerResourcesCpu // json('0.5') //500m
            memory: containerResourcesMemory // Gi
          }
miqm commented 2 years ago

I have a similar requirement:

I try to configure an Azure Container App setting a pram to set the Resources CPU&RAM. I hit The value must be a compile-time constant.bicep(BCP032):

How to implement this please ?


@allowed([
  json('0.25')
  json('0.5')
  json('0.75')
  json('1.0')  
  json('1.25')
  json('1.5')
  json('1.75')
  json('2.0')    
])
@description('The container Resources CPU')
param containerResourcesCpu object = json('0.25')

@allowed([
  json('0.5')
  json('1.0')  
  json('1.5')
  json('2.0')    
  json('2.5')
  json('3.0')  
  json('3.5')
  json('4.0')    
])
@description('The container Resources Memory')
param containerResourcesMemory object = json('0.5')

resource ContainerApp 'Microsoft.App/containerApps@2022-03-01' = {
  properties: {
    template: {
      containers: [
        {
          resources: {
            cpu: containerResourcesCpu // json('0.5') //500m
            memory: containerResourcesMemory // Gi
          }

Your problem is related to lack of floating point number support in bicep: #1386

ezYakaEagle442 commented 2 years ago

Indeed, it is now fixed with :


@allowed([
  '0.25'
  '0.5'
  '0.75'
  '1.0' 
  '1.25'
  '1.5'
  '1.75'
  '2.0'    
])
@description('The container Resources CPU')
param containerResourcesCpu string = '0.5'

@allowed([
  '0.5'
  '1.0'  
  '1.5'
  '2.0'    
  '2.5'
  '3.0'  
  '3.5'
  '4.0'    
])
@description('The container Resources Memory')
param containerResourcesMemory string = '1.0'

resource ContainerApp 'Microsoft.App/containerApps@2022-03-01' = {
  properties: {
    template: {
      containers: [
        {
          resources: {
            cpu: json(containerResourcesCpu) // 250m
            memory: json(containerResourcesMemory) // Gi
          }
ZWMiller commented 2 years ago

I believe this is a similar problem and wanted to see if there is a resolution yet. I'm working on generalizing deployment of VMs with BICEP templating. One of the things I want to do is manage which machine sizes the deployment has access to, but also control for different deployment configuration for GPU machines vs CPU machines. Thus, I'd like to do:

var allowed_cpu = array([
  'standard_b1ms'
])
var allowed_gpu = array([
  'Standard_NC6s_v3'
  'Standard_NC12s_v3'
  'Standard_NC4as_T4_v3'
])
var allowed_vms = concat(allowed_cpu, allowed_gpu)
@allowed(allowed_vms)
param vm_type string = 'standard_b1ms'

However, doing so marks the @allowed attempt as "must be a compile time constant". Is it still impossible to uses var as part of @allowed?

miqm commented 2 years ago

I believe this is a similar problem and wanted to see if there is a resolution yet. I'm working on generalizing deployment of VMs with BICEP templating. One of the things I want to do is manage which machine sizes the deployment has access to, but also control for different deployment configuration for GPU machines vs CPU machines. Thus, I'd like to do:

var allowed_cpu = array([
  'standard_b1ms'
])
var allowed_gpu = array([
  'Standard_NC6s_v3'
  'Standard_NC12s_v3'
  'Standard_NC4as_T4_v3'
])
var allowed_vms = concat(allowed_cpu, allowed_gpu)
@allowed(allowed_vms)
param vm_type string = 'standard_b1ms'

However, doing so marks the @allowed attempt as "must be a compile time constant". Is it still impossible to uses var as part of @allowed?

var yes, but not concat - as this is a runtime function.

However you should be able to use loadJsonContent inside @allowed and keep VM list in separate json and use query parameter to get what you need - but I see it does not work :/ I'll put an issue and try to fix it.

jeskew commented 1 year ago

The runtime engine doesn't support variable() function expressions within an allowedValues parameter property, so we would either need to update the engine to support such expressions or convert the result of loadJsonContent to a string literal rather than a template expression.

image image

dontpanic003 commented 1 year ago

+1 to the request.

akhilthomas011 commented 1 year ago

@alex-frankel +1. This is a frequent use case for us.

Kaloszer commented 1 year ago

+1

sofia-valles commented 1 year ago

+1

Gabriel123N commented 1 year ago

+1

tuomaskujala commented 1 year ago

+1

asos-gurpreetsingh commented 1 year ago

+100 from my company

brwilkinson commented 1 year ago

Adding comment for support.

I currently pull in 3 files with Global or Regional metadata. 1) Global metadata 1) Regional metadata 1) Custom role definition lookups

    # Read in the Rolegroups Lookup.
    $RolesGroupsLookup = Get-Content -Path $Artifacts/tenants/$App/Global-Config.json | ConvertFrom-Json -Depth 10 | ForEach-Object RolesGroupsLookup
    $Global.Add('RolesGroupsLookup', ($RolesGroupsLookup | ConvertTo-Json -Compress -Depth 10))

I would like to move this into the Bicep template file itself.

param App string = 'LAB'
var computeGlobal = json(loadTextContent('../tenants/${App}/Global-Config.json'))

Since the project can deploy to multiple subscriptions, I need to pass in the App to differentiate which Role lookup file to use,

The value must be a compile-time constant.

The deployment script that I currently use for ALL deployments.

Examples of the metadata files that I have-to pass in at runtime (per tenant/App) in the above script.

UmairSyed commented 1 year ago

+1 please

Kielar-Peter commented 1 year ago

+1

AshutoshNirkhe commented 1 year ago

This will be really a value addition!

jamesSampica commented 1 year ago

This is something Terraform supports so Bicep should too. It is not possible to download and deploy a certificate resource unless the path is known at deploy time. Only workaround is using a keyvault instead.

Terraform:

resource "azurerm_app_service_certificate" "webapp_certificate" {
    ...
    pfx_blob = filebase64(var.certificate_path) // Works
}

Bicep:

resource certificate 'Microsoft.Web/certificates@2022-09-01' = {
    ...
    properties: {
        pfx_blob: loadFileAsBase64(certificate_path) // Compile error
    }
}
brwilkinson commented 1 year ago

An additional comment on this... If you test out bicepparam you can create a parameter that uses loadtextcontent() or other loadX() to read the file. By having this in the parameter file instead of the template, you get some degree in flexibility to read different files based on the parameter file you are using.

jamesSampica commented 1 year ago

Good to know. I dont think a parameter file will work in my case because I want to download a secure file from Azure DevOps and read the contents in.

brwilkinson commented 1 year ago

@jamesSampica

Bicep functions are supported in bicepparam.

https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-string#base64tostring

https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-files#loadfileasbase64

brwilkinson commented 1 year ago

Still exploring bicepparam

I like that you can reference other parameters in bicepparam file. E.g. Prefix below.

using '../../bicep/00-ALL-SUB.bicep'

param Global = union(
  loadJsonContent('Global-${Prefix}.json'), // Can reference this file based on the region I am in... clone the file, update the Prefix
  loadJsonContent('Global-Global.json'),
  loadJsonContent('Global-Config.json')
  )

param Prefix = 'AWU3'
param Environment = 'P'
param DeploymentID = '0'

I haven't completed testing all scenarios, however I believe this covers my needs from my previous comment in this thread, where I achieved the above union with PowerShell custom code.

sohirani commented 1 year ago

+1

gscorrea commented 1 year ago

+1 I am in the middle of defining variables that need to ready yaml/json/text contents from files to build AOSM NFDV for a bicep publisher file. The thing is that besides these variables the bicep publisher is common to several of my NFs to be published, so I do not want to have to make copies of same files several times to build the NFDV for my NFs, but I want to use params/variables to build the load(Yaml/Json/Text)Content filePath and use same bicep to publish my NFs NFDVs. Commented out below is how I tried to implement with params on the filePath and did not work, but this is needed. I understand that at this time the bicep build process happens before the params are read, but I have a need to have the params available at bicep build time.

param aksClusterGroup string
param aksClusterName string
param aosmParametersLocation string
param aosmSchemaLocation string
param aosmSetSetFilesParametersLocation string
param artifactManifestName string
param artifactStore string
param cgsName string
param cgvName string
param cnfName string
param helmPackageName string
param helmPackageVersion string
param helmReleaseName string
param location string = resourceGroup().location
param nfdgroupName string
param nfdvNfName string
param nfdversion string
param nsdGroup string
param nsdvName string
param publisherName string
param releaseNamespace string
param resourceExists bool = false
param siteName string
param snsName string
param valuesLocation string
param valuesParam object
param yamlLocation string
//********************************************
//LOADING ALL VALUES ON allTogether
//********************************************
//var mval = loadYamlContent(valuesLocation'/'cnfName'-values.yaml') //parameters passed to helm chart
//var svals = loadJsonContent(aosmSetSetFilesParametersLocation'/'cnfName'_nf_deployment_set_parameters.json') //file with --set parameters for helm
var mval = loadYamlContent('../values.final/my_cnf-values.yaml') //parameters passed to helm chart
var svals = loadJsonContent('../aosmSetSetFilesParameters.final/my_cnf_nf_deployment_set_parameters.json') //file with --set parameters for helm
var allTogether= union(mval, svals) //joining parameters. change order of list to use correct precedence.
Afsalmc commented 1 year ago

I am facing the same issue while loading a script for deploymentscript bicep module.

obriankevin11 commented 11 months ago

same here

lesca commented 11 months ago

+1

thisispr commented 10 months ago

+1

hybridtechie commented 6 months ago

+1

Saulopv commented 6 months ago

+1

SvenRelijveld1995 commented 6 months ago

+1

dewolfs commented 6 months ago

Running into the error

Error BCP032: The value must be a compile-time constant.

when trying to import an API into Azure API Management based on the OpenAPI specification. Our API is private so we need to use format openapi+json.

Ref: https://learn.microsoft.com/en-us/javascript/api/@azure/arm-apimanagement/contentformat?view=azure-node-latest

param openApiSpecJSON string

resource apis 'Microsoft.ApiManagement/service/apis@2022-08-01' = {
    ...
    description: api.apiDescription
    format: 'openapi+json'
    value: loadTextContent('${openApiSpecJSON}')
  }
}