Increase MaxLiteralCharacterLimit for loadTextContent (specifically for non-JSON content)

[JayDoubleu]

Increase MaxLiteralCharacterLimit for loadTextContent https://github.com/Azure/bicep/blob/main/src/Bicep.Core/LanguageConstants.cs#L32

Some of logic app JSON code as well as Azure Workbooks cannot be used with this function due to them being larger than 131072 characters. A lot of these resources would be created via azure portal and then used within arm templates/bicep for deployment. It would be great to have the ability to load it's content from within a separate file.

[slavizh]

+1

[majastrz]

Options to implement this:

• We could increase the limit in the runtime.
• We could put the content into a variable instead of a JSON expression string literal. (Would have to avoid clashes with user-defined variables, though.)

[alex-frankel]

Related:

• 3315

[minhquang]

Exactly I got the same issue with string literal limit length

[alex-frankel]

We are going to investigate increasing the runtime limit

[stutommi]

Is there any update on this investigation? I've found the same character limitation brings down an idea of using otherwise fully functioning API Management api publishments using bicep, because openapi files can be larger than the current 131072 character limitation allows.

[BCOps]

I'd really like this feature too. I have a separate process that adds a lot of elements to an app gateway, which I then update intermittently via Bicep as part of my core infrastructure maintenance. I read in the appGw config as it stands and can output it to a file for Bicep to use so that the appGw template is made up of all the features I need without overwriting the separate process changes that were made. The file size limit really restricts this as my appGw grows in size though, so runtime changes would be brilliant to amend this higher... @alex-frankel is this on a roadmap for future implementation at all?

[alex-frankel]

It's something we would like to do, but I doubt it will be dealt with in the next 3-6 months. It would be helpful if this issue gets more upvotes, so we can shift priorities accordingly.

[slavizh]

@alex-frankel I have a monitor workbook with more than 131072 characters. I wonder can you somehow make the function to not account for tabs/whitespaces or that does not matter for the function?

[mariussm]

You can try to deploy the SOC Process Framework Sentinel workbook from https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Workbooks/SOCProcessFramework.json - I have found no way to do this using Bicep, because of this limitation. The same number appears when i try to add the content using json() with a string rather than through loadTextContent

[raunaknarooka]

It is a blocker for deploying open api spec files for an API. Those files can get larger if they have more operations :(

[jessicalavoie]

Any ETA on when this could be implemented ? im hitting this problem right now with a workbook im trying to use.

[lukaszgx]

As someone mentioned above it's a blocker really. Bicep seems ok when you want to build something simple but if you got something advanced to do it might be better to stick to something else like terraform. Is there any information when we can expect solution for that issue?

[alex-frankel]

We'll discuss it again in triage this week, but I'm moving this to the 1.0 milestone for now. We'll see if we can get it done sooner.

[enterpriseSoftware]

maximum size of 131072 characters. Please make sure you understand this is characters not bytes. This makes this feature unusable. Any work arounds?

[verschaevesiebe]

Upvote

[qualitas-software]

I hit the same problem today with a swagger import file that was 213,293 bytes.

I found a workaround for large json files, by using loadJsonContent() then wrapping the result with string(..) as follows...

var swagger = string(loadJsonContent('api.swagger.json'))

[enterpriseSoftware]

@qualitas-software thank you for posting that, I needed that.

[verschaevesiebe]

@qualitas-software i see you highlight json, i assume this doesn't work for loadTextContent ? 😄

[qualitas-software]

@verschaevesiebe yeah the original issue is about loadTextContent(). I haven't tried it, but could you put the encoded file into json, use a version of the method above and extract it? Eg- { "encoded": "<base64 encoded file>" } then load it var fileContent = string(loadJsonContent('file.encoded.as.json').encoded) ? Good luck.

[verschaevesiebe]

@qualitas-software nice workaround, would have come up with it myself but I need structure so I've translated all OpenAPI Specs from YAML to JSON and we're now using the loadJsonContent 😄

Goodluck to anyone else running into file size limitations 🤗

[davidbambridge]

loadJsonContent

Nice this save me so much searching.

[Kaloszer]

+1

[alex-frankel]

Turns out that loadJsonContent() does not have this same problem. It seems most, if not all, of the cases where someone would run into this limit is for JSON content. Given that, we are going to close this and recommend you use loadJsonContent() instead.

If we are missing any cases that do not involve JSON content, please let us know and we can revisit.

[slavizh]

@alex-frankel may be for both commands will be best to document the maximum limits of size/characters? If not already, I have not checked.

[johan-burman]

If we are missing any cases that do not involve JSON content, please let us know and we can revisit.

I have several cases of xml files (edifact xsd schemas) which exceeds the text limit by far, so this is absolutely not for just json documents.

Why are there even different size limits for json and text content?

[JayDoubleu]

It would be great to have some transparency on this. Is there some sort of limitation that would prevent this limit to be raised globally?

Without this transparency we might be guessing the reasons for this not being implemented which isn't exactly helpful both for Microsoft nor open-source community.

[alex-frankel]

Why are there even different size limits for json and text content?

@miqm knows better than me, but my guess is we can handle JSON more efficiently because it's the same format as the template.

Thank you for sharing the examples @johan-burman. Which resource types are you using those XML schemas for? I will re-open the issue and make it more specific to non-JSON content.

@JayDoubleu -- hopefully you have seen that we always try to be as transparent as possible. The reason we closed this is because we thought it only needed to cover JSON content. At this point, we will need to take another look and see if we need to prioritize this sooner.

[miqm]

The limit for loadTextContent is a limit for ARM template expression as noted here

As Alex pointed, loadJsonContent leverages ARM's JSON structure to "omit" the bicep compilation and put file directly into Intermediate Language - which is ARM JSON (load is not direct though - any potential ARM Template expression escaping is done to avoid loaded file being processed by ARM runtime).

LoadJsonContent does not enforce any limit - the only limit is 4MB - which is a limit for ARM template. So even if you have a 4MB JSON file your template most probably will fail due to other compiled resources "taking space" in the intermediate ARM file.

AFAIK the expression limit is in ARM Runtime to not process too long expressions. However @anthony-c-martin or @majastrz can give more light on this.

Perhaps a solution would be to give ability to provide some storage account details which could be used as intermediate artifact storage during deployment?

[johan-burman]

Thanks for re-opening the ticket!

Thank you for sharing the examples @johan-burman. Which resource types are you using those XML schemas for? I will re-open the issue and make it more specific to non-JSON content.

The case I'm looking at right now is adding a large edi flatfile schema (xsd) to an integration account. Before this I came across the same limit with an Xslt, but was able to make it fit by removing whitespace.

If this limit needs to stay as-is, maybe there's some other way to achieve the same result, which we are overlooking?

[slavizh]

@alex-frankel one of the biggest cases for loadTextContent is with Microsoft.Insights/workbooks resource. You can have quite big workbooks there and you cannot use loadJsonContent() as it does not take json input but rather escaped json as string as far as I know. It is easier if you save the json generated from workbook and load it to deploy it as you can make small changes to the json without going to Portal. At the same time when you edit in portal you have good option to just copy the json and than you will paste it in your json file for deployment.

[JayDoubleu]

+1 Azure Workbooks is the entire reason I created this issue (To allow creation of Azure Workbooks via bicep)

If I could have an acceptance criteria it would be to allow creation of most azure workbooks via bicep which is currently not possible ? Unless prehaps there's a way to serialize it somehow to base64 and then back into serialized string after loading as json..

[miqm]

For workbooks just use string function on a bicep object.

[slavizh]

@miqm thanks. string(loadJsonContent('workbook.json')) works.

[dansnellgrove]

So I'm trying to load an openapi json file into Microsoft.ApiManagement/service/apis bicep resource and ran into the character limit discussed above. I then set the value: string(loadJsonContent('../schemas/openapi.json')) which gets round the immediate issue but then there is another failure when attempting to deploy it - Unhandled exception. Azure.Deployments.Core.Exceptions.ExpressionException: The language expression literal limit exceeded. Limit: '131072' and actual: '191488'. at Azure.Deployments.Expression.Expressions.JTokenExpression..ctor(String value, TemplateErrorAdditionalInfo additionalInfo) at Bicep.Core.Emit.ExpressionConverter.ConvertExpression(Expression expression). So I'm confused how others have managed to get this working.

[frastrom]

Basically this limitation is making it impossible to automate deployment of end-to-end Logic app flows including XSLT in an integration account. Removing whitespace is only an option for close-to-limit file sizes, but we have much larger maps than that. One workaround would be to use a single-tenant logic app and skip the integration account, but that comes with a lot of other issues (such as the XSLT transform action never fails when the transformation fails, which makes it retry every 5 seconds for all eternity...).

[scschneider]

We are using this to load ADX scripts and hitting the max character limit using loadTextContent. string(loadJsonContent()) is not a workable solution for our usecase.

[devnomadic]

+1 For importing large automation account runbooks & DSC configurations.

[slavizh]

@devnomadic if I am not mistaken you cannot use the above function in runbooks API anyway as that one requires URL link to the runbook published on public place rather providing the text of the runbook.

[devnomadic]

@slavizh There is an API to import draft runbook content directly: replace-content API

For DSC configuration you can pass the script content directly in Bicep using the 'Microsoft.Automation/automationAccounts/configurations' resource. There is an 'embeddedContent' type in the source object & the value you can use the loadTextContent function.

I have used the Az Powershell module in the past to import runbooks and DSC configurations without using a URI for the source. Looking a the Azure Powershell repo & following the code for the Import-AzAutomationRunbook cmdlet it uses the above API to import runbooks.

[slavizh]

@devnomadic The API is for a draft rather for published runbook. Also I do not think this API is even Bicep enabled. Not every ARM API is applicable for Bicep. For example any actions like Start or Stop VMs are like operations rather setting configuration. The draft one seems exactly like that - https://learn.microsoft.com/en-us/azure/templates/microsoft.automation/automationaccounts/runbooks/draft-content?pivots=deployment-language-bicep. Have you tested the API with Bicep and if so do you have example Bicep code that creates/updates runbook via content rather link to http endpoint?

[devnomadic]

@slavizh Thank you for the suggestion. I did try the draft-content resource but it looks like it is not working: https://github.com/Azure/bicep-types-az/issues/1688

[slavizh]

@devnomadic I highly doubt that will be fixed. The API is made like action so it does not support ARM Template/Bicep. Probably will be the same effort to implement creating runbook from content than making that API compliant with Bicep. Overall the runbook API should support that not the draft one as draft is more of API that is used when you are doing development via portal and you update runbooks.

[sdball]

I've run into an aspect of this issue as well. I have a logic app that is serving a deploy-time configured HTTP response that serves up a bespoke front-end web application. Built with a static single file generating framework the HTML exceeds the 131072 character limit. I can manually create a logic app and save the HTML as an HTTP response but I cannot export the logic app as a template nor can I deploy the logic app from Bicep or JSON.

[sachinpachori]

Yes I am having same issues.

[Kaloszer]

Just ran into this issue myself :(, seems that I need to make another workaround

[ngetchell-pi]

Our team ran into this when setting APIM definitions. We minified our json to squeeze ourselves under the character limit.

jq -r tostring .\infrastructure\swagger.json > .\infrastructure\swagger-mini.json

get-content .\infrastructure\swagger.json -raw |
     convertfrom-json |
     ConvertTo-Json -Compress -Depth 100 |
     Out-File .\infrastructure\swagger-mini.json

[sdball]

That'll work great until even your minified JSON is too large.

[ngetchell-pi]

That'll work great until even your minified JSON is too large.

Correct. I would still like to see an increase to the limit.

[antsok]

We have run into this issue, when using a single JSON file to define and deploy all our azure policies that make up the Enterprise Landing Zone guardrails.

This involves loadJsonContent()

var customPolicies = string(loadJsonContent('../../inputs/policies.json'))
var customPoliciesModified1 = replace(customPolicies, "valueOld", "valueNew")
var customPoliciesModified = array(json(customPoliciesModified1))

The error comes from the ARM, not bicep. "Deployment template validation failed: 'The template variable 'customPoliciesModified1' is not valid: The template language expression literal limit exceeded. Limit: '131072' and actual: '131072'. Please see https://aka.ms/arm-functions for usage details"