LennyBEL commented 1 year ago

Bicep version v0.18.4

Describe the bug Deployment via AZ DevOps pipelines suddenly fails on a weird error after changing our .json parameter files to .bicepparam. When reverting back to .json files, the error disappears. We use a AzureCLI@2 task.

/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command . '/home/vsts/work/_temp/azureclitaskscript1688638767862.ps1' WARNING: The configuration value of bicep.use_binary_from_path has been set to 'false'. ERROR: Unhandled exception. System.InvalidOperationException: Sequence contains no elements at System.Linq.ThrowHelper.ThrowNoElementsException() at System.Linq.Enumerable.FirstTSource at Bicep.Core.Workspaces.SourceFileGrouping.TryGetErrorDiagnostic(StatementSyntax statement) at Bicep.Core.Semantics.ModuleSymbol.TryGetSemanticModel(ISemanticModel& semanticModel, ErrorDiagnostic& failureDiagnostic) at Bicep.Core.TypeSystem.DeclaredTypeManager.GetDeclaredModuleType(ModuleDeclarationSyntax module) at Bicep.Core.TypeSystem.DeclaredTypeManager.GetModuleType(ModuleDeclarationSyntax syntax) at Bicep.Core.TypeSystem.DeclaredTypeManager.GetTypeAssignment(SyntaxBase syntax) at Bicep.Core.TypeSystem.DeclaredTypeManager.b__6_0(SyntaxBase key) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey, Func2)

To Reproduce Use .bicepparam files and attempt to deploy it using AZ DevOps pipelines with a AzureCLI@2 task.

Additional context

Absolutely nothing changed other than a switch to .bicepparam, so it's not a misinput, especially since it works fine with .json. Also removed the bicepconfig.

Relevant part of our pipeline: strategy: runOnce: deploy: steps:

checkout: self
task: AzureCLI@2 ${{ if eq(parameters.whatIf, true) }}: displayName: 'Validate bicep template on ${{ parameters.environment }}' ${{ if eq(parameters.whatIf, false) }}: displayName: 'Provision bicep template on ${{ parameters.environment }}' inputs: azureSubscription: ${{ parameters.serviceConnection }} scriptType: 'pscore' scriptLocation: 'inlineScript' inlineScript: | $(theCommand) --name "$(theName)" --mode "$(theMode)" --subscription "${{ parameters.azureSubscription }}" --template-file "$(Build.SourcesDirectory)${{ parameters.templatePath }}" --parameters "$(Build.SourcesDirectory)${{ parameters.parametersPath }}" --resource-group "${{ parameters.azureResourceGroup }}"

OskarKlintrot commented 1 year ago

I also hit this when trying to switch from json to bicepparam. It works just fine locally (W10) but fails on Azures Ubuntu-agents.

Deploying groups still works just fine with bicepparam, this only happens when I try to deploy subs on ubuntu-agents.

Are you trying to deploy group, sub or something else?

anthony-c-martin commented 1 year ago

@Usman0111 would you mind investigating?

LennyBEL commented 1 year ago

Are you trying to deploy group, sub or something else?

We use deployment groups

Usman0111 commented 1 year ago

Are you able to build your .bicepparam file locally using a bicep build-params command?

alex-frankel commented 1 year ago

I would also try running az version and az bicep version. My guess would be one of the Pipeline Agents has not been updated with the latest bits.

OskarKlintrot commented 1 year ago

Are you able to build your .bicepparam file locally using a bicep build-params command?

Locally (W10) everything works just fine, generating .bicepparam, building the .bicepparam, what-if, depoy etc. It only fails on Ubuntu.

I would also try running az version and az bicep version. My guess would be one of the Pipeline Agents has not been updated with the latest bits.

az is 2.49.0 and bicep 0.18.4. It failed consistently for hours and I never got an agent with lower versions.

LennyBEL commented 1 year ago

Are you able to build your .bicepparam file locally using a bicep build-params command?

Yes, locally it will deploy fine. As OskarKlintrot has said, it only fails on an Ubuntu(-latest) DevOps pipeline.

I would also try running az version and az bicep version. My guess would be one of the Pipeline Agents has not been updated with the latest bits.

Same as above, AZ CLI is 2.49.0 and Bicep is 0.18.4.

anthony-c-martin commented 1 year ago

@LennyBEL, @OskarKlintrot would mind trying to repro this with Bicep tracing enabled?

Should be a case of setting the BICEP_TRACING_ENABLED env var to "true" before executing the AzCLI script - e.g. on Linux:

export BICEP_TRACING_ENABLED=true

LennyBEL commented 1 year ago

@LennyBEL, @OskarKlintrot would mind trying to repro this with Bicep tracing enabled?

Should be a case of setting the BICEP_TRACING_ENABLED env var to "true" before executing the AzCLI script - e.g. on Linux:
export BICEP_TRACING_ENABLED=true

Sure can. I've removed parts of the logs (which explains timestamp jumps) to avoid any potentially sensitive data being involved (but that part of the logs was mostly just retrieving the modules) and have obfuscated the rest, but let me know if you need any more logging or information.

https://[pastebin.com/raw/WR0mxC3d](https://pastebin.com/raw/WR0mxC3d)

LennyBEL commented 1 year ago

After posting the above reply, I tried to run the same templates (in what-if mode) but without the registry modules we have (for some resources we use shared modules stored in a Container Registry) and that seemed to work...

The moment I re-added one of the modules, it failed to deploy with the same error as above. Although I should add that the parameters to the registry modules are passed indirectly (so not directly from the bicepparam but from another module within the templates).

It seems to be failing when it begins with "Building semantic model" - maybe there's an issue when building this model for any registry modules?

@OskarKlintrot do you happen to use registry modules too?

anthony-c-martin commented 1 year ago

After posting the above reply, I tried to run the same templates (in what-if mode) but without the registry modules we have (for some resources we use shared modules stored in a Container Registry) and that seemed to work...

The moment I re-added one of the modules, it failed to deploy with the same error as above. Although I should add that the parameters to the registry modules are passed indirectly (so not directly from the bicepparam but from another module within the templates).

It seems to be failing when it begins with "Building semantic model" - maybe there's an issue when building this model for any registry modules?

Thanks for providing the log output + narrowing it down to the registry module, that's really helpful information! It's also interesting to see that things work locally on Windows but not on a build agent (Ubuntu).

OskarKlintrot commented 1 year ago

Yes @LennyBEL, I didn't think of that but yes we do actually. And the pipelines that is working for me when using bicepparam is not using any (private) modules. I'm off on vacation but I don't think the ones that is working uses any modules at all.

swilkodev commented 1 year ago

I ran into this issue today as well on our devops ubuntu agent. I was also able to reproduce locally on windows by doing the following.

create a bicep file which refers to a private module in a registry.
Delete registry cache under %USERPROFILE%\.bicep\br\[name of your registry].azurecr.io
run bicep build-params [param file] and the error should occur.

When tracing is turned on via the env variable, bicep build-params restored the private modules (I could see them in the cache folder), however the same error occured.

client assembly: Azure.Containers.ContainerRegistry
TRACE: Azure-Core: [Informational] Response [8482c92c-d135-48f8-8e46-d1389cce3d8d] 200 OK (00.1s)

TRACE: Restore module br:[redacted].azurecr.io/bicep/operational-insights/workspaces:1.0.1: Succeeded (Elapsed = 2106 ms)
TRACE: Building semantic model for file:///[redacted]/main.bicep (BicepFile)
Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1)
   at Bicep.Core.Workspaces.SourceFileGrouping.TryGetErrorDiagnostic(StatementSyntax statement)
   at Bicep.Core.Semantics.ModuleSymbol.TryGetSemanticModel(ISemanticModel& semanticModel, ErrorDiagnostic& failureDiagnostic)

Running the command again worked without error.

To reproduce

create a bicep file which uses a module from a private registry (example below)

module logAnalyticsWorkspace 'br/MyModules:operational-insights/workspaces:1.0.1' = {
  name: 'log-analytics-aaaaa'
  params: {
    name: 'aaaa'
    location: 'aaaa'
  }
}

create a bicep parameters file (my file only add a using statement, no params)

using 'main.bicep'

delete the registry cache, ensure private module is not cached
run bicep build-params, should result in error
run bicep build-params, should not result in error

In regard to my issue with the ubuntu build agent, I solved the problem by ensuring az bicep build command ran first to restore the private modules. When the modules are restored/cached the bicep build-params command works fine. It seems when they are not the command fails.

For my AzureCLI@2 task I had

bicep build-params dev.parameters.bicepparam
az bicep build --file main.bicep --outdir '$(buildFolder)'

I swapped them around and it worked.

LennyBEL commented 1 year ago

I can confirm that building/restoring the modules manually before running the deployment command (in our case: az deployment group create) seems to fix the issue.

We solved it by adding this right before our deployment group creation:

az bicep restore --file /path/to/bicep --force

Thank you very much @swilkodev

JeroenvdBurg commented 1 year ago

we had the same issue (with private registry modules). Can confirm that is works when you first build the main file and then build parameters.

@swilkodev thanks for supplying the quickfix for now, helped us a lot

maikvandergaag commented 1 year ago

We had the same issue with the private registry. As suggested we have added the two steps:

    az bicep restore --file ${{ parameters.templatefile }} --force
    bicep build-params '${{ parameters.parametersfile }}.bicepparam'

mthemark commented 12 months ago

Similar issue with bicepparam file with AzurePowerShell@5 in DevOps pipeline. Task : Azure PowerShell Version : 5.226.0 Bicep version: 0.21.1+d4acbd2a9f

Simple of creating resource group, converted over to bicepparam from parameters.json

VSCode Bicep Deploy Pane successfully Validated, What-if and Deploy the .bicepparam file

Fails in DevOps pipeline [using same Service Principal that deployed alz-bicep repo using parameters.json files]

with ps1 script line 32 - Path cannot be parsed (The "using" path of the module, also tried side-by-side i.e. same directory with same error)
with ps1 script line 33 - cant find file (seems bicep build-params has not output the file yet)

workaround that seems to work lines 21-25 (with line 34) use ps to create the json file from the output of bicep build-params

screenshot of files used

screenshot of the bicepparam failure

screenshot of workaround success (line 34)

Usman0111 commented 11 months ago

@mthemark please create a discussion topic as your scenario is potentially a usage issue

Usman0111 commented 11 months ago

For those experiencing the module restore issue, it should be fixed with Bicep v0.22 release. Closing the issue for now. Please re-open if the issue persists.

mvromer commented 9 months ago

For what it's worth, I'm still seeing this on Bicep 0.23.1 and Azure CLI 2.55.0 inside my pipeline when I run az bicep generate-params. I can trigger the same error locally if I delete my local cache of modules I've restored from my private registry. In that case, the first run generates the following stack trace:

$ az bicep generate-params --file ./main.bicep --stdout

Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1)
   at Bicep.Core.Workspaces.SourceFileGrouping.TryGetSourceFile(IArtifactReferenceSyntax foreignTemplateReference)
   at Bicep.Core.Semantics.SemanticModelHelper.TryGetSourceFile(ISourceFileLookup sourceFileLookup, IArtifactReferenceSyntax reference)
   at Bicep.Core.Semantics.SemanticModelHelper.TryGetModelForArtifactReference(ISourceFileLookup sourceFileLookup, IArtifactReferenceSyntax reference, ISemanticModelLookup semanticModelLookup)
   at Bicep.Core.Semantics.DeclarationVisitor.GetImportSourceModel(CompileTimeImportDeclarationSyntax syntax)
   at Bicep.Core.Semantics.DeclarationVisitor.VisitCompileTimeImportDeclarationSyntax(CompileTimeImportDeclarationSyntax syntax)
   at Bicep.Core.Syntax.CompileTimeImportDeclarationSyntax.Accept(ISyntaxVisitor visitor)
   at Bicep.Core.Syntax.SyntaxVisitor.VisitInternal(SyntaxBase node)
   at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node)
   at Bicep.Core.Syntax.SyntaxVisitor.VisitNodes(IEnumerable`1 nodes)
   at Bicep.Core.Syntax.AstVisitor.VisitProgramSyntax(ProgramSyntax syntax)
   at Bicep.Core.Semantics.DeclarationVisitor.VisitProgramSyntax(ProgramSyntax syntax)
   at Bicep.Core.Syntax.ProgramSyntax.Accept(ISyntaxVisitor visitor)
   at Bicep.Core.Syntax.SyntaxVisitor.VisitInternal(SyntaxBase node)
   at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node)
   at Bicep.Core.Semantics.DeclarationVisitor.GetDeclarations(INamespaceProvider namespaceProvider, IFeatureProvider features, ISourceFileLookup sourceFileLookup, ISemanticModelLookup modelLookup, ResourceScope targetScope, BicepSourceFile sourceFile, ISymbolContext symbolContext)
   at Bicep.Core.Semantics.Binder..ctor(INamespaceProvider namespaceProvider, IFeatureProvider features, ISourceFileLookup sourceFileLookup, ISemanticModelLookup modelLookup, BicepSourceFile sourceFile, ISymbolContext symbolContext)
   at Bicep.Core.Semantics.SemanticModel..ctor(Compilation compilation, BicepSourceFile sourceFile, IEnvironment environment, IFileResolver fileResolver, IBicepAnalyzer linterAnalyzer, RootConfiguration configuration, IFeatureProvider features)
   at Bicep.Core.Semantics.Compilation.CreateSemanticModel(BicepSourceFile bicepFile)
   at Bicep.Core.Semantics.Compilation.<>c__DisplayClass5_1.<.ctor>b__2()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper, Boolean)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Bicep.Core.Semantics.Compilation.GetSemanticModel(ISourceFile sourceFile)
   at Bicep.Core.Semantics.Compilation.GetSemanticModel[T](ISourceFile sourceFile)
   at Bicep.Core.Semantics.Compilation.GetSemanticModel(BicepSourceFile bicepFile)
   at Bicep.Core.Semantics.Compilation.<GetAllDiagnosticsByBicepFile>b__19_1(BicepSourceFile bicepFile)
   at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1, MutationInput, KeyCollisionBehavior )
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1, Boolean)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 )
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1, Func`2, Func`2, IEqualityComparer`1 , IEqualityComparer`1 )
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1, Func`2, Func`2)
   at Bicep.Core.Semantics.Compilation.GetAllDiagnosticsByBicepFile()
   at Bicep.Cli.Services.CompilationService.LogDiagnostics(Compilation compilation)
   at Bicep.Cli.Services.CompilationService.CompileAsync(String inputPath, Boolean skipRestore, Workspace workspace, Action`1 validateFunc)
   at Bicep.Cli.Commands.GenerateParametersFileCommand.RunAsync(GenerateParametersFileArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

If I immediately rerun the command, it succeeds and writes the generated parameters to standard out. It might be unrelated, but it seems like the issue I'm seeing with generate-params might be the same that happened with build-params. I couldn't find a commit or PR that seemed related to whatever fix went out in 0.22.

For what it's worth, I see a similar error when manually running az bicep restore. In that case, the stack trace is this:

ERROR: Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1)
   at Bicep.Core.Workspaces.SourceFileGrouping.TryGetSourceFile(IArtifactReferenceSyntax foreignTemplateReference)
   at Bicep.Cli.Services.CompilationService.<GetModuleRestoreDiagnosticsByBicepFile>g__DiagnosticForModule|14_0(SourceFileGrouping grouping, IArtifactReferenceSyntax moduleDeclaration)
   at Bicep.Cli.Services.CompilationService.<GetModuleRestoreDiagnosticsByBicepFile>g__GetDiagnosticsForModulesToRestore|14_1(SourceFileGrouping grouping, ImmutableHashSet`1 originalArtifactsToRestore)+MoveNext()
   at System.Linq.Lookup`2.Create[TSource](IEnumerable`1, Func`2, Func`2, IEqualityComparer`1)
   at System.Linq.Enumerable.ToLookup[TSource,TKey,TElement](IEnumerable`1, Func`2, Func`2, IEqualityComparer`1 )
   at System.Linq.Enumerable.ToLookup[TSource,TKey,TElement](IEnumerable`1, Func`2, Func`2)
   at Bicep.Cli.Services.CompilationService.GetModuleRestoreDiagnosticsByBicepFile(SourceFileGrouping sourceFileGrouping, ImmutableHashSet`1 originalModulesToRestore, Boolean forceModulesRestore)
   at Bicep.Cli.Services.CompilationService.RestoreAsync(String inputPath, Boolean forceModulesRestore)
   at Bicep.Cli.Commands.RestoreCommand.RunAsync(RestoreArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

Running the restore again immediately afterward succeeds.

fvilches17 commented 8 months ago

Probably still an issue with AzureDevOps pipelines AzureCLI@2 task version 2.55.0 *

Local ✅

az deployment group create `
  --resource-group myrg `
  --name local `
  --template-file main.bicep `
  --parameters foo.bicepparam

Azure DevOps Workaround

When running the same but using in AzureDevOps pipelines workaround is two-step

build .bicepparam file
deploy with JSON param file

# Build foo.bicepparam -> foo.json
- pwsh: az bicep build-params --file foo.bicepparam
  displayName: Build Bicep Parameters

# Deploy to AZ Group, use foo.json
- task: AzureCLI@2
  name: AzGroupDeployment
  inputs:
    azureSubscription: mySub
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
      az deployment group create `
        --resource-group myrg `
        --name someDeploymentName `
        --template-file main.bicep `
        --parameters foo.json

OskarKlintrot commented 8 months ago

@fvilches17 It's the Bicep version that matters, can you log that as well?

fvilches17 commented 8 months ago

@fvilches17 It's the Bicep version that matters, can you log that as well?

@OskarKlintrot good point.

For more context,

Ran az bicep version on my pc Ran - script: az bicep version on pipeline

Bicep CLI version 0.24.24 (5646341b0c) my PC Bicep CLI version 0.24.24 (5646341b0c) pipeline (Microsoft ubuntu-latest agent) - script task Bicep CLI version 0.24.24 (5646341b0c) pipeline (Microsoft ubuntu-latest agent) - @AzureCLI@2 task

anthony-c-martin commented 8 months ago

@fvilches17 please could you open a new issue for this, to make sure it gets attention?

fvilches17 commented 8 months ago

@fvilches17 please could you open a new issue for this, to make sure it gets attention?

@anthony-c-martin I tried again and it's working. Clearly something on my part. Please disregard

Azure / bicep

DevOps deployment fails after switch to bicepparam #11142

Local ✅

Azure DevOps Workaround