"existing" functionality for modules

[johnnyreilly]

Is your feature request related to a problem? Please describe.

We have a Bicep module registry which we use to wrap Bicep templates, such that they are delivered to match our organisations collectively agreed security posture. So to create a storage account we'd do something like this:

module deadLetterStorageAccount 'br:icebox.azurecr.io/bicep/ice/providers/storage/storage-accounts:v1.0' = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
  params: {
    tags: tags
    location: location
    storageAccountName: opticalDeadLetterStorageAccountName
    skuName: 'Standard_ZRS'
  }
}

If we want to reference the resource created by that module later elsewhere, we can do that using existing like so:

resource opticalDeadLetterStorageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: opticalDeadLetterStorageAccountName
  scope: resourceGroup(opticalStorageResourceGroupName)
}

But this is not a great developer experience as:

1. We have flipped from module syntax to resource syntax which is confusing
2. We have to go and look up what resource is used in the module to be able to write the module statement
3. If that module is updated to use a different resource, we have to find out the new one and then copy/pasta that across all our resource ... existing references

Describe the solution you'd like

What would be delightful is if there was some kind of existing syntax for modules. Imagine if, instead of:

resource opticalDeadLetterStorageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: opticalDeadLetterStorageAccountName
  scope: resourceGroup(opticalStorageResourceGroupName)
}

we could do:

module deadLetterStorageAccount 'br:icebox.azurecr.io/bicep/ice/providers/storage/storage-accounts:v1.0' existing = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
  params: {
    tags: tags
    location: location
    storageAccountName: opticalDeadLetterStorageAccountName
    skuName: 'Standard_ZRS'
  }
}

This would make upgrades much more straightforward / intuitive.

There's obviously a whole bunch of complexity in what I'm suggesting - but something like this would be amazing. Something that allows you to acquire the resources created by a module directly rather than indirectly. What do you think?

[jeskew]

Are you looking to have the resources within the module converted into existing resources (i.e., does it not matter if the resources were deployed via the module) or is it the module itself that you would want to refer back to?

For the latter, you can sort of do this today, since Bicep deploys modules as nested templates. This does require switching to resource syntax, so instead of

module deadLetterStorageAccount 'br:icebox.azurecr.io/bicep/ice/providers/storage/storage-accounts:v1.0' existing = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
  params: {
    tags: tags
    location: location
    storageAccountName: opticalDeadLetterStorageAccountName
    skuName: 'Standard_ZRS'
  }
}

you could use

resource deadLetterStorageAccount 'Microsoft.Resources/deployments@2021-04-01' existing = {
  name: 'lens-dead-letter-storage-account-${branchHash}'
}

Outputs would be nested under properties, e.g., deadLetterStorageAccount.properties.outputs..., so it's not a drop-in replacement.

[johnnyreilly]

Are you looking to have the resources within the module converted into existing resources

Kind of this I think. It would be tremendous if we could have a way to reference the resources created by a module without explicitly having to know what exact resources they were.

In my head I think of the current situation as analogous to having a class which performs operations on your behalf which are abstracted away from you, but the class is presently unable to offer an API that allows reacquisition of same.

That's probably slightly confusing. But yeah, the thing you're suggesting as the other option is what we're already doing and it feels unintuitive.

Also, we run a Bicep registry in our organisation, so there's necessarily extra steps required to discover what the module is actually provisioning

[anthony-c-martin]

This is an interesting suggestion! Just to clarify - the module you're trying to establish a reference to is part of the same overall deployment, right? (e.g. you're not trying to access a module deployed by a totally independent deployment)

If we extended #2246 to be able to pass module references as params/outputs, would that solve the problem for you?

[johnnyreilly]

Just to clarify - the module you're trying to establish a reference to is part of the same overall deployment, right?

Yes I believe so. To be totally clear, in our present case we have:

main.bicep
storage.bicep
compute.bicep

main.bicep references storage.bicep and compute.bicep. storage.bicep creates a resource that we reference in compute.bicep as well. We achieve that presently using the resource ... existing syntax.

If we extended #2246 to be able to pass module references as params/outputs, would that solve the problem for you?

I think so - if that means you are able to pass around the reference to the resource then yes.

Having the ability to reference something from an independent deployment would also be amazing. But just module references would be a massive win. As a rough estimate, module references would cover 60% of my resource ... existing use cases. The other 40% being independent deployments.

[jeskew]

Referencing modules from independent deployments would be trickier, as ARM would look up the deployment by name and the Bicep compiler would look up type information from the referenced file path. This introduces the possibility that Bicep will block expressions that would resolve correctly during a deployment (and allow statements that fail at deploy time) due to drift between what outputs existed on the template when it was last deployed vs. what outputs exist in the template at compile time. There's also the possibility that the same name was used by a completely different deployment, in which case Bicep's understanding of what outputs would be available could be totally wrong.