Add Azure Event Grid Aspire Component

eerhardt commented 10 months ago

We should have an Azure Event Grid component. See

WhitWaldo commented 4 months ago

Following up here, please also add support not just for the Event Grid namespace capabilities, but also for Event Grid System Topics (which don't support namespace).

danmoseley commented 4 months ago

Is this something you're interested in contributing to @WhitWaldo ?

WhitWaldo commented 4 months ago

Is this something you're interested in contributing to @WhitWaldo ?

@danmoseley I'd be happy to, except for two factors.

1) I submitted a PR for adding support for Front Door/DNS back in February that hasn't been accepted nor has it received any feedback about why it might not be compatible with the project design. That gives me pause on taking the time to contribute whole resources if they'll just linger in the PR queue indefinitely. 2) I don't see that Azure.Provisioning has any support yet for any of the Event Grid SKUs. Based on the new Event Hub project taking a dependency on that, should I read into that being a necessary dependency for resource projects going forward (this and the other one)?

davidfowl commented 4 months ago

I submitted a PR for adding support for https://github.com/dotnet/aspire/pull/2526 back in February that hasn't been accepted nor has it received any feedback about why it might not be compatible with the project design. That gives me pause on taking the time to contribute whole resources if they'll just linger in the PR queue indefinitely.

This is my fault. I'll leave a comment on the PR as I don't think we're ready to merge these types of azure resources into aspire.

I don't see that Azure.Provisioning has any support yet for any of the Event Grid SKUs. Based on the new Event Hub project taking a dependency on that, should I read into that being a necessary dependency for resource projects going forward (this and the other one)?

Yes.

WhitWaldo commented 4 months ago

@davidfowl While I had difficulty building/testing the Azure SDK on my machine (undoubtedly because of all the package reference magic happening in that solution), I submitted a PR for Azure Batch this morning as it's got support in Azure.ResourceManager.Batch, but doesn't yet have an Azure.Provisioning.Batch package. Should that get accepted without much difficulty, that'll be the impetus for me to also build out the same for Azure.Provisioning.EventGrid, Azure.Provisioning.Dns and Azure.Provisioning.FrontDoor so I can write/update the appropriate PRs for each of those asks as well.

davidfowl commented 4 months ago

Like I said in that PR, there were 2 things stopping us from merging. The lack of provisioning library and other compute services that we have no integration for. We're more interested in event grid than azure front door and dns at the moment. There are other fundamental things we need to flesh out before we merge those types of dependencies.

That said, you should be able to use the Azure batch provisioning library with the AddAzureConstruct primitive to unblock integration between those 2 systems. First class support as an aspire resource will be delayed while we figure out the patterns. There's other important work in flight to better understand that.

WhitWaldo commented 4 months ago

I would love to get to the point that I can deploy my entire solution via Aspire and utilize all the built-in referencing and secret management so I can entirely avoid their respective SDKs (especially having to double up on them), but as you might have gathered, my own solution touches on a bunch of different Azure services.

I get that DNS isn't the most interesting or revolutionary of Azure services out there, but it's fundamental to certificate verification and issuance without going the LettuceEncrypt route of requiring hosted apps for the same - as soon as you've got a better pattern in mind, I'd be happy to put in the time to build out support as again, they're all pieces I've got to use myself and I'd love to be able to provision and use them more easily.

davidfowl commented 4 months ago

We have the same goals, but we are trying to make sure the platform enables scenarios we don't envision. We use examples to extract required features that will enable many more scenarios than we plan out of the box. It's early and there are some more fundamental pieces we need to flesh out before the end to end comes together in a way that you can use like this. For new scenarios we don't have a solution for, the most useful thing you can do to help us is to build examples using the primitives we have. That will help us tease apart what the required features from the "nice to have" ones.

At this point in the project, we're less interested in adding extra azure resources (because eventually we plan to generate all of them, well the azure sdk team will 😄) and more interested in the features we would need to build to enable a slew of new scenarios. As an example, if you told me you wanted to add a vnet resource, I would say we need to spend time looking at how to model networking in the app host instead.

PS: That motives projects like this https://www.nuget.org/packages/Achieve.Aspire.AzureProvisioning which help the team out a lot.

davidfowl commented 4 months ago

Back on topic:

The event grid resource, we got this working in the latest azd https://github.com/Azure/azure-dev/issues/3839 so I'm pretty excited to see an end-to-end working when deployed to azure.

To make local dev work, I am hoping we can integrate https://github.com/dotnet/aspire/issues/813 in a pretty seamless way.

WhitWaldo commented 4 months ago

Jumping off topic with you for a moment: It's something that's been brought up with the Bicep folks for several years now, but I see a couple of problems with taking an ARM dependency straight off an auto-generated resource with little else in the way of overhead management:

Some resources (vnets are an excellent example) don't support patch updates and only allow full-resource replacements
Several resources (RBAC comes to mind, but there are a great many of them reported on this thread) will mark as completed in deployments even though they haven't actually propagated to all the places they're intended to be used). I reported Azure Data Explorer as another one that reports complete for ARM purposes, but goes into some maintenance mode and isn't eligible for further ARM updates until that completes.

In other words, ARM (and Bicep) doesn't yet have a retry mechanism and the complete deployment state cannot necessarily be trusted. The ARM management SDKs (Azure.ResourceManager.*) don't accommodate this as they're auto-generated and leave it to the consumer to deal with. I'd recommend as part of factoring into longer-term plans either getting the Azure teams to fix ARM completion reporting on their end (ideal), waiting to see if Bicep adds something to address this or add something into Aspire itself to accommodate this. Personally, I have a bunch of delay PowerShell deployment scripts in my own Bicep project that intentionally add waiting periods with a check loop to accommodate this.

Do you have a list anywhere of known "unknowns" in terms of design areas you'd welcome suggestions of handling or are you more looking for external solutions to crop up to identify perceived shortcomings like Achieve.Aspire.AzureProvisioning? I know storing declarative secrets in Key Vault is one I raised before, but I don't know that I've seen it called out in a specific list anywhere.

On topic: I'm not super familiar with dev tunnels, but please, please, please avoid the issue that plagued local Service Fabric development - namely that because the local SF cluster ran under a different user account, it wasn't able to utilize Azure.Identity for anything but certificate-based authentication to Entra SPs which was a real pain. It's nice to be able to lean on chained token credentials and specify a UAI client ID and a VS credential fallback and have development experience otherwise Just Work without having different authentication in different deployment environments.

davidfowl commented 4 months ago

I'd recommend as part of factoring into longer-term plans either getting the Azure teams to fix ARM completion reporting on their end (ideal), waiting to see if Bicep adds something to https://github.com/Azure/bicep/issues/4023#issuecomment-1451243315 or add something into Aspire itself to accommodate this.

We've been working with these teams, so we'll continue to do this and will raise appropriate issues. No promises though, they have a LOT on their plate.

Do you have a list anywhere of known "unknowns" in terms of design areas you'd welcome suggestions of handling or are you more looking for external solutions to crop up to identify perceived shortcomings like Achieve.Aspire.AzureProvisioning?

The latter. There are lots of existing gaps in the deployment experience, so much so that it really only works well for dev/test scenarios unless you infra synth and tweak various things. Our priorities will be around making it so that you never have to change the infra synth-ed content and that you can model things via the apphost project. For this, we want to rely on the Azure.Provisioning libraries.

We also expect several rounds of iteration on the design with prototypes before we merge anything into the product. That's why we have lots of experimental branches and code in samples to flesh out scenarios. That's how most of these new integrations are built.

For deployment specifically, we will be focused on making what we already have modifiable without manually editing bicep and yaml. This branch is where we're looking at moving compute infrastructure generation into Aspire.Hosting.Azure.

Once we get here then we'll need to revisit how to handle role assignments and managed identities. I'll allow us to do a better job a least privilege role assignments.

I know storing declarative secrets in Key Vault is one I raised before, but I don't know that I've seen it called out in a specific list anywhere.

Lots of interesting issues are tracked here https://github.com/dotnet/aspire/issues?q=is%3Aopen+label%3Aarea-app-model+label%3Aazure. I'm not sure yet how many we will get to. Right now we're working on plans for post GA and I can tell you these aren't the highest thing on the list.

On topic: I'm not super familiar with dev tunnels, but please, please, please avoid the issue that plagued local Service Fabric development - namely that because the local SF cluster ran under a different user account, it wasn't able to utilize Azure.Identity for anything but certificate-based authentication to Entra SPs which was a real pain. It's nice to be able to lean on chained token credentials and specify a UAI client ID and a VS credential fallback and have development experience otherwise Just Work without having different authentication in different deployment environments.

Our bias is making sure the developer doesn't need to make code changes between local and deployed environments.

dotnet / aspire

Add Azure Event Grid Aspire Component #788