dotnet / aspire

An opinionated, cloud ready stack for building observable, production ready, distributed applications in .NET

https://learn.microsoft.com/dotnet/aspire

MIT License

3.37k stars 348 forks source link

[tests] Workload tests #2689

Open radical opened 3 months ago

radical commented 3 months ago

These would be tests that exercise the workload, like:

Template tests

[ ] test for aspire-apphost, and aspire-servicedefaults - take the ApiService, and Web project from the aspire-starter template and then add apphost, and servicedefaults from templates
[ ] with no workload installed, a project with $(IsAspireHost)=true would fail to build due to missing aspire workload
[x] basic templates can build and run
[x] Starter template can run (WIP)
[x] Starter template's included test project works (WIP)
[x] create+build+run starter template, and run some basic checks including checking the dashboard (WIP)

Dashboard

[x] verify all the resources show up (WIP)

Others

[ ] Test per published nuget, to use the nuget and perform some basic validation
[ ] manifest publishing works with dotnet run
[x] default https works on linux
[x] ASPIRE_ALLOW_UNSECURED_TRANSPORT

Future:

radical commented 3 months ago

cc @joperezr

eerhardt commented 3 months ago

If I was dreaming, we would also have tests that publish an app to Azure. But that is going to take a lot of infrastructure.

with no workload installed, a project with $(IsAspireHost)=true would fail to build due to missing aspire workload

I'd put this tests pretty low on the priority list.

mitchdenny commented 2 months ago

I'd love to have tests that work on Azure as well. In the Azure SDK repo they have the ability to do something like this: /azp run net - servicex - tests

This is a dedicated pipeline that only people with appropriate permissions can trigger which results in provisioning resources in Azure. Because the identity that provisions the resources is an Owner of the sub it uses a dedicated sub for this purpose, and the identity is restricted in what it can access.

You also need to have a bot that goes through and deletes resources.

@radical if you are looking at this I recommend going and having a chat to @weshaggard about what their current state of the art is on the Azure SDK side.

radical commented 2 months ago

This sounds interesting. thoughts, @eerhardt ?

mitchdenny commented 2 months ago

I think the test matrix is going to be fairly interesting. For example, do you want to test Aspire across multiple clouds and regions. API versions can sometimes take time to rollout so if we adopt a new version of an ARM API via CDK and publish our packages it's possible that we could block someone from upgrading their app in a region which tends to receive API updates late - similar for different clouds. China and USGov (not to mention others) can lag behind significantly.

We probably will also want to look at real world upgrade scenarios. This probably means that in our repo we'll need small free standing applications that have been built using various versions of the Aspire stack and then have scripts that sequentially deploy the updates to make sure we don't break upgrade scenarios.

These would be a bit different to the playgrounds, and unit tests aren't good candidates for this because the "earlier versions" won't be using the latest versions of packages but would be tied to the latest servicing release for that generation.

eerhardt commented 2 months ago

What is the medium-to-long-term vision for where the Azure Hosting code lives? If we intend to move it to https://github.com/Azure/azure-sdk-for-net/, it may make sense to wait until then to hook it up to automated tests.

On the Components side, my goal is to eventually move them to the Azure repo, so they can ship and service along with the Azure SDK packages.