Allow configuring volume behavior when using Aspire.Hosting.Testing

[DamianEdwards]

Currently, when testing AppHost projects that include container resources configured to use volumes, the same volumes are used in the test as that are used in normal development. This isn't great because the test isn't as isolated from the normal running of the AppHost as it could be and could mutate the data in the volume.

We should allow for configuring the behavior of volumes when using Aspire.Hosting.Testing, and by default remove volumes from all container resources under test so that they are better isolated. Volume behavior should be configurable globally for all resources, or on a per resource basis (overriding the configured global behavior), by way of new API to support both scenarios.

[mitchdenny]

I wonder if this is something that we can do in conjunction with DCP where we some how nominate a volume/bindmount as being volatile. So we still generate the volumes and bind mount but when DCP tears down the the container it also deletes the volume/bindmount contents.

/cc @karolz-ms @danegsta

[karolz-ms]

Interesting idea. Currently volumes are never deleted automatically, but we could have an opt-in flag for cleaning them up, sort of like Containers have their persistent flag. Let us know if you want to pursue this idea and we can make it happen in DCP

[DamianEdwards]

I think that's a good idea. It's likely required to make the randomization of volume names in tests reasonable.

[mitchdenny]

@karolz-ms I'll assign this to you for the DCP side first. Once you've got something that we can set in the spec for volumes/bindmounts then you can assign back to me and I'll work on the app model hook up.

[karolz-ms]

@mitchdenny Roger, makes sense. @DamianEdwards I do not think this will fit into 8.1. Scheduling for 8.2 tentatively, please make a noise if you disagree.

[DamianEdwards]

@karolz-ms sounds fair.

What do folks think of doing a click-stop improvement in 8.1 of simply removing all volumes when using DistributedApplicationTestingBuilder so that tests are better isolated from non-tests scenarios? The DCP side isn't required for that. We could choose to make this behavior the new default or make it opt-in/mutable via a new extension method void SetContainerResourceVolumeBehavior(this IDistributedApplicationTestingBuilder, VolumeBehavior behavior)?

[karolz-ms]

We can try an experiment with the DeleteResourcesOnShutdown option that we have: https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting/Dcp/DcpHostService.cs#L75 But AFAIK this is not getting much use as the default is to rely on DCP to do the cleanup, so I do not know if we have enough time in 8.1 to test this alternate way of cleaning things up.

[DamianEdwards]

Sorry I wasn't clear, I mean simply removing mount annotations from resources when under test.

[karolz-ms]

Oh, sounds like I completely misunderstood the ask here, sorry. You want the tests to use ephemeral volumes instead of persistent volumes, correct? Makes perfect sense. If I am not mistaken, this should work today with DCP. That is, you can have a Container resource that has VolumeMount in the spec, but no corresponding ContainerVolume objects. This should result in a creation of ephemeral volume with the lifetime corresponding to the lifetime of the Container object. LMK if that does not work for you.

@mitchdenny If my understanding of the ask is correct, there should be no work on the DCP orchestrator side to satisfy it.

[DamianEdwards]

Ah that's wonderful to hear, thanks! So now it comes down to timing/resourcing/scheduling/sequencing 😄

Do we push to implement support for two volume behavior under test modes in 8.1: no volumes, and ephemeral volumes; or do we just for the initial click-stop of a "no volumes" behavior in 8.1 and introduce the ephemeral volumes option later, given that the ephemeral option is likely to be more work.

Thoughts @mitchdenny @davidfowl?