Open galvesribeiro opened 2 months ago
cc @karolz-ms @danegsta
Hmm. @galvesribeiro can you reproduce the problem without Aspire? I mean something like this
mkdir thedir
docker run --mount type=bind,src=thedir,dst=/dst yourimage
docker stop containerid;docker rm containerid
rm -rf thedir
Aspire does not really do anything more fancy than this for bind mounts.
Hello!
Found the actual exceptions on the AppHost:
fail: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
could not create the container {"Container": {"name":"localstack-wahkpvus"}, "Reconciliation": 4, "error": "docker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Error response from daemon: invalid mount config for type \"bind\": bind source path does not exist: /<redacted>/.data/localstack\n'"}
fail: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
could not create the container {"Container": {"name":"postgres-ngkrhpap"}, "Reconciliation": 9, "error": "docker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Error response from daemon: invalid mount config for type \"bind\": bind source path does not exist: /<redacted>/.data/postgres\n'"}
fail: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
could not create the container {"Container": {"name":"minio-vydntvdp"}, "Reconciliation": 10, "error": "docker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Error response from daemon: invalid mount config for type \"bind\": bind source path does not exist: /<redacted>/.data/minio\n'"}
Then if I run this:
docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword \
-v /<redacted>/.data/postgres:/var/lib/postgresql/data postgres
It work just fine.
If at the same time I mix the manual docker run
with the Aspire run, Aspire always fail while docker run
always work.
In other words, that only happens when I'm running the AppHost.
Something seems to be weird on how DCP is interacting with Docker API and the mounts. With -v
it works 100% of the time. Not sure about --mount
and its parameters.
Just for reference, this directory is created before the resources are even built:
var builder = DistributedApplication.CreateBuilder(args);
var usernameParameter = builder.AddParameter("Postgres-User");
var passwordParameter = builder.AddParameter("Postgres-Password");
var localDataVolumePath = builder.AddParameter("Data-Path");
var localPostgresVolumePath = builder.AddParameter("Postgres-Path");
var staticResourcePath = builder.AddParameter("StaticResource-Path");
var minioUsernameParameter = builder.AddParameter("Minio-User");
var minioPasswordParameter = builder.AddParameter("Minio-Password");
var localMinioVolumePath = builder.AddParameter("Minio-Path");
var localLocalStackVolumePath = builder.AddParameter("LocalStack-Path");
DataDirectoryFactory.CreateDataDirectoryIfNotExists(localDataVolumePath.Resource.Value);
var localStack = builder.AddLocalStack("localstack", port: 4566)
.WithDataBindMount(Path.Combine(localDataVolumePath.Resource.Value, localLocalStackVolumePath.Resource.Value));
var postgresCluster = builder
.AddPostgres("postgres", usernameParameter, passwordParameter, port: 5432)
.WithHealthCheck()
.WithBindMount(
Path.Combine(localDataVolumePath.Resource.Value, localPostgresVolumePath.Resource.Value),
"/var/lib/postgresql/data"
);
// Rest of the resources here
And the directory is created as simple as this:
public static class DataDirectoryFactory
{
public static void CreateDataDirectoryIfNotExists(string dataDirectory)
{
Directory.CreateDirectory(dataDirectory);
var subdirectories = new[] { "postgres", "minio", "static-resources", "localstack" };
foreach (var subdirectory in subdirectories)
{
var path = Path.Combine(dataDirectory, subdirectory);
Directory.CreateDirectory(path);
}
}
}
Also it is not just one specific container. All of those which have the mounts are failing until we restart Docker.
@galvesribeiro Based on the behavior you're describing, I'm suspicious that CreateDataDirectoryIfNotExists
isn't always actually recreating the expected bind mount source folder. The main difference between -v
and --mount
is that, if given a bind mount source folder that doesn't exist, -v
will create the missing folder on the host system but --mount
will throw the error you're seeing in the AppHost logs. Can you add some logging and/or breakpoints to your AppHost to check what folder paths are being created (and that they exist on disk before the AppHost creates the container resources)?
The latest version of the Aspire workload includes updated logic to automatically create missing bind mount folders the same way -v
would have done. Can you run dotnet workload list
to see what workload version you're on?
Also, as another thought, it looks like Docker Desktop on MacOS supports multiple different bind mount implementations; there's VirtioFS
, gRPC FUSE
, and osxfs (Legacy)
listed as options. If you check the general settings in Docker Desktop, what implementation are you using?
@galvesribeiro Based on the behavior you're describing, I'm suspicious that
CreateDataDirectoryIfNotExists
isn't always actually recreating the expected bind mount source folder. The main difference between-v
and--mount
is that, if given a bind mount source folder that doesn't exist,-v
will create the missing folder on the host system but--mount
will throw the error you're seeing in the AppHost logs. Can you add some logging and/or breakpoints to your AppHost to check what folder paths are being created (and that they exist on disk before the AppHost creates the container resources)?The latest version of the Aspire workload includes updated logic to automatically create missing bind mount folders the same way
-v
would have done. Can you rundotnet workload list
to see what workload version you're on?
Sorry for the delay.
dotnet workload list
Installed Workload Id Manifest Version Installation Source
--------------------------------------------------------------------
aspire 8.2.0/8.0.100 SDK 8.0.300
I did checked with a breakpoint and the code as you see it indeed recreate the folders every time. It is also called very early so the container resources which uses it are not even defined yet:
@danegsta in regards to the file share implementation, I've never changed it and I'm using the default:
The thing is, that behavior doesn't exist when I'm doing the -v
outside Aspire which makes it weird as you are saying DCP is not doing anything weird. Tried multiple ways to reproduce it with Docker alone without Aspire but I couldn't. It always works.
@galvesribeiro, do you have a repro project you can share? We haven't been able to reproduce this and haven't seen it reported elsewhere.
@dbreshears sure. Bear with me and I'll setup one and share here.
Is there an existing issue for this?
Describe the bug
I've noticed that if on your AppHost you create some local directory then map it on one or more container resources like for example databases in order to persist the data, there is something hanging on either Aspire or Docker. If we stop the debug session, delete the directory, and start again, even tho the directories are created, somehow the containers aren't able to see it again and fail to start. A docker restart is required after deleting such directories before it works again.
The reason why someone would delete those directories is that we are either testing migrations on the database or just want to have a clean slate to start working on it. However, with this current issue, we always have to remember and restart Docker.
Expected Behavior
After deleting the directory and restart the Aspire process, the directories should be usable again.
Steps To Reproduce
Directory.CreateDirectory(path)
;WithBindMount()
orWithDataBindMount()
and map that created directory somewhere into the container;Exceptions (if any)
No response
.NET Version info
Anything else?
All latest Aspire and ASP.Net versions were used.