search

dotnet / aspire

Tools, templates, and packages to accelerate building observable, production-ready apps
https://learn.microsoft.com/dotnet/aspire
MIT License
3.95k stars 485 forks source link

[Aspire.Hosting.Azure.CosmosDB] Allow setting partition count and persistence #5136

Closed hansmbakker closed 2 months ago

hansmbakker commented 4 months ago

Is there an existing issue for this?

Is your feature request related to a problem? Please describe the problem.

Aspire.Hosting.Azure.CosmosDB component allows running CosmosDB as emulator which spins up the cosmosdb emulator docker image.

Now, the container is started with the default partition count (11) which makes the CosmosDB emulator slow to become available and take more memory. For me, 4 partitions is already enough.

Also, I believe the data is not persisted across runs.

Describe the solution you'd like

Extension methods to set the number of partitions and to easily configure persistence. Something like:

var cosmos = builder.AddAzureCosmosDB("CosmosDb")
    .RunAsEmulator(containerBuilder =>
        {
            containerBuilder.WithDataVolume()
                            .WithPartitionCount(4);
        });

See https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/persist-data-volumes for other Aspire components.

Additional context

The emulator supports a few environment variables that govern the behavior of the emulator: Description
AZURE_COSMOS_EMULATOR_PARTITION_COUNT (Optional) Specify the number of partitions to use.
AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE (Optional) Enable data persistence between emulator runs.
AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE (Optional) Override the emulator's default IP address.
hansmbakker commented 4 months ago

I found it is possible to do something like this

var cosmos = builder.AddAzureCosmosDB("CosmosDb")
                    .AddDatabase("mydatabase");

if (builder.Environment.IsDevelopment())
{
    cosmos.RunAsEmulator(containerBuilder =>
            {
                containerBuilder.WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "3")
                                .WithEnvironment("AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE", "true")
                                .WithBindMount("VolumeMount.AppHost-cosmos-data", "/tmp/cosmos/appdata");
            })
          .WithHttpsEndpoint(8081, 8081, "emulator-port");
}

⚠️ However, adding the .WithBindMount("VolumeMount.AppHost-cosmos-data", "/tmp/cosmos/appdata"); crashes the emulator.

Container logs of the crash:

2024-08-02T15:06:29.2400000 3e5710d8d56acc08796dfaaffd25a0721a6b3ce55008b25fca98044030259859

2024-08-02T15:06:30.2840000 3e5710d8d56acc08796dfaaffd25a0721a6b3ce55008b25fca98044030259859

2024-08-02T15:06:32.1043786 This is an evaluation version.  There are [72] days left in the evaluation period.
2024-08-02T15:07:35.7916901 ** ERROR: [AppLoader] Failed to load LSA: 0xc0070102
2024-08-02T15:07:35.7935346 AppLoader: Exiting with status=0xc0070102
2024-08-02T15:07:35.7939911 This program has encountered a fatal error and cannot continue running at Fri Aug  2 13:07:35 2024
2024-08-02T15:07:35.7940141 The following diagnostic information is available:
2024-08-02T15:07:35.7940190 
2024-08-02T15:07:35.7940222          Reason: OS Fatal Error (0x00000006)
2024-08-02T15:07:35.7940254         Message: Termination of \SystemRoot\system32\AppLoader.exe was due to fatal error 0xC0000001
2024-08-02T15:07:35.7940293         Address: 0x3fff81253014
2024-08-02T15:07:35.7940325     Stack Trace:
2024-08-02T15:07:35.7940356                  file://package1/windows/system32/sqlpal.dll+0x0000000000298F1C
2024-08-02T15:07:35.7940770                  file://package1/windows/system32/sqlpal.dll+0x0000000000296EC1
2024-08-02T15:07:35.7940803                  file://package1/windows/system32/sqlpal.dll+0x0000000000253180
2024-08-02T15:07:35.7940835                  file://package1/windows/system32/sqlpal.dll+0x0000000000253014
2024-08-02T15:07:35.7940866                  file://package1/windows/system32/sqlpal.dll+0x0000000000251E6B
2024-08-02T15:07:35.7940898                  file://package1/windows/system32/sqlpal.dll+0x00000000002ACD9B
2024-08-02T15:07:35.7940934                  file://package1/windows/system32/sqlpal.dll+0x0000000000394658
2024-08-02T15:07:35.7940968         Process: 24 - cosmosdb-emulator
2024-08-02T15:07:35.7940999          Thread: 59 (application thread 0x88)
2024-08-02T15:07:35.7941030     Instance Id: ecf991a4-a507-47ef-a674-578e598819db
2024-08-02T15:07:35.7941060        Crash Id: d1351578-e9ad-41bd-b439-882c8f4c9af9
2024-08-02T15:07:35.7941104     Build stamp: (null)
2024-08-02T15:07:35.7941134    Distribution: Ubuntu 20.04.6 LTS
2024-08-02T15:07:35.7941165      Processors: 4
2024-08-02T15:07:35.7941196    Total Memory: 16776032256 bytes
2024-08-02T15:07:35.7941226       Timestamp: Fri Aug  2 13:07:35 2024
2024-08-02T15:07:35.7941256      Last errno: -34938881
2024-08-02T15:07:35.7941287 Last errno text: Unknown error -34938881
2024-08-02T15:07:35.8050011 *********** PAL PANIC CORE DUMP GENERATION FAILED **********
2024-08-02T15:07:35.8050241 Unable to locate handle-crash.sh. Error: File: signals.cpp:483 [Status: 0xC0000034 Object name not found errno = 0x2(2) No such file or directory]
2024-08-02T15:07:35.8051143 Executing: /usr/local/bin/cosmos/handle-crash.sh with parameters
2024-08-02T15:07:35.8051166      handle-crash.sh
2024-08-02T15:07:35.8051179      /usr/local/bin/cosmos/cosmosdb-emulator
2024-08-02T15:07:35.8051279      24
2024-08-02T15:07:35.8051293      /usr/local/bin/cosmos
2024-08-02T15:07:35.8051305      /tmp/cosmos/appdata/log/
2024-08-02T15:07:35.8051317      
2024-08-02T15:07:35.8051328      ecf991a4-a507-47ef-a674-578e598819db
2024-08-02T15:07:35.8051341      d1351578-e9ad-41bd-b439-882c8f4c9af9
2024-08-02T15:07:35.8051374      
2024-08-02T15:07:35.8052387 *********** PANIC CORE DUMP GENERATION FAILED **********
2024-08-02T15:07:35.8052412 Attempt to launch handle-crash.sh failed.
2024-08-02T15:07:35.8097573 This program has encountered a fatal error and cannot continue running at Fri Aug  2 13:07:35 2024
2024-08-02T15:07:35.8097980 The following diagnostic information is available:
2024-08-02T15:07:35.8098000 
2024-08-02T15:07:35.8098012          Reason: Host Extension RTL_ASSERT (0x00000003)
2024-08-02T15:07:35.8098024          Status: STATUS_WAIT_2 (0x00000002)
2024-08-02T15:07:35.8098040         Message: !killTheTarget
2024-08-02T15:07:35.8098051           Stack:
2024-08-02T15:07:35.8098063                  SP               IP               Function
2024-08-02T15:07:35.8098075                  ---------------- ---------------- ----------------
2024-08-02T15:07:35.8098090                  00007f0a9fed3df0 000055c9444f6bfa <unknown>
2024-08-02T15:07:35.8098106                  00007f0a9fed4e50 000055c9444f65cf <unknown>
2024-08-02T15:07:35.8098118                  00007f0a9fed50b0 000055c944460b66 <unknown>
2024-08-02T15:07:35.8098129                  00007f0a9fed50e0 000055c9444fa0e4 <unknown>
2024-08-02T15:07:35.8098142                  00007f0a9fed6310 000055c9444f97a9 <unknown>
2024-08-02T15:07:35.8098154                  00007f0a9fed6440 00007f0aa202d609 start_thread+0xd9
2024-08-02T15:07:35.8098165                  00007f0a9fed6500 00007f0aa1aae353 clone+0x43
2024-08-02T15:07:35.8098177 
2024-08-02T15:07:35.8098187         Process: 22 - cosmosdb-emulator
2024-08-02T15:07:35.8098199          Thread: 23
2024-08-02T15:07:35.8098210     Instance Id: ecf991a4-a507-47ef-a674-578e598819db
2024-08-02T15:07:35.8098221        Crash Id: d1351578-e9ad-41bd-b439-882c8f4c9af9
2024-08-02T15:07:35.8098233     Build stamp: (null)
2024-08-02T15:07:35.8098244    Distribution: Ubuntu 20.04.6 LTS
2024-08-02T15:07:35.8098255      Processors: 4
2024-08-02T15:07:35.8098266    Total Memory: 16776032256 bytes
2024-08-02T15:07:35.8098277       Timestamp: Fri Aug  2 13:07:35 2024
2024-08-02T15:07:35.8098288      Last errno: 2
2024-08-02T15:07:35.8098300 Last errno text: No such file or directory
2024-08-02T15:07:35.9061306 Aborted
hansmbakker commented 4 months ago

I created a start for the partition count in https://github.com/dotnet/aspire/pull/5157

However I am not sure what to do about the persistence as I cannot get it to work - I would need help here.

hansmbakker commented 4 months ago

I found https://github.com/dotnet/aspire/issues/3295 for the data persistence and included it in the PR