MSA.BuildingBlocks.CosmosDbMigrations package

[kanilsz]

It would be great to execute Cosmos db migrations with console application. Necessary to provide basic operations:

1. On database

• [x] Delete container
• [x] Recreate container with new partition key
• [x] Clone container
• [x] Add indexing policy
• [x] Create container
• [x] Replace indexing policy
• [x] Switch to another container for operations

2. On container

• [x] Upsert items
• [x] Remove items by query
• [x] Add property to items on any nesting level
• [x] Remove property from items on any nesting level
• [x] Switch to another container for operations
• [x] Switch to another database and container for operations

Also performed that you could inject any logger from ILogger for RUs cost logs.

[Marusyk]

great idea. let's start doing it one by one.

[Marusyk]

One more requirement. Currently, the migration process starts on the startup of the application. We could have another option - run it as a sidecar. I'll try to explain what I mean.

There is sidecar pattern: https://www.baeldung.com/ops/kubernetes-pods-sidecar-containers and https://www.baeldung.com/ops/kubernetes-init-containers and https://andrewlock.net/deploying-asp-net-core-applications-to-kubernetes-part-8-running-database-migrations-using-jobs-and-init-containers/ where you can run 1+ containers for k8s deployment. It could fix the issue with Race Conditions (which we fixed with @kanilsz a long time ago). Sidecar init container runs before starting our application instances.

and there are 2 strategies for implementation:

1. The separated docker image for containers - very interesting approach but has extra complexity and CI pipeline is required for such migration-image
2. Run the same application with arguments. It is similar to what we have for now but we should not start migration at startup instead add some arguments. in this case, we don't need one more image. If you run the app with arguments dotnet run --migrate then only migration runs. so, we can run it as a init sidecar. Then run the same app but without any arguments dotnet run.

somewhere in the Program.cs, Main mathod

if (args.Length >= 1 && args[0] == "migrate")
    Migrate();
else
    Run();

private static void Migrate()
{
    Log.Information("Database migration started");

   CosmosDbConfiguration cosmosConfig = configuration.GetSection("CosmosDbGlobal").Get<CosmosDbConfiguration>();

   services.AddMigrations(cosmosConfig)
      .Run();

    Log.Information("Database migration completed");
}

So, the application has 2 modes for startup: 1) run migrations and close 2) run the app when we deploy it to k8s, first we run the app in the migration mode as an init sidecar, once it has been done - run the app as usual.

[Marusyk]

glad to hear that 😉 I'd propose to have 1 generic issue like this with the name "[package name] package" and use it as an epic to define the requirements. and then the more specific issues can be created for each requirement or unit of work (1PR = 1 issue)

[Marusyk]

There is a Migration tool for MongoDb and I see it is very similar to what we have. I like our approach even more https://bitbucket.org/i_am_a_kernel/mongodbmigrations/src/master/

[Marusyk]

one more: instead of ARepository.Init(database); BRepository.Init(database); CRepository.Init(database); we could have a factory patter implemented.

[kanilsz]

I have more thoughts about something like a building pattern for migration executions. I see it as: (For all items)

var result = MakeMigrations(items)
.AddProperty("UA", object) -- could add nestings
.RemoveProperty("ru") -- full remove with nestings
.UpdateProperty("us", "eu")
.Execute()

I see that it could decrease the rus of executions. Because only one replacement could be done. But it will cost the memory of the executor. We could improve and add everything that is necessary with some extension.

[kanilsz]

Cosmos DB possibly could have more than a million records. So for an executor it will be a loooong time to execute the migrations. Moreover, it could be crashed during the execution. What I propose:

1. Provide migration by some chunk (1000 items)
2. If it falls we tell to executer that item 600 has failed to migrate.
3. Let him fix it by himself.
4. Start the execution from 600.

For me, it looks very hard to implement but I think the use case is real.

[kanilsz]

It would be great to have integration tests that will validate and ensure that the migration was made successfully.

There are two possible options that I know.

1. TestContainers https://testcontainers.com/modules/cosmodb/
2. Up the cosmos db container by ourselves (not sure that the issue with certificates that we have is resolved)

Probably some unit tests for a future builder will ensure that operations were made successfully.