Squash migrations

[rowanmiller]

It would be good to have the ability to squash several migrations into a single file to help reduce the number of files in a project.

We probably want to keep track of the original list of migration names so that we can reason about this when targeting an existing database that the original migrations were applied to in their un-squashed form.

[ajcvickers]

@MoishyS It is not unreasonable to argue that everything needed should be in the migration, but this is currently not always the case. I don't have the specific cases immediately at hand, and they can also change from release to release for the same migration.

[bricelam]

One example is for table rebuilds on SQLite. The migration just contains operations like AlterColumn, AddForeignKey, AddCheckConstraint, etc. but SQLite doesn't support those operations natively, so we have to use the backing EF model to re-create the entire table.

On SQL Server, we occasionally use it to rebuild indexes.

[achikhv]

Also came to the point where we need to squash migrations. Build time and memory consumption became unacceptable.

We use idempotent migrations to update our customer's databases, and now my plan is:

1. Create full idempotent script that creates latest up-to-date database.
2. Delete all migrations and create new one.
3. Delete all content of the new migration and execute idempotent script in it.

It seems that this steps does the trick, and we can generate new idempotent script that is compatible with all versions of databases.

I would like to ask, are there any drawbacks in this plan? Will it be compatible with squash architecture that is in mind of EF Core team?

[jeff-pang]

Also came to the point where we need to squash migrations. Build time and memory consumption became unacceptable.

We use idempotent migrations to update our customer's databases, and now my is:

1. Create full idempotent script that creates latest up-to-date database.

2. Delete all migrations and create new one.

3. Delete all content of the new migration and execute idempotent script in it.

It seems that this steps does the trick, and we can generate new idempotent script that is compatible with all versions of databases.

I would like to ask, are there any drawbacks in this plan? Will it be compatible with squash architecture that is in mind of EF Core team?

This comment perfectly captures how important to get this asap

1. Many of us are dealing with large migrations especially in a rapid development environment
2. Even if we think of a possible workaround we have no way to know if our workaround is safe

[achikhv]

Also came to the point where we need to squash migrations. Build time and memory consumption became unacceptable.

We use idempotent migrations to update our customer's databases, and now my plan is:

1. Create full idempotent script that creates latest up-to-date database.
2. Delete all migrations and create new one.
3. Delete all content of the new migration and execute idempotent script in it.

It seems that this steps does the trick, and we can generate new idempotent script that is compatible with all versions of databases.

I would like to ask, are there any drawbacks in this plan? Will it be compatible with squash architecture that is in mind of EF Core team?

To answer myself there is major drawback in postgres idempotent script. After all migrations squashed we need to create new idempotent script. SQL Server's script is good, but postgres created nested `DO $EF$ ... END $EF$; statements that caused syntax error when executed. So nesting idempotent scripts is not a great idea right now.

[roji]

@achikhv if you're encountering a syntax error with idempotent migrations on PostgreSQL, please open an issue on the PG provider repo with a minimal, runnable code sample.

[TsengSR]

@jeff-pang

This comment perfectly captures how important to get this asap

1. Many of us are dealing with large migrations especially in a rapid development environment
2. Even if we think of a possible workaround we have no way to know if our workaround is safe

Well there is a viable workaround, which can be applied easily between major version releases (or even minors or arbitrary, but imho its too much hassle for minor releases)

Let's say, for version 1.x you have your regular migrations and for 2.x you decide you want to "squash" it.

1. When you first start tagging 2.x you run dotnet ef database scripts to export all of your migrations into a single SQL file.
2. Optionally you also add an empty migration before that naming it Initial_2_x, more on later. Remember the name of your old Initial migration.
3. Then you drop all your migrations and create a new one from scratch (dotnet ef migrations add Initial_2_x)

4. If you did a empty Initial_2_x migration in 1.x release, then make sure your new Initial_2_x is named exactly the same as your previous version (including the leading timestamp).

   The reason you may want to do that, is so that for new installations of your software the new Initial_2_x migration is used but for old upgrades it's skipped (EF Core compares the name of your migration if its in the __EFMigrationHistory table and if its there it will skip the migration)

5. Now, when your application starts, you need to check if the Initial_1_x exists, if so apply your exported migration script from step 1, because its an old installation and you need to bring the database up to the latest schema, before 2.x starts.

   If Initial_1_x isn't in the migration table, skip it (its a new installation) and EF Core will create the new schema with Initial_2_x instead.

Works best between major releases. Keep the 1.x migration till 3.x and drop it, or replace it with a new one which includes 1.x and 2.x migration steps, if you want support migrations for more than 1 major upgrade.

This gives you the advantage, that even if a customer skips migrations, you can still upgrade the schema to the latest version before the "squash". And since the previous migrations are outsourced as sql files, they also do not need to be compiled anymore, just ship it with your release and apply it on the applications first start if required (upgrade installation only).

@ajcvickers The key here lies in, naming the last (dummy) migration of the previous release be named exactly same as the new initial migration. Maybe that's something one could implement as an extension to ef-core cli tool?

Or maybe even extending the __EFMigrationHistory table with a major migration version? The likes of, when you "reset" your migrations, it gets bumped up by 1 and the dotnet ef migration squash creates a single release-1.sql file, that will be applied if the initial migration of the release exists and then you, next time you do it you get release-2.sql etc. which are applied in order till the newest one. On fresh installations these are simply skipped and the initial of the latest release is just applied.

Seems like a reasonable compromise, since after tagging a new version ("squashing"), downgrades won't be required anyways, most companies don't support version downgrades without resetting the data. But even if, it would still be possible although manually (roll back latest "squash", then previous until the version you are looking for)

[TsengSR]

Because, let's face it. The possibility to downgrade is only used in two cases

1. Local development, to roll back a change that wasn't good or to squash it before the final commit.
2. Once rolled out to production or the customer, reverting only happens by an compensation migration (remove field or table X)
3. When a customer wants to downgrade to an older version, because they are having an issue with the current one that can't be fixed quickly enough or can't afford to wait for an fix

First case is a non-case, as it never makes it to the main branch. Last one, may happen rarely, but shouldn't be an issue if the snapshots are "erased" with every major or minor release, since you'd rolll back the migration scripts back to the version you want to use and the version itself already has all the snappshots (removed earlier in the past when you squashed it)

[MikaelEliasson]

At Bokio we went for a mitigation to this issue where we placed our migrations in a separate project. But unlike the official documentation for this we were able to completely remove the reference to the migrations project from our app. We instead have a dedicated entry point project for running the migrations.

The benefits of this approach: 1) In dev we can unload the whole migrations project in VS. Which saves both a lot of memory, compile time and makes the searches in VS work better. 2) The migrations are not included in our production builds meaning a smaller artifact and less memory consumption when the app is running.

Point 1) is achievable with the approach recommended here https://learn.microsoft.com/en-us/ef/core/managing-schemas/migrations/projects?tabs=dotnet-core-cli

But to get the benefit of 2) we use a structure like this. I'll see if I can blog this later. But if you look at the dependencies you can see that the only project that refers to the migrations is a small helper project we use for adding and applying migrations and also generating migration scripts.

PS. We also squah migrations regularily. But with this approach we can do it less often at least.

[ErikEJ]

@MikaelEliasson This is great. You could even use a .slnf to filter out the migrations project.

[titobf]

Hi @MikaelEliasson. That's nice. How are then the migrations applied? (Who runs the MigrationHelper console app?) Thanks for sharing your solution.

[MikaelEliasson]

Hi @MikaelEliasson. That's nice. How are then the migrations applied? (Who runs the MigrationHelper console app?) Thanks for sharing your solution.

In dev I as a developers apply them from the CLI just as usual. I have to change the command slightly though to point out the migrations project if I run them from MigrationHelper or the startup project if I run them from Migrations. We have some yarn shortcuts for doing things like this though so in day to day work I don't see the difference.

In our CI/CD pipelines it works like before. We just made the same change to the commands as above.

[ajcvickers]

Note for triage: we should consider documenting this approach: https://github.com/dotnet/efcore/issues/2174#issuecomment-1483833787

[jasekiw]

@ajcvickers I believe that comment https://github.com/dotnet/efcore/issues/2174#issuecomment-1483833787 is related to my question here: https://github.com/dotnet/efcore/issues/30510

[begerard]

@MikaelEliasson That's an interesting approach, but if your Web project don't reference your Migrations project, you don't auto apply the migrations on a dev start (ie. dbContext.Database.Migrate();)? When another developer add a migration, he have to tell you to manually run the migrations?

[TsengSR]

@MikaelEliasson That's an interesting approach, but if your Web project don't reference your Migrations project, you don't auto apply the migrations on a dev start (ie. dbContext.Database.Migrate();)? When another developer add a migration, he have to tell you to manually run the migrations?

Technically you can do this as part of the build/deployment pipeline or as a sidecar container if you use docker/containers

[ErikEJ]

@begerard you should never use dbContext.Database.Migrate();)

[begerard]

@MikaelEliasson That's an interesting approach, but if your Web project don't reference your Migrations project, you don't auto apply the migrations on a dev start (ie. dbContext.Database.Migrate();)? When another developer add a migration, he have to tell you to manually run the migrations?

Technically you can do this as part of the build/deployment pipeline or as a sidecar container if you use docker/containers

I use this only in development ( if (app.Environment.IsDevelopment()) ), the CI do use the bundle (which is a sidecar indeed).

@begerard you should never use dbContext.Database.Migrate();)

Isn't this the recommended practice for keeping a local dev database up to date? (I don't use docker locally)

[ajcvickers]

See https://github.com/dotnet/EntityFramework.Docs/issues/4304

[titobf]

@begerard you should never use dbContext.Database.Migrate();)

Hi @ErikEJ. Can you briefly explain why it should never be used? Thanks

[roji]

@titobf more information on the shortcomings of Migrate() is available in our docs.

[begerard]

more information on the shortcomings of Migrate() is available in our docs.

Thanks @roji but while the .Migrate() is rightfully discouraged in production, is there better alternatives for local development? Maybe some docs helping configuring a dev env? I never found official resources on that.

[roji]

I'd guess the simplest way would be to simply use the CLI: dotnet ef database update.

[ajcvickers]

@begerard To add to what Shay said, I think it's reasonable to use .Migrate() in test environments, if that works for you. Usually, I find it easier to use EnsureDeleted and EnsureCreated. The point is that having the database automatically migrate when the application starts has important drawbacks, especially in a multi-user environment where it happens when the first user makes a request.

[MichelZ]

I guess the punted-for-6.0 label can be removed from this one

[roji]

@MichelZ we keep those on as a historical record for how many releases we punted on it, that can sometimes help us prioritize better.

[ronnyek]

I've used the delete the migrations folder and created a new migration that effectively bundles everything up into one... but I'm trying to have migrations between releases both up and down directions.

Eg, I've got versioned releases 1.0.0, 1.0.1. I want to upgrade forward is easy, I can just run all migrations and my db will be upgraded, but if I then want to roll back all those changes that happened between 1.0.0 and 1.0.1, how do I do that?

I feel like that implies that as far as released code is concerned, migrations would be 1.0.0, 1.0.1 etc, but for development, it might be 1.0.0, , but at the end of the day, I'd like to leave 1.0.0 in tact, and bundle those two migrations into a 1.0.1 migration.

I feel like migrations are generally very good, but I'm very confused about how people who bundle a version of their db with a service, and how they deal with a downgrade when a service is downgraded? Just deploy old code, and run migrate and it'll migrate to the older version of the db? (because the migrations assembly wont have those changes in the newer code?)

[pdevito3]

TIL this has been floating around for 9 years. I definitely could have missed it, but I’ve never seen it on any of the .NET roadmap plans even for consideration which is surprising given the number of upvotes.

This is easily one of my biggest pain points I’ve just kind of lived with. It's obviously a hard problem, but I hope this makes it into one of the upcoming releases as it will be a huge value add and a great nod to promoting project longevity in the .NET ecosystem.

[yepeekai]

I read it all!!! In our case, as it was mention previously here, the designer.cs files are huge and there is one per migration which means A LOT. We had a problem with azure build job where building the project containing the migrations would take an hour... It has negative impact in visual studio on dev machines too. I think those files should be representative of the associated .cs file: those should represent the increment instead of the complete model. So a reference to the previous migration it is based on and then it should only include the changes. Another solution (from a guy that doesn't know a lot about internal ef code) would be to have those designer.cs files compiled on demand instead of building them all ahead of time assuming most of the time, they are not needed, it would speed up most use cases.

[yepeekai]

out of the 8 most popular issues, this one is in third position and it is the only one that is not "consider for current release"

[jasekiw]

@ajcvickers I would like to ping you since I know that the planning for ef 9 has probably started. I know this is a really difficult task. Let us know if there is anything we can do to help.

[ajcvickers]

@jasekiw Unfortunately, the person who wrote all the Migrations code has now left the team. That isn't to say we won't still get to this at some point in the future, but I think it's unlikely to get done soon.

[yepeekai]

A part of me is sad... but I am, as always, grateful for this project transparency. thanks.

[jasekiw]

@ajcvickers Thanks for the update! I really appreciate the transparency.

[voroninp]

@ajcvickers so you are a team of 4 now?

You, Shay, Andriy and Brice?

[ronnyek]

I feel like this is becoming the single most outstanding "missing or broken" features for me. The team has done an amazing job adding things we've needed to make efcore core component of our software.

Managing migrations over time has been really the only thing that has been problematic... or at least added lots of confusion as to how to deal with long term management of migrations over time best.

Is this something efcore team would be open to PR's on? Has any research or planning been done as far as the direction the team/framework intends to go with it?

[AndriySvyryd]

You, Shay, Andriy and Brice?

Arthur, Shay, Maurycy and me

[ghosttie]

@jasekiw Unfortunately, the person who wrote all the Migrations code has now left the team. That isn't to say we won't still get to this at some point in the future, but I think it's unlikely to get done soon.

I find this really concerning - Microsoft is one of the biggest companies in the world, and yet there are so few people working on this feature that one person leaving can stall development.

From my point of view EF Core is an important product and migrations are an important feature - my company's products rely on these things - but it seems like they're not very important to Microsoft.

[titobf]

One good approach would be to have an "optional" switch to generate the migration directly on sql and without the designer file. For example, now I'm doing this manually. I add a migration (Add-Migration) and then use Script-Migration to generate the sql, drop it into the Up method (using migrationBuilder.Sql();) replacing the existing code, then copy the attributes DbContext and Migration from the designer file, and finally delete the designer file. This can be also done for the Down method. This way the migration becomes really lightweight. The disadvantage is that you lock on the db engine and compatibility level, but it would be optional.

[TsengSR]

The disadvantage is that you lock on the db engine and compatibility level, but it would be optional.

Migrations already lock you into a db and db engine, as of right now you already have to create multiple migrations for each database engine you intend to support, ideally in different assemblies

[ronnyek]

@ajcvickers Is there any opportunity where someone could potentially provide ideas/ submit PR's for this functionality? I wouldn't mind beginning to look into this... It's a massive problem we've run into time and time again, and wouldn't mind contributing some time and doing some research, if I thought it was something that might be evaluated for inclusion

[ajcvickers]

@ronnyek Yes, we're open to ideas. In particular, we are open radical ideas that don't involve this feature being implemented, since it is difficult to do right. For example, the idea @titobf posted is interesting.

Things to consider:

• This could be something that deals with existing migrations, but this may be difficult because custom migrations are difficult to handle in an automated way.
• Instead, we could implement a new flavor of migrations which doesn't leave a lot of artifacts in its wake.
  • This system might have limitations. For example, it might be up-only.
  • This system should not have a backing model for each migration. This means that a migration cannot depend on the current model, which is relatively common today.

We were talking about this on the team last week, so it was interesting to see @titobf's idea.

The team is unlikely to have time to spend a lot of time in the near future, but we will pay attention to ideas posted here and give feedback. You may need to be patient; we are more busy than usual this year.

[ronnyek]

Forgive me for perhaps missing something... but validating my understanding here. (not talking custom migration steps here) Make changes to context/entities Create Migration (creates migration cs files for up/down) Apply Migration (runs up(), and updates __EFMigrationHistory with the name of the latest applied migration)

My particular needs are basically collapsing many migrations into a release migration for a specific version of our software. While we'd potentially want to be able to roll back to the last version, we may not need granular rollback for everything that happened before last version was released.

Seems like you could effectively have a way of marking a particular migration as a snapshot, and then say when you want to publish next version of your app... some way of basically removing all the migrations back to last snapshot (code, and efmigrationhistory... not current schema/data), and regenerating the migration from current model until now also marking the current migration as a snapshot.

Am I over simplifying?

I'm not sure how crazy custom migrations would need to work. So far our needs have been that we need to seed stuff from time to time, but never had the need (YET) to do things like an actual serious data migration where data gets moved to other tables, or updated with other values or anything.

[TomGathercole]

@ronnyek Yes, we're open to ideas. In particular, we are open radical ideas that don't involve this feature being implemented, since it is difficult to do right. For example, the idea @titobf posted is interesting.

Things to consider:

* This could be something that deals with existing migrations, but this may be difficult because custom migrations are difficult to handle in an automated way.

* Instead, we could implement a new flavor of migrations which doesn't leave a lot of artifacts in its wake.

  * This system might have limitations. For example, it might be up-only.
  * This system should not have a backing model for each migration. This means that a migration cannot depend on the current model, which is relatively common today.

We were talking about this on the team last week, so it was interesting to see @titobf's idea.

The team is unlikely to have time to spend a lot of time in the near future, but we will pay attention to ideas posted here and give feedback. You may need to be patient; we are more busy than usual this year.

So we already have a process scripted to do this squash, but it feels super hacky. I've included the full script below, but the nastiest points are:

1. We have to time travel with git to generate a new snapshot from a particular point in time.
2. We have to do string manipulation to make a new migration look like the old InitialMigration so that the tooling doesn't try to re-apply it.
3. We have to do more string manipulation to add a trigger to the initial migration.

Now that last part is probably solved by This Issue, or more generally by not customizing anything in the migration that isn't strictly related to migrating data from A to B. Also, I can't really imagine any general solution that could possibly solve that without just jamming all the migrations into one file.

For the other two points, I think these could potentially be solved by having some special handling around initial migrations. Honestly, I'm not entirely sure how this would work but potentially something like:

• Have some project level flag to opt into auto-generating an initial migration.
• This would always be generated with the same name (e.g. 00000000000000_Init).
• If this migration is found in the history (or maybe if the __EFMigrationsHistory table exists full stop), then do nothing.
• Otherwise:
  • Find the first migration sequentially
  • Generate a 'default' migration based on the model snapshot of this migration

I don't actually think that last part would work, because the snapshot is the state after that migration is applied, but perhaps someone smarter than me can figure out some way to make it work? I've only just realised it doesn't work, otherwise I maybe wouldn't have bothered typing those thoughts up...

Anyhow, that feels like it might be nice because it lets us randomly delete some series of early migrations, and things will 'just work', but you guys don't have to offer a guarantee that customized scripts will be handled in any way.

Messy migration squash script, to illustrate the hoops we're jumping through!

Note, this is less opinionated version of our actual script, so this might not work properly. Hopefully it illustrates the broad strokes anyway.

Function First-Migration {
    Get-ChildItem $migrationFolder | Sort Basename | Select -First 1 -ExpandProperty Basename
}

Function Get-Class {    
    Param ($migration)
    $migration -replace '^[^_]*', ''
}

Function Rewrite-Migration {
    Param ($suffix)

    $source = "$migrationFolder\$newMigration$suffix"

    $content = (Get-Content -Raw $source) `
        -replace $newMigration,$initialMigration `
        -replace (Get-Class $newMigration),(Get-Class $initialMigration) `
        -replace "(\s+}\s+protected override void Down)","$footer`$1" `

    Remove-Item $source

    return $content
}

Function Save-Migration {
    Param ($content, $suffix)

    $target = "$migrationFolder\$initialMigration$suffix"
    $content | Set-Content $target
}

Function Remove-IfExists {
    Param ($filesToRemove)

    ForEach ($f in $filesToRemove) {
        If (Test-Path $f) {
            Remove-Item $f
        }
    }
}

# 4 week sprints means that this covers two 28 day sprints.
# Technically we should only need one sprint, but this gives some contingency.
$days = 60

# Triggers cannot be managed in EF core yet, so we need to add these to the migration 'manually'
$footer = @" 
    migrationBuilder.Sql("CREATE OR ALTER TRIGGER MyTrigger ...");
"@

$migrationFolder = ".\MyProject\Migrations"

# Create a new branch for the job, updated to the latest main
git checkout main
git pull
git checkout -B "$jobNumber-squash-migrations"

# Get the name of the first migration file
$initialMigration = First-Migration

# Reset branch to the first migration created more than $days ago
$hash = git log --before="$days days ago" -1 --pretty=format:"%h" $migrationFolder
git reset --hard $hash

# Make a note of all the migrations which existed at this point, then delete them
# We want to delete the snapshot now, but ensure that we don't delete it a second time later on
$migrationsPattern = "$migrationFolder\*.cs"
$migrationsToDelete = Get-ChildItem $migrationsPattern -Exclude "*ContextModelSnapshot.cs"
Remove-Item $migrationsPattern

# Create a new initial migration
dotnet ef migrations add "$($jobNumber)_Squash"

# Get the name of the newly added migration
$newMigration = First-Migration

# Read out the new migration, make any replacements, then store in a varable
$rewrittenMigration = Rewrite-Migration ".cs"
$rewrittenDesigner = Rewrite-Migration ".Designer.cs"

# Reset back to main, and reapply the changes
git reset --hard main

Remove-IfExists $migrationsToDelete
Save-Migration $rewrittenMigration ".cs"
Save-Migration $rewrittenDesigner ".Designer.cs"

# Commit changes
git add .
git commit -m "Ref #$($jobNumber): Squash $stack migrations"

[ronnyek]

@TomGathercole I'm somewhat curious as to what role the git stuff here is providing. Is this simply as a mechanism to rollback migrations to an earlier state? Is this just some automation you are doing to auto collapse migrations every <time interval> ?

It seems to me like if you have a sort of order all the migrations were executed (whether that was by .cs file name) you could provide the last legitimate migration step, and collapse everything between that one.

Also seems like application of the migration may be weird since the db would have record of migrations that have been applied, but code/efcore cli tools wouldn't have anyway to tell what those migrations were.

[TomGathercole]

@ronnyek The purpose of the git stuff is to get to a point in history where the model generated from code matches some given 'initial migration' state. The main criteria is that all migration which make up this must have been applied to all environments which will ever be migrated going forward.

There's definitely a fair bit of weirdness here. So far we haven't had any problems with it, since the__EFMigrationsHistory doesn't contain muchh aside from the names of the migrations. It would be nice to have some official-ish solution so that what we're doing (or something that achieves the same benefits) could be truly safe to rely on.

I think you might be on to something in regards to collapsing migrations. We could take the model in the .Designer.cs file for some last-known-applied-everywhere migration and re-generate the code based migration using this model. I don't think there's anything in the CLI tooling to do this, but there's probably something in the underlying library to generate part of a ScaffoldedMigration from an IRelationalModel

[deanvr]

Y'all are over thinking this... the problem statement is that on a real world project that spans years the migrations folder can get rather large and us poor dev folks end up manually cleaning up the folder by moving files to an archive folder. Any new dev often needs to get a DB backup and start from there... We need to be able to generate an initial migration from which a dev can use this to build up a full schema based off the current model.

We need a command e.g. Add-Initial-Migration which runs the migrations and intentionally runs the model compare against an 'empty' schema, it then generates a migration class which could be run to generate the full schema upto that point. If the EFMigrationsHistory table already contains migrations it will never run on update-database command, if for example you are a new dev and you run update database command and there are no existing EFMigrationsHistory records or table it can run and generate the schema in the database from that migration class.

Sounds easy peasy... ;)

Other option is some poor soul goes through all the migrations and literally combine all the ups, and all the downs into one migration :) Ohh couldn't we just get a code generator to do this mmmmm

[TomGathercole]

Y'all are over thinking this... the problem statement is that on a real world project that spans years the migrations folder can get rather large and us poor dev folks end up manually cleaning up the folder by moving files to an archive folder. Any new dev often needs to get a DB backup and start from there... We need to be able to generate an initial migration from which a dev can use this to build up a full schema based off the current model.

We need a command e.g. Add-Initial-Migration which runs the migrations and intentionally runs the model compare against an 'empty' schema, it then generates a migration class which could be run to generate the full schema upto that point. If the EFMigrationsHistory table already contains migrations it will never run on update-database command, if for example you are a new dev and you run update database command and there are no existing EFMigrationsHistory records or table it can run and generate the schema in the database from that migration class.

Sounds easy peasy... ;)

Other option is some poor soul goes through all the migrations and literally combine all the ups, and all the downs into one migration :) Ohh couldn't we just get a code generator to do this mmmmm

This is essentially what we're doing, but the trouble is that there's no safe time to just delete all migrations and start again. At any given time, we'll usually have ~10 migrations pending deployment in a 2-weekly release which we cannot squash until they're deployed to every production environment.

I would absolutely be in favour of any solution that totally drops any manual migration tweaks though - I think that's going to be far too hard to deal with in a way that provides any real benefits.

[douglasg14b]

This would be helpful when doing development work.

I may be developing against a clone of staging, and making changes that generate migrations. I may make these changes incrementally, and running the application necessitates that I apply a migration for the changes made. The result is that while in development I may generate multiple migrations, and need to squash them all down to 1 by the end to clear the dev clutter.

This is a tedious, error prone, process when done manually. An automated way to squash multiple migrations into 1 would be a nice QoL tool.

---

Idea:

Understandably, there are challenges here. Such as "How do you preserve the ordering of custom commands & hand-crafted queries that are part of a migration"?.

How about we sidestep those problems with something like an "aggregate migration"? A migration that is the sum of multiple migrations, maintaining the separate, and ordered, SQL commands that multiple migrations would create. With one final up state and one final down state? Preserving the nuances of the individual SQL commands within each "child", but with only one snapshot that needs to be stored.

This doesn't aim to solve the "We have too many migrations" problem, but rather the "We only need 1 migration for this release, and we have 20" problem. Which, incidentally, may help alleviate the former.