Sqlite Migrations: Table rebuilds

[bricelam]

In SQLite, the following operations will require rebuilding the table - https://docs.microsoft.com/en-us/ef/core/providers/sqlite/limitations.

Other databases also have a smaller set of limitations that require a table rebuild (i.e. changing Identity on a column in SQL Server).

[julielerman]

also here's the error :

 data git:(TEST_E1) ✗ dotnet ef migrations add init -s ../console
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.EntityFrameworkCore.Update.Internal.SharedTableEntryMap`1.CreateSharedTableEntryMapFactory(IReadOnlyList`1 entityTypes, IStateManager stateManager, String tableName, String schema)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.DiffData(TableMapping source, TableMapping target, DiffContext diffContext)
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.Diff(TableMapping source, TableMapping target, DiffContext diffContext)+MoveNext()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.DiffCollection[T](IEnumerable`1 sources, IEnumerable`1 targets, DiffContext diffContext, Func`4 diff, Func`3 add, Func`3 remove, Func`4[] predicates)+MoveNext()
   at System.Linq.Enumerable.ConcatIterator`1.MoveNext()
   at Microsoft.EntityFrameworkCore.Migrations.Internal.MigrationsModelDiffer.Sort(IEnumerable`1 operations, DiffContext diffContext)
   at Microsoft.EntityFrameworkCore.Migrations.Design.MigrationsScaffolder.ScaffoldMigration(String migrationName, String rootNamespace, String subNamespace, String language)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
Object reference not set to an instance of an object.

[bricelam]

@julielerman Looks related to data seeding. Could you submit a new issue so we can investigate?

[julielerman]

for clarification, do you mean the data seeding APIs? Because I don't have any seeding code in the context. Thanks

[julielerman]

(and will do re new issue)

[bricelam]

modelBuilder.HasData()

[julielerman]

Sorry, my question wasn't clear. Do you mean that even if I'm not using HasData, the seeding APIs that drive the response to HasData are messing this up? Because I'm not using HasData in the context.

[julielerman]

Update: After a lot of trial and error (also fails w SQL Server, succeeds on a simple repro,succeeds in 2.0.3 but fails with 2.1.0-2.14) I finally discovered that it succeeds again with the 2.2 preview. So unless you want a proper breadcrumb for this problem which I can't get to the bottom of, I won't bother creating an issue because it seems to be okay going forward. It would be interesting to debug it into internal EF Core code, but probably not worth the time. Let me know if you feel differently about that, @bricelam . Sorry to have bogged down this issue with the side-track.

[ghost1372]

is there any news?

[bricelam]

@ghost1372 www.microsoft.com/newsapp :trollface:

[ghost1372]

@ghost1372 www.microsoft.com/newsapp :trollface:

:D i mean news about table rebuilds 😋

[bricelam]

We're still in the process of planning features for the 3.0 release. I put the consider-for-next-release label on it so I can advocate for it like I do every release. The Entity Framework Core Roadmap has some of our early thoughts about the 3.0 release as well as the process we follow when considering features.

[ajcvickers]

See also #5662, which is related since it was going to be covered by rebuilds for all providers.

[alexstrickland]

Xamarin has become an integral part of the development offerings from Microsoft, and therefore it is natural to want to use EF Core on Android and iOS and therefore also use SQLite (what choice?) and yet over 5 years after this issue was opened migrations are still not fully supported. I honestly could not care if rollback is not possible for certain operations or it is inefficient, please just make it seamless and warn about it in the docs.

[bricelam]

please ... warn about it in the docs.

@alexstrickland In which docs specifically would you like to see it mentioned? It's covered in SQLite EF Core Database Provider Limitations

[alexstrickland]

@bricelam What I mean is please add the functionality to the sqlite migrations to perform the currently unsupported migrations. If they are a bit crummy then perhaps make them opt-in, and document that the migrations are a bit crummy. By crummy I mean that rollback is not possible. You guys have made a glorious migrations system which I don't want to have to hand patch when a change involves an unsupported migration. Particularly as my hand patch will be just as "bad" as what EF will do (no rollback). I get the extra work and no benefit. Am I oversimplifying this?

[nbelley]

To add to that, would it be possible at least to generate a throw exception instead of an incorect migration command for the SQLite provider?

[mauruskuehne]

We'd love to have this feature as well (even without rollbacks). SQLite with EFCore is a great way to handle local data with Xamarin. But this limitation makes even small changes to the data model hard to implement.

[nbelley]

We'd love to have this feature as well (even without rollbacks). SQLite with EFCore is a great way to handle local data with Xamarin. But this limitation makes even small changes to the data model hard to implement.

I usually just delete all and force a new initial migration... sucks for the user, but it's bullet proof.

[leak]

Meanwhile my little experiment is running in production for 1.5 years.

The only issue I can recall is that one time for a more complex change the order of operations that the migration code generator produced was not applicable and needed manually editing (reordering) of the generated migration. Other than that it is running smoothly.

In case that CLA/PR subject comes up again. I do not have the time of bringing that code to MS-grade quality level (cleanup, update, unit tests, etc.), but my code is released under WTFPL, meaning I literally do not care the slightest bit how or by whom it is used.

I do have a version updated to 2.2.4 in a private branch and would be willing to share that too in case of interest.

[bricelam]

I'm as eager as anyone to get this into EF Core. Unfortunately, it won't be coming in 3.1 as that's a stabilization release and any significant new feature is inherently risky. I promise I'll keep advocating for it.

[alexstrickland]

I do have a version updated to 2.2.4 in a private branch and would be willing to share that too in case of interest.

@leak I'd be interested. As an only moderately experienced .net programmer, is it relatively easy to integrate your changes into the stable branch and compile? Thank you for the offer.

[InteXX]

@nbelley

I usually just delete all and force a new initial migration... sucks for the user, but it's bullet proof.

Could you elaborate? What do you do to accomplish this?

[AdamJachocki]

Come on, guys. It's been 5 years and still nothing? Really? Is Xamarin and EfCore just a sandbox or real tool for creating real apps? I even started to create my own migration mechanism to work with different RDBMS - also in Sqlite. And it's not so hard. I know that Sqlite has its problems and restrictions, but really. This is just shameful that no one from company like Microsoft isn't able to do this for 5 years.

[InteXX]

@nbelley

I usually just delete all and force a new initial migration... sucks for the user, but it's bullet proof.

Could you elaborate? What do you do to accomplish this?

[AndriySvyryd]

Come on, guys. It's been 5 years and still nothing? Really? Is Xamarin and EfCore just a sandbox or real tool for creating real apps? I even started to create my own migration mechanism to work with different RDBMS - also in Sqlite. And it's not so hard. I know that Sqlite has its problems and restrictions, but really. This is just shameful that no one from company like Microsoft isn't able to do this for 5 years.

@AdamJachocki There are only 5 people working on EF Core and there's only so much we can do. See https://github.com/aspnet/EntityFrameworkCore/issues/1368#issuecomment-546085835 and https://github.com/aspnet/EntityFrameworkCore/issues/2266#issuecomment-524077785

[AdamJachocki]

OK, maybe I overreacted, but I think that this is really strategic functionality, especially if you want to promote Xamarin.

Maybe my ideas would help to push this implementation further.

In my solution (own database migration mechanism), I have:

• fluent part, that creates some kind of universal database schema - tables, indices, foreign keys - as C# objects.

Next I have DatabaseCreator class that receives dialect*, IDbAccess, logger and so on.

Now, dialect. Dialect is a interface expecially built for particular RDBMS. So I have two dialect classes implementing the interface - one for MSSQL, and one for Sqlite.

I won't give you any MSSQL info, because you know it much better, I will give you some Sqlite hints that I use.

First of all, I have list of recreated tables:

List<string> recreatedTableNames = new List<string>();

Next I have the most important method: GetRecreateTableCommand

My mechanism works like that: DatabaseCreator gets things to do (for example, create table, create index). Now, every step is taken from dialect like this (for example creating table):

void CreateTable(ITable table)
{
//ITable has table name collection of IColumn - just schema in C# objects

  ExecuteQueryList(dialect.GetCreateTableCommand(table))
}

ExecuteQueryList just loggs and executes list of queries returned by dialect. Dialect returns mostly really only one query, but may return several queries. So it's better to have a list instead of one query.

Now, let's get back to GetRecreateTableCommand -> this is, as I said, the most important method in Sqlite dialect:

IList<string> GetRecreateTableCommand(ITable tab, List<string> commonColNames = null)           
{
    string originalName = tab.Name;
    string tempName = "newTempName_" + tab.Name;
    tab.Name = tempName;

    string newTableCreateCommand = string.Join(Environment.NewLine, GetCreateTableCommand(tab));

    string colNames = string.Empty;
    if (commonColNames == null)
        colNames = string.Join(", ", tab.Select(c => c.Name));
    else
        colNames = string.Join(", ", commonColNames);

    tab.Name = originalName;

    return StrToList1($@"PRAGMA foreign_keys=OFF;
                    {newTableCreateCommand}
                    INSERT INTO {tempName}({colNames}) SELECT {colNames} FROM {originalName}
                    DROP TABLE {originalName};
                    ALTER TABLE {tempName} RENAME TO {originalName};

                    PRAGMA foreign_key_check;
                    PRAGMA foreign_keys=ON;");              
}

So this method just does what Sqlite team tells to - copies the table to temp table, drops the original table and recreates it. Todo -> copy data back to recreated table.

So for example if you have to change column type name on a table, it's just:

public IList<string> GetAlterColumnChangeTypeCommand(ITable tab, IColumn col)
{
    if (recreatedTableNames.Contains(tab.Name))
        return null;

    recreatedTableNames.Add(tab.Name);
    return GetRecreateTableCommand(tab);
}

So the thing is:

• user by fluent api creates database schema
• database creator does needed changes
• as far as sqlite is concerned, most changes are just - recreate the table. So, when we have to drop foreign key, whole table is recreated using the new schema, and then this table name is being added to helper list (recreatedTableNames).

Now, some Sqlite queries that are helpful:

public string GetColumnExistsCommand(string tableName, string columnName)
{
    return $@"select count(*) as result
            from pragma_table_info('{tableName}')
            WHERE name = '{columnName}'";
}

//this one is really helpful. It gets most of table information as you can get.

public string GetColumnsInfoCommand(string tableName)
{
    return $@"select
            ti.name as Name,
            ti.type as TypeName,
            ti.dflt_value as DefaultValue,
            (case when ti.dflt_value IS NULL THEN 0 ELSE 1 END) as HasDefaultValue,
            ti.pk as IsPrimaryKey,
            (case when ti.[notnull] = 1 THEN 0 ELSE 1 END) as Nullable,
            (case when indexInfo.indexName IS NULL THEN 0 ELSE 1 END) as IsIndex,
            (case when indexInfo.IndexDescending = 0 then 1 else 0 end) as IndexAscending,
            (case when fkInfo.[ownerColName] IS NULL THEN 0 ELSE 1 END) as IsForeignKey,
            fkInfo.foreignTableName as ForeignTableName,
            fkInfo.foreignColName as ForeignColumnName,
            (case 
                when fkInfo.on_update = 'CASCADE' THEN 'Cascade' 
                when fkInfo.on_update = 'SET DEFAULT' THEN 'SetDefault'
                when fkInfo.on_update = 'SET NULL' THEN 'SetNull'        
            ELSE 'NoAction' END) as ForeignKeyOnUpdateAction,
            (case 
                when fkInfo.on_delete = 'CASCADE' THEN 'Cascade' 
                when fkInfo.on_delete = 'SET DEFAULT' THEN 'SetDefault'
                when fkInfo.on_delete = 'SET NULL' THEN 'SetNull'        
            ELSE 'NoAction' END) as ForeignKeyOnDeleteAction
        from pragma_table_info('{tableName}') as ti
        left join
        (
            select m.name as indexName, ii.name as colname, ii.[desc] as IndexDescending
            FROM sqlite_master as m
            left join pragma_index_list('{tableName}') pil on pil.name = m.name
            left join pragma_index_xinfo(m.name) as ii on ii.name IS NOT NULL

            where m.type = 'index' and m.tbl_name = '{tableName}' and pil.origin<> 'u' and pil.origin<> 'pk'
        ) as indexInfo on indexInfo.colname = ti.name
        left join
        (
            select[table] as foreignTableName, [from] as ownerColName, [to] as foreignColName, on_update, on_delete
            from pragma_foreign_key_list('{tableName}')
        ) as fkInfo on fkInfo.ownerColName = ti.name";
}

//And my columnInfo class is just:

public class BasicColInfo
{
    public string Name { get; set; }
    public string TypeName { get; set; }
    public object DefaultValue { get; set; }
    public bool HasDefaultValue { get; set; }
    public bool IsPrimaryKey { get; set; }
    public bool Nullable { get; set; }
    public bool IsIndex { get; set; }
    public bool IndexAscending { get; set; }
    public bool IsForeignKey { get; set; }
    public string ForeignTableName { get; set; }
    public string ForeignColumnName { get; set; }
    public ForeignKeyRule ForeignKeyOnUpdateAction { get; set; }
    public ForeignKeyRule ForeignKeyOnDeleteAction { get; set; }
}

public string GetTableExistsCommand(string tableName)
{
    return $@"SELECT count(*) FROM sqlite_master WHERE type = 'table' AND name = '{tableName}'";
}

I have also ColumnComparer class that is used to detect schema alterations and update the database schema. I think you have much more better mechanism for this, but this is mine.

I check if the table exists - if yes, there may be alteration required. Now I check columns in the table, using query from GetColumnsInfoCommand and create list of ColumnInfo class objects.

Next, I just compare it like that:

enum ColumnCompareResult
{
    ChangeNullable,
    ChangeDefaultValue,
    ChangePrimaryKey,
    ChangeType,
    ChangeIndex,
    ChangeForeignKey
}

class ColumnComparer
{

    IDbDialect dialect;

    public ColumnComparer(IDbDialect dialect)
    {
        this.dialect = dialect;
    }

    bool NullableChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return requiredSchemaColumn.CanBeNull != existingColumn.Nullable;
    }

    bool DefaultValueChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return requiredSchemaColumn.DefaultValue != existingColumn.DefaultValue;
    }

    bool PrimaryKeyChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return requiredSchemaColumn.IsIdentity != existingColumn.IsPrimaryKey;
    }

    bool FieldTypeChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        string sqlFieldType = dialect.GetFieldTypeFromDbType(requiredSchemaColumn.ColType, requiredSchemaColumn.Length);
        if (string.Compare(sqlFieldType, existingColumn.TypeName, true) != 0)
            return true;
        else
            return false;
    }

    bool IsIndexChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return (!(requiredSchemaColumn.IndexInfo != null && existingColumn.IsIndex));
    }

    bool IndexSortChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        if (requiredSchemaColumn.IndexInfo == null)
            return false;

        IndexSortOrder existingSortOrder = IndexSortOrder.Descending;
        if (existingColumn.IndexAscending)
            existingSortOrder = IndexSortOrder.Ascending;

        return requiredSchemaColumn.IndexInfo.SortOrder != existingSortOrder;
    }

    bool IndexChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return IsIndexChanged(requiredSchemaColumn, existingColumn) || (IndexSortChanged(requiredSchemaColumn, existingColumn));
    }

    bool IsFkChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return (!(requiredSchemaColumn.ForeignKeyInfo != null && existingColumn.IsForeignKey));
    }

    bool FkKeyRuleChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        if (requiredSchemaColumn.ForeignKeyInfo == null)
            return false;

        return requiredSchemaColumn.ForeignKeyInfo.UpdateRule != existingColumn.ForeignKeyOnUpdateAction ||
            requiredSchemaColumn.ForeignKeyInfo.DeleteRule != existingColumn.ForeignKeyOnDeleteAction;
    }

    bool FkTableChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        if (requiredSchemaColumn.ForeignKeyInfo == null)
            return false;

        return string.Compare(requiredSchemaColumn.ForeignKeyInfo.ForeignTableName, existingColumn.ForeignTableName, true) != 0;
    }

    bool FkColumnChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        if (requiredSchemaColumn.ForeignKeyInfo == null)
            return false;

        return string.Compare(requiredSchemaColumn.ForeignKeyInfo.ForeignTableName, existingColumn.ForeignTableName, true) != 0;
    }

    bool ForeignKeyChanged(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        return IsFkChanged(requiredSchemaColumn, existingColumn) ||
            FkKeyRuleChanged(requiredSchemaColumn, existingColumn) ||
            FkTableChanged(requiredSchemaColumn, existingColumn) ||
            FkColumnChanged(requiredSchemaColumn, existingColumn);
    }

    public List<ColumnCompareResult> CompareColumns(IColumn requiredSchemaColumn, BasicColInfo existingColumn)
    {
        List<ColumnCompareResult> result = new List<ColumnCompareResult>();

        if (NullableChanged(requiredSchemaColumn, existingColumn))
            result.Add(ColumnCompareResult.ChangeNullable);

        if (DefaultValueChanged(requiredSchemaColumn, existingColumn))
            result.Add(ColumnCompareResult.ChangeDefaultValue);

        if (PrimaryKeyChanged(requiredSchemaColumn, existingColumn))
            result.Add(ColumnCompareResult.ChangePrimaryKey);

        if (FieldTypeChanged(requiredSchemaColumn, existingColumn))
            result.Add(ColumnCompareResult.ChangeType);

        if (IndexChanged(requiredSchemaColumn, existingColumn))
            result.Add(ColumnCompareResult.ChangeIndex);

        if (ForeignKeyChanged(requiredSchemaColumn, existingColumn))
            result.Add(ColumnCompareResult.ChangeForeignKey);

        return result;
    }
}

So, I hope I could help and that Sqlite will be better serviced in EfCore.

[InteXX]

@AdamJachocki

Thank you for the comprehensive code samples. Could you edit your post and set the markdown formatting for snippets, i.e. correct the indentation so that all the code is displayed as code? Thanks.

[AdamJachocki]

@AdamJachocki

Thank you for the comprehensive code samples. Could you edit your post and set the markdown formatting for snippets, i.e. correct the indentation so that all the code is displayed as code? Thanks.

Done

[InteXX]

@AdamJachocki

Very good, thank you.

This appears surprisingly simple. Hopefully the team will be able to consider using some of these ideas.

[leak]

Glad to see that I'm not alone in the SQLite+EF woods, I was always wondering by the lack of activity in this issue..

So I updated my code from above to 2.2.6 [1], see here: https://github.com/leak/EntityFrameworkCore/commit/71d3a35cb100bcdbd84eee362d49385134b28126

I also added the nuget packages that I am using in my project here: https://github.com/leak/EntityFrameworkCore/releases/tag/2261

Few notes:

• The packages are drop-in replacements for the official 2.2.6 packages, just throw them in a folder, create a nuget.config and point them to that folder, that's it
• My tablerebuild226 branch is based on the release commit for the 2.2.6 nuget packages
• I changed the .UseSqlite() method to .UseSqliteX() to really make sure my code is used and I wouldn't get screwed by package caches
• The 2.2.6.1 nuget packages linked above are pure release builds from my branch, with just the version number edited, again to avoid package caching issues

And the usual disclaimer: This code works for me in production, but there still might be dragons. Again my code is released under (WTFPL), use it, leave it, make a PR in this repo - I simply do not care.

[1] Since MS decided to drop full framework support for anything 3.x (yay..) I did not really attempt to run this on anything beyond 2.x

[TomasHubelbauer]

I'm also very much interested in SQLite + EF Core! Do not have anything substantial to add, just excited to see a hint of activity on this topic.

[ErikEJ]

Sqlite engine supports column rename now, no need for rebuild

[InteXX]

@ErikEJ

This is very good news. Thank you for keeping up with their progress on this.

[bricelam]

Note, we added support for column renames in EF Core 2.2

[1Mgtrn3]

@leak Thank you! It worked, although it may also require to install Microsoft.CodeAnalysis.CSharp.Workspaces.

I had an issue "Could not get the reflection type for dbcontext..." when I tried to scaffold a controller using a DbContext added with UseSqliteX(). Installing Workspaces package resolved that issue. I can say that the issue was caused by installing that custom package because I had scaffolded a couple of controllers successfully in that project before changing Sqlite.Core to a custom version.

[leak]

@1Mgtrn3 Thanks for the feedback.

To be honest I never used scaffolding, so since I changed a public api there there is a chance it messed with that.

Just out of curiosity, did you try renaming that method back and see if the issue goes away?

[teajaysee]

if a table cant be altered - ie migration fails - is there anyway of forcing it to recreate the tables (even if that loses data) , rather than having to recreate the whole database and lose everything?

[Epstone]

Just as an information how to generate sql code for the migration with some tooling support:

Download sqlite studio, do your changes manually in the schema editor until your migration is mirrored there. Before commit you can export the created sql code for migration and merely copy and paste that as replacement for the migration steps.

'Down' migration may be done the other way around.

Still error prone and code needs to be reviewed and tested, but less pain for writing table rebuilds again and again. Maybe it helps some one right now

[bricelam]

Nothing really new here, just jotting down some notes since I started thinking about this again today:

// UNDONE: Not supported by SQLite
//migrationBuilder.AlterColumn<string>(
//    name: "Title",
//    table: "Posts",
//    nullable: false,
//    defaultValue: "",
//    oldClrType: typeof(string),
//    oldType: "TEXT",
//    oldNullable: true);

// Create a new table with the desired schema
migrationBuilder.CreateTable(
    name: "ef_temp_Posts",
    columns: table => new
    {
        Id = table.Column<int>(nullable: false)
            .Annotation("Sqlite:Autoincrement", true),
        Title = table.Column<string>(nullable: false)
    },
    constraints: table =>
    {
        table.PrimaryKey("PK_Posts", x => x.Id);
    });

// Create temporary UNIQUE indexes to enforce during INSERT
migrationBuilder.CreateIndex(
    name: "ef_temp_IX_Posts_Title",
    table: "ef_temp_Posts",
    column: "Title",
    unique: true);

// Copy data from the old table. Use IFNULL to specify a default value for newly
// required columns
migrationBuilder.Sql(@"
    INSERT INTO ef_temp_Posts (Id, Title)
    SELECT Id, IFNULL(Title, '')
    FROM Posts;
");

// Suspend foreign key enforcement during the swap
// NB: This commits the current transaction. We can't rollback the migration if
// anything after this fails. We can mitigate it by doing rebuilds as late as
// possible
migrationBuilder.Sql("PRAGMA foreign_keys = 0;", suppressTransaction: true);

// Swap in the new table
migrationBuilder.DropTable(
    name: "Posts");
migrationBuilder.RenameTable(
    name: "ef_temp_Posts",
    newName: "Posts");

// NB: This will cause an integrity check which may fail if foreign keys were
// not previously enforced and inconsistent data has crept in
migrationBuilder.Sql("PRAGMA foreign_keys = 1;", suppressTransaction: true);

// Drop temporary UNIQUE indexes
migrationBuilder.DropIndex(
    name: "ef_temp_IX_Posts_Title",
    table: "Posts",
    column: "Title");

// Rebuild any indexes
migrationBuilder.CreateIndex(
    name: "IX_Posts_Title",
    table: "Posts",
    column: "Title",
    unique: true);

[bricelam]

I looked into using PRAGMA defer_foreign_keys to avoid suppressTransaction: true, but dropping the table sill cascade-deletes dependents.

We discussed the consequences of suppressTransaction: true in a design meeting today and decided that it is OK to do this by default on SQLite. We did this for certain scenarios in EF6 on SQL Server and never heard any negative feedback. An option to disable it (and thus table rebuilds) would, however, be nice to have.

We also discussed blindly turning foreign keys back on (when we don't know if they were enabled in the first place) and decided that it's OK. If you turn them off for performance reasons, your data should still be consistent otherwise additional things in EF will break. The default foreign key behavior will just be reset when the connection is closed and re-opened after migrating anyway.

Other points that came up in the discussion:

• We should warn when we do a rebuild that any changes you've made outside of your EF model may be lost. (But this shouldn't really be an issue if you're using migraitons to manage your database schema)
• There's no value in re-using parts of the query or update pipeline to generate the INSERT INTO..SELECT statements. Just use the usual mechanism for generating SQL in migrations
• No need to support table rebuilds on SQL Server as part of this. Re-openeing #2100 for doing column rebuilds when IDENTITY changes

[danielmeza]

From the SQLiteStudio there is a short sintax to clone the original table

CREATE TABLE sqlitestudio_temp_table AS SELECT *
                                          FROM Table;

this way we can reduce the complexity.

[ir2000]

@danielmeza you would lose primary key and foreign keys

A table created using CREATE TABLE AS has no PRIMARY KEY and no constraints of any kind. The default value of each column is NULL. The default collation sequence for each column of the new table is BINARY. https://www.sqlite.org/lang_createtable.html#createtabas

[InteXX]

@ir2000 @danielmeza

Yes, in that single statement data is lost. But it's a valuable technique for cloning the schema.

One could then clear out the data and repopulate, bringing all keys along this time.

[danielmeza]

@ir2000 yes but we not need that info in the temp table.

[danielmeza]

@bricelam we should cover in this version escenarios with an existing db? Example: code first from db escenarios.

[Damus765]

@danielmeza I would say we need to create every constraint in the "temp table". Because what you call "temp table" is not really temporary. It is in fact the new table that will be kept once renamed. So I'm not sure CREATE TABLE AS is the best way to create the new table.

[danielmeza]

@Damus765 I was missing the rename process, I thought that the data will be copy to the new table so the process goes like this. Create the temp table -> delete the old table -> create the new table with the new schema -> copy data back from the temp table -> delete the temp table.

[Damus765]

@danielmeza According to the SQLite documentation, the recommended steps are as follows:

1. Create new table
2. Copy data from old table into new table
3. Drop old table
4. Rename new into old

Only one data copy is required and you don't need a temp table.

[bricelam]

@danielmeza Our strategy for using migrations with existing databases is tracked by #2167

[bricelam]

More related issues for SQL Server: https://github.com/dotnet/efcore/issues/12586, https://github.com/dotnet/efcore/issues/16758, #18965 & https://github.com/dotnet/efcore/issues/20943