Update model from database

[rowanmiller]

We'll add a new comand (e.g. Update-DbContext and dotnet ef dbcontext update) to re-scaffold your DbContext and entity types to incorporate any changes you've been made to the database schema. This will allow you to update your model to be compatible with the database but preserve any customizations you've made (e.g. navigation property names) to the originally scaffolded code.

Scope

The folowing types of customizations will be preserved.

• Entity type names
• Entity type property names (including navigation properties)
• Entity type inheritance
• DbSet property names

The following are not a priority during the initial implementation.

• Non-EF annotations on entity type properties (could be enabled via [MetadataType])
• Configuration that deviates from the database (database always wins)
  • Additional constraints added to the model

Files will be blindly overwritten during this process. To allow additional customizations to the model, we'll continue generating partial classes and reccommend adding custom members to a new file with another partial definition of the class.

Implementation

We'll leverage the existing Reverse Engineering assets for this work. Specifically:

• IDatabaseModelFactory--Reads the database schema.
• IScaffoldingModelFactory--Transforms the schema into a compatible IModel.
• IScaffoldingCodeGenerator--Transforms the IModel into code.

We'll read the current database schema, transform it into a model and compare it to the current model. Anything that has changed will be merged with the current model and the resulting model will be scaffolded into code. Existing files will be overwritten and removed types will be deleted.

During the inital reverse engineer, we scaffold an OnConfiguring() stub. We'll need to move this into it's own file so we can generate it once, but not during update.

TODO

• [ ] Refactor generated code to be update friendly (consider #8434 & #9318)
• [ ] Create a scaffolding model differ
• [ ] Update IScaffoldingCodeGenerator to handle new concepts (e.g. inheritance)

[bricelam]

In theory, this isn't too different than Migrations: Diff the current model with the reverse engineered model from database and generate operations to perform on the Roslyn syntax tree.

[rowanmiller]

Yeah, I think we'd want to keep a snapshot when the model was reverse engineered, so that we can work out what changed without loosing changes the developer has manually made to the model. Probably ok if we overwrite changes on something that also changed in the database, but not things that remained the same. But, as you said, all those pieces are there from migrations.

[RandyBuchholz]

One of the things I encounter often when working on the early stages of a project and scaffolding from PM> is not being able to update when I can't build. I may have changed my business data and database to match, but now my project is full of errors. In some cases, running "update-model" is the solution to removing the errors, but I can't because of the errors. I'm not sure why scaffold-dbcontext requires code to compile since it is only creating new objects and has AFAIK, no dependencies on the target project business objects. When I work DB-first I have a rule to never touch the EF code. I don't want to go in and fix the all the errors just so I can overwrite what I just did. I usually end up having a separate project I scaffold into and cut and paste. But then I have to go back and adjust all of the namespaces.

It would be nice to be able to scaffold when there are errors, and to have drop-recreate capability. Sometimes the best "update" is starting over. For example scaffold-update-model -o Models -witherrors -recreate would delete everything in the Models folder, invoke scaffolding even if there are errors in my code, and then do a clean construction into /Models/.

It would also be nice to have a -namespace switch.

[bricelam]

...why scaffold-dbcontext requires code to compile...

There are a couple reasons we do this:

• To pick up any design-time services that have been overridden in the project (e.g. customizing the code generation)
• If the provider's design-time package was just installed, we need to build to ensure it's copied into the output directory.

It is also just done in general for all commands since every other command besides Scaffold-DbContext needs to operate on your project's assembly.

In theory, thouth, it is optional for Scaffold-DbContext, and we could consider using the -Force switch to continue anyway (with a warning).

[bricelam]

It would also be nice to have a -namespace switch.

I think the -OutputDir parameter can be used to configure the namespace and directory of the model. If so, it looks like we should update the documentation.

[RandyBuchholz]

Thanks for the explanation.

Skipping compile on -force with a warning would work great.

-o -OutputDir does add the namespace of the folder you gen to. I may be the rarer case, but I don't keep a strict 1:1: folder to namespace mapping. I often use container folders to subdivide or group things, and skip them in the namespace. Generally I'll do this by architecture areas. I group my database related folders under DB/ to keep them together in the explorer, but skip it in my namespace. So my models are in the DB/Models folder, but in the Models namespace. It would be cool to be able to get that with -o DB/Models -namespace Models.

I put a feature request in to VS a while ago for a "Use Parent Namespace" bool property on folders. Anytime you right click and add a file, it skips those folders when it creates the namespace.

[marchy]

Just to throw a very critical scenario into the mix, some of us that started out our codebases on EF Core 1.0, then had to downgrade to EF6 due to the slew of limitations along the development path for it (ie: pulling full database tables in memory so the framework can perform operations that were not yet implemented --> how was this ever an idea that gained any traction? This is so unfeasible for any web server I don't even know where to begin.... but I digress) - we've now had migrations diverge as we have evolved our models in EF6.

At some point, we will want to go back to EFCore and it means we have:

• EFCore1 migration
• EFCore1 migration
• EFCore1 migration ....
• EF6 migration
• EF6 migration
• EF6 migration ....
• EFCore2 migration (future)

There will likely need to be a big catch-up migration on the EFCore side to capture the deltas between the last EFCore migration and all the migrations that happened in EF6 which are not captured in the EFCore model. Being able to reverse-engineer model updates from the database could do loads to make this pain anything less than excruciating --> though I'm not entirely certain how that would work: would it add/change properties to the POCO classes? What about the DBContext mappings? I'm afraid for any changes in indexes/specifications that might not get caught up (or might differ from what EF6 performed).

Any other approaches you guys can think of for dealing with this situation?

[bricelam]

(Updated the description with a draft "spec" based on our previous design meetings.)

[madhav5589]

I am curious to know if scaffolding specific tables from database using dotnet ef dbcontext scaffold [arguments] [options] command can update the dbContext if we run the same command multiple times. I see there is an option -f | --force which overwrites the existing dbContext.cs file but doesn't update based on all the tables added through the command. When we use -f, it removes the content. Any thoughts ?

[papercity4]

Our team just started using EF Core 2 and use a database first approach. We are running into several problems.

1. Running Scaffold-DbContext works great, but we use DI to pass in the context. We end up having to remove the OnConfiguring method and add in the Constructor to pass in the context. This would be fine except this has to be re-done anytime we re-run this command to get new database changes.

2. The Scaffold-DbContext doesn't work if there are build errors. This is very inconvenient, as you may need the files to be generated to remove the build errors.

3. We use the -Tables option when running for the first time as we don't want to migrate all the tables. However, when running subsequent updates, from what I can tell, you must specify the same exact tables even if you only want to grab a new table that was added. When I tried this the Context class got updated and removed the other tables.

Please take these into consideration, I imagine others that are using database first will or have run into these.

[ErikEJ]

re 1: The generated classes are partial, so you can just add a partial class file with the additional constructor.

re 2: My "EF Core Power Tools" extension does not require the project to build

re 3: "EF Core Power Tools" remembers the list of tables generated for subsequent runs.

[bludev]

In the "DB-First"context, another useful feature of EF6 is the ability to modify the "model.tt" file to customize the generated code for each entity. This allows me to add custom attributes to the class or let it inherit from a base class or even implement an interface. Each time I update the model from the database, entities are properly created or regenerated as instructed in the tt file. In EF Core the tt file no longer exists but I have not found any other workaround to alter the generated code of the entities. Some ideas?

[ErikEJ]

EF Core Power Tools allows you to customize the classes in various ways using Handlebars templates. https://github.com/ErikEJ/SqlCeToolbox/wiki/EF-Core-Power-Tools

[bricelam]

@bludev Tracked on our side by #4038

[john-larson]

Hello,

I have the following questions about the issue:

1. Is the experience going to be the same as when we were using edmx in EF6? For example, when a navigation property is deleted from the model, updating from database does not add it back in EF6. Will these kind of behaviors be the same in EF Core?

2. Another question is, are you going to implement it using some kind of external xml mapping or will it be implemented using attributes residing in the generated classes?

3. This is an issue not resolved since 2014. Am I right to assume EF Core favors code first over db first? (Because anyone I know using db first does customizations and would need this feature.) Is there a specific reason for this? Is code first considered superior to db first in terms of ORM best practices?

[ajcvickers]

@john-larson

1. That's the plan.
2. No mapping changes; it will use code-based mapping with attributes and/or modelBuilder API.
3. Depends what you mean by "code first" and "database first". EF Core uses code-based mapping, which was unfortunately marketed as "Code First". However, code-based mapping is equally applicable to mapping to an existing database, and we recognize that a lot of people do have existing databases. There are different ways of approaching this, one of which is using the database to drive updates to the model. For that experience, not having to re-scaffold when the database changes is an important feature, and that is what is being tracked here. The reason it hasn't been done yet is that it is a big feature and other more fundamental/strategic work has so far been deemed higher priority.

[Varorbc]

@rowanmiller Can add comments for tables, columns, navigation properties?

[devdeer-stephan]

@ajcvickers We are currently running into the same problem where re-scaffolding the whole model on each change on the database is really really inconvenient. Also, having influence on the generated code would be really nice (but I'll look into EF Core Power Tools for that first)

If I understand your third point correctly this has been an issue for almost four YEARS now and in all that time

other more fundamental/strategic work has so far been deemed higher priority

Is there anything we can do to actually raise the priority on this topic? We're currently even evaluating moving our whole application "back" to .Net 4.7.2 and EF6 because the dev process is heavily impacted by the inconvenience of the scaffolding

[ajcvickers]

@devdeer-stephan We try to gather as much information as we can when making priority decisions, so making us aware that you need it (for example by giving a thumbs-up/+1 on the issue) is the best way to influence the priority. That being said, resources are very constrained, so even things that we consider important will, unfortunately, continue to get bumped.

[bludev]

I too thought that the lack of 'update model' was a problem. However, I noticed that in EFCore the scaffolding of the entire database (in my case about 300 tables) is much faster than the update that I did before with EF6 + EDMX. Using 'HandleBars' I was able to customize the creation of context, entities and names of navigation properties. I must say that I no longer miss the old graphic tool of EF6. With 'EF Core Power Tools' you can also select which tables to import.

The current defects:

• I am forced to set the "data layer" as a startup project (Set as StartUp project), otherwise the generation of the entities does not use the custom handlebars.
• after each scaffolding, I am forced to remove the 'OnConfiguring' method (which I have defined elsewhere in a partial class). It would be nice to be able to exclude creation through an option in Scaffold-DbContext.
• the first application startup is very slow, due to the length of 'OnModelCreating', but this is a different topic and probably would have been the same in EF6.

My 2 cents :)

[ErikEJ]

@bludev I think you first 2 defects are addressed in EF Core Power Tools

[blogcraft]

I have a problem with EF Core Power Tools, when I reverse Engineer a table in my database, with "EntityTypes only" selected, it deletes my current context, and if I choose to create de context it removes my previously added tables. (I just need to add a new entity, and leave alone whatever I had)

[ErikEJ]

@blogcraft You must leave everything you want included selected every time you run the tool

[blogcraft]

Ohh, that's a bummer, :( It's the UI version of runing the scaffold-dbcontext command with -f and the list of tables.

Right now my app has ~50 entities out of a 500+ tables of my legacy DB. So, I was actually searching for an easier way to go.

[ErikEJ]

What is your issue? The selection is stored between each run in the efpt.config.json file !

[ErikEJ]

And you can search in the table list...

[blogcraft]

The thing is that I started the project way ago, so, the first time I runned the Power Tool the efpt.config.json was obviously empty. So I had to select all my tables manually.

[ErikEJ]

@blogcraft Sure, but once you have done that, the selection remains in the file...

[joshmouch]

How does this tool determine what a "word" is for the purposes of casing class names, and such? I'd like to add a few of our common acronyms to this list before the classes are generated.

[smitpatel]

@joshmouch - https://github.com/aspnet/EntityFrameworkCore/blob/master/src/EFCore.Design/Scaffolding/Internal/CandidateNamingService.cs Above service is responsible for doing casing. You can add your custom ICandidateNamingService It is in .Internal namespace so it can break between minor release.

[Qtaza]

How to use my implementation ICandidateNamingService in code?

[Diaskhan]

Hey Fellows. I have searched in github and finded this repo https://github.com/sergey-kolodiy/PolyGen

PolyGen is a code generator that produces database schema, ORM layer, REST API and a (coming soon — stay tuned!) single-page web UI for your business model.

Could help in Realizing the issue !

[grantbowering]

...why scaffold-dbcontext requires code to compile...

In theory, thouth, it is optional for Scaffold-DbContext, and we could consider using the -Force switch to continue anyway (with a warning).

In the 2 years since that was posted, has there been anything further on considering -Force (or some other switch) to allow you to re-scaffold without the entire solution being able to build?

[ErikEJ]

@grantbowering Try EF Core Power Tools 😄

[perezdev]

Has there been an update to commands being added to update the schema without using -force? I see the tags reference this being punted to 2.0 and then 2.1. But now we're on EF Core 2.2 and I don't see any update notes mentioning this.

[ajcvickers]

@perezdev I don't believe this has been implemented yet. @bricelam should be able to confirm.

[bricelam]

I'm not sure I understand the question. @perezdev are you asking about progress on this feature? Or are you encountering a specific issue with -Force?

[perezdev]

@bricelam I'm not encountering a problem with -Force. But since it basically wipes out everything and re-creates it, I have to keep a file with the scaffold command up to date with every new table. It's not the end of the world. It would just be nice to have an update command so I can just run scaffold-update or something to that effect without having to specify the creation of every table.

[bricelam]

No progress from us (the EF Team) beyond the initial design (in the description).

There has been some work to ease the re-scaffolding process by the community. Check out EF Core Power Tools and LoreSoft's Entity Framework Core Generator.

I'm also working on a sample that uses T4 templates to fix some of my pet peeves (#4038, #8434, #9580 & #10890) that I plan to blog about when it's finished.

The best strategy today is to take advantage of partial classes to keep the generated code separate from your customized code. (#14544 might help a bit here too) This allows you to re-generate code without losing your customizations. This falls down when you rename things, but you might be able to preserve the renames in you template code (either the Handlebars ones in the Power Tools or the T4 ones I'm working on).

Prioritizing work is difficult on any project, but we try to be open about the process we follow.

[perezdev]

@bricelam Great. thanks for the info.

[joshmouch]

I've been using https://github.com/iQuarc/Geco and like it quite a bit. You just download the project then customize the code as needed. I'd prefer a T4 template, though. All the existing handlebar generators I found had severe limitations.

[foofoodog]

I think the fix at least for me would be to not generate the OnConfiguring code if I already have a partial out there with the OnConfiguring override in it.

[GunboatDiplomat]

Any updates on this feature? Is it marked for any near future release?

[ajcvickers]

@GunboatDiplomat This issue is in the Backlog milestone. This means that it is not going to happen for the 3.0 release. We will re-assess the backlog following the 3.0 release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[shyamal890]

Would this be released in EF Core 5.0?

[ajcvickers]

@shyamal890 This issue is in the Backlog milestone. This means that it is not planned for the next release (EF Core 5.0). We will re-assess the backlog following the this release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources.

[cgountanis]

It would be cool if it also add/update, would check for entities that do not exist any more on the DB side. I find when things get renamed or removed, scaffold leaves orphaned files in the models folder. Maybe just a message that says these files were found but did not have a db-first match or something?

[miloshavlicek]

More than 5 years on backlog. So sad.

[miloshavlicek]

Is anyone considering implementing it in the near future or should I implement it on my own?

[ajcvickers]

@miloshavlicek It's not currently scheduled. A robust implementation of this is non-trivial, but if you do come up with a good solution then you might consider contributing. We would certainly be interested in seeing whatever you come up with.