Handle migrations

[alexgleason]

How do you handle it when someone changes the name of a field? Or changes the type of a field, or adds or removes a field? Django for instance has built-in migrations. It makes files that keep track of the changes (writing the change ID to the database each time you migrate), and prompts the user when changes are made if it can't determine automatically what the user wants.

[Emkaytoo]

Just noting: one way to do this would be to have the files stored in the plugin folder and then run an "Update" function that checks for the existing fields, gets the ID, and then updates the field with the new information. It wouldn't be automatic, but would happen when the user selects a specific model.json and then would only compare the data in that model.

One thing to think about is the content in the fields that are changing.

[reganlawton]

@Emkaytoo Could you possible hook into Events and check for changes to section, field, etc. Then create a migration file with a unique key, timestramp, etc. Also create and store pass run migrations in a table in the DB so you don't re-run migrations again.

You can also automatic the migrations but comparing the files to DB using some type of method be it a hash check or using a loop with unique key check, all those this would have performance issues.

Symfony Framework has a bundle that does something similar called "Doctrine Migrations".

[spAnser]

Is this something that people want semi automated? Or is this something people want to handle semi-manually?

The onSave events are triggering before Structures are fully saved causing Field Layouts to be empty and some other things. Along with no event when a field gets saved.

Item conflicts will be field types changing in a way that causes data loss. or maybe Field Layouts having fields being removed also causing data loss.

[alexgleason]

I think it's possible to design it in such a way that it can be done manually, but a running a script could also automate it.

If you keep track of migrations in files, someone could manually write that file or use a tool to automate its creation. So should you decide to take this path, you could implement it for manual creation first then implement a tool to generate the migrations later.

It definitely shouldn't be "fully automatic" in the sense that there's no user input at all and changing the schema just changes the database without warning. That could lead to disastrous data loss by mistake, and there's no opportunity for the user to review the migration.

Django has two CLI commands: makemigrations and migrate. This is the most safe and versatile, despite being kinda a pain to type repeatedly. makemigrations let's the user review the generated migration before processing it, and then migrate accepts any migrations, user-written or generated.

Also, Ian, using a JSON file may be fine if the migrations were all guaranteed to be very simple. But in practice a migration might sometimes have to execute arbitrary code. That's why Django stores them as executable *.py scripts.

For instance, converting from a MarkdownField to a RichTextField is not as simple as changing the field type and pulling in the old data. We have to process the data for every row through a markdown to HTML converter so it actually shows up correctly after the conversion.

Alternatively you could use pluggable conversion scripts which you reference from a JSON migration. There are advantages and disadvantages to this. But the point is, sometimes you want to be able to execute your own code during a migration so it's worth planning for that.

[reganlawton]

Converting fields and preventing data loss is the big task here, and is something Craft itself hasn't really sorted yet.

The two step process is basically standard across the migration systems in most frameworks and should be the step The Architect takes. On the note of JSON migration file you should be able to to add additional SQL which cant be injected along side the and before the migration changes.

This would allow newbies to use the standard conversions/ migrations, but power users will be able to control everything they wish.

YOu should be testing migrations on staging servers in the ideal world and you can semi-automate by using a general config option/ setting maybe? Also automatic migrations would probably best suit a DB backup pre-migration.

[spAnser]

So if your looking for 2 stage process with scripting like Django, Craft 3 Migrations looks to be the solution there.

The thing is Craft doesn't entirely need the whole script running unless your dealing with content. The structure of the entry types and fields can all be updated without scripting.

If there was an automated part of this. It would stop itself from running if a field type got changed. Prompting the user the confirm the new field type before proceeding. The automated update part is something like it detecting if all field types are matching.

[spAnser]

Here is a quick screen cap of the current migrations page from the develop branch.

[23d1]

Having installed and taken a look at this branch, would it be a good idea to have a way to import migrations based on field handles instead of IDs as an option? I noticed, as I'm building two separate sites in parallel, that when I try to migrate between them the fields end up having different IDs (probably because some fields needed to be done manually, and so the IDs were offset). If I migrate, fields get converted all over the place. I suppose that function would be sort of like the original Raw Input tab functionality, only it would edit and update a field's name, content, and so forth, everything but the handle.

[spAnser]

@23d1

That could be on the slate. The migration would break if anyone ever updates a fields handle though. Even though that should be a rare circumstance.

Originally I wanted to make sure the actual migration files worked in any circumstance. Where the work flow was DEV > STAGE > LIVE. You only edit the structures on DEV. Which then migrate to stage then live.

Going back and forth could be a small hassle. However something I was considering before migrations was a checkbox on the Raw Input page that was to attempt to update fields with matching handles. Is that something that might work?

[23d1]

@spAnser

Totally understandable. Too many options is also way more prone to human errors.

That sounds like THE solution. So it would update everything. From relations, in terms of where it is laid out and so forth (whether in a Neo Field or on a Structure or in a Global section), to instructions, groups, layouts and so forth...?

An example scenario would be like the rats nest I find myself in at the moment; Two identical installs that have diverged from one another in terms of field IDs due to manual creation and layout in the admin. Leading to manually have to go through and update accordingly, through the admin. Granted, The Architect has saved me a lot of time so far. 😄

[Emkaytoo]

One thing to keep in mind is the scope of the plugin. It was developed to be a way to manage the models themselves, and not the data. We want to respect the data and not change unless the user is sure, but the Architect is designed to be a way to manage and configure the models.

I think migrations from the models perspective is absolutely in the scope of the plugin, but there are things outside of the control of the plugin that affect how migrating the data would happen. I don't foresee (at least in the near future) that the Architect is the way to completely alter all the content in all the fields.

Some examples below:

• In scope: say there was an Assets field and 4 Asset Sources. If you added a 5th source, but then deselected it in that one Assets field model, I can see the Architect updating the assets field based on the model.
• Out of scope: say you have an entryType with a main call-to-action button that's managed using an Entries field and you switch it to LinkIt, I don't see the Architect converting them from one to the other.
• In scope: say there's a Neo field, and you add two blocks and reconfigure the groupings. I can see the Architect changing that because the underlying fields aren't altered.
• Out of scope: same Neo field, but instead you change the parent/child relationships with allowed and disallowed blocks. I don't see the Architect reordering the content on an entry that exists.

I think we can get the plugin to a point where you can alter models in your dev environment, add and subtract things, and alter some fields that you then push to stage and prod. I don't ever see it parsing a Markdown field, running it through a transcoder, and converting it to RichText.

[23d1]

Makes sense. Converting the field, or the handle wouldn't make sense for the purpose of the plugin, but altering content would be awesome. Especially Neo fields and field layouts in general.

Cheers!