github / gh-ost

GitHub's Online Schema-migration Tool for MySQL
MIT License
12.14k stars 1.23k forks source link

Integrating gh-ost with flyway or liquibase #356

Open ramkireddy opened 7 years ago

ramkireddy commented 7 years ago

Hi, Wanted to find out if you are using any database migration tools at GitHub along with gh-ost to make the process of applying database migrations seamless, we are looking to integrate gh-ost with either flyway or liquibase tools and would like to see if you are already doing it.

We are into continuous delivery and would want to have the database migrations applied without any manual intervention.

Thanks, Ramki

shlomi-noach commented 7 years ago

Hi,

Are using our very own flow, based on Pull Requests and CI tests (I expect we will be writing & speaking about this).

We do not use liquibase nor flyway.

In the past, and before being at GitHub, I looked into both. My knowledge and understanding is now 3 years old and may be outdated, mind you. I, too, was concerned with continuous deliveries. What I found at the time was that both liquibase and flyway seemed like excellent products, but very strict in the order of events. That is, there was only one way to order schema changes.

So what? The thing was I was looking to be able to deploy some changes to one testing environment, others to another, being able to apply to prod in different order, without engineers having to wait on one another (and of course, taking the optimistic approach, i.e. assuming different engineers do not overstep or collide or interfere with each other's schema changes). At the time, that was a main reason for me to not use either the two solutions.

Things may have changed; and I haven't been looking into these solutions in a while.

Another point for consideration is that I do not agree with the notion of "rolling back". That is, that for every migration there should be the opposite, anti-migration or "rollback" migration (e.g. add index->drop index).

This are meant to be a safety feature, such that if your migration caused damage, you would roll it back.

For some types of migrations this makes some sense (logically, though perhaps not in the time it takes to apply). But for others, it doesn't event make a logical sense. If you add column, then that column has data. If you drop it, you lose the data. If you re-add it, your data is already lost. If you add unique key then drop it, what happens to your constraints? Are you certain you'd be able to later re-add it? Or would the data be changed again and you'd need to manaully cleanup?

Moreover, if a migration goes bad due to wrong decision making, then I wouldn't trust the owner of the migration to provide the anti-migration in the first place. Even for machine generated migrations (which we use) there are flaws and limitations. And I'm yet to see a machine generated schema migration that knows how to properly handle partitioning and foreign keys.

My 2cents

ramkireddy commented 7 years ago

Thanks @shlomi-noach!, I was hoping that you guys must be using a sophisticated database migration tool by now, but I am not surprised either as these tools have their own drawbacks when trying to fit them into your process of applying database migrations.

Liquibase now supports JSON, YAML and SQL formats, earlier it used to only support XML format and we can also be extend it's functionality and privides API's and there is also a plugin to support pt-osc makes it an abvious choice to try out.

https://liquibase.jira.com/wiki/display/CONTRIB/Percona+Online+Schema+Change

I am not seeing any open issues with this plugin not sure if anyone is using this plugin for liquibase.....I will find out....looks like there are some recent bug fixes done for this software.

Thanks, Ramki

Slach commented 1 year ago

another tool https://github.com/bytebase/bytebase/releases/tag/1.2.2 now supports gh-ost