Support for golang migrations

[jwntrs]

We've been using migrate for a while and we recently ran into an issue where we needed some fairly complex logic that was only possible running custom migrations in golang. Has there been any consideration for supporting golang migrations? It's a strange case because it doesn't really fit the database model. In theory a migration written in golang could be run against any db backend.

It would be similar to the shell migrations that have been suggested but not implemented. It seems like @JensRantil has put a lot of thought into this idea. Is this something thats still on the radar?

[dhui]

I see the usefulness of running custom migrations in Go and/or bash and am not opposed it.

However, I don't have the bandwidth to implement it myself but will participate in design discussions and review the subsequent PR(s).

We may need to change the driver interface to support custom Go/bash migrations

[denismakogon]

Faced with the same problem where we need to run complex migrations involving data generation, ended up implementing our own migration framework.

[ahammond]

This would allow us to finally stop using goose. The non-static migrations could just be type .so golang plugins which implement an interface with an Up(input interface{}) and Down(input interface{}) method. Our driver for this is that we want migrations to be managed by our app, not a 3rd party binary, and we certainly don't want to be compiling code at runtime the way goose dynamic migrations do.

[denismakogon]

@ahammond It's worth starting a new framework. The amount of actual work that is needed here is huge (i'm not blaming this framework, no) comparing to the amount of work needed to extract migrations framework from here: https://github.com/fnproject/fn/tree/master/api/datastore/sql/migrations

[dhui]

Using Go plugins is an interesting idea! It looks like plugins are currently only support in Linux and Mac OS.

From: https://golang.org/pkg/plugin/

Currently plugins are only supported on Linux and macOS. Please report any issues.

I'm not familiar with Go plugins (yet), so I'm not sure sure if plugins will work across platforms. e.g. built on Mac OS but used/run on Linux.

Also, I'm not sure about the requirements for this feature. Do people want to run Go migrations with SQL migrations? e.g. run 5 SQL migrations, then 1 Go migration, followed by 3 SQL migrations.

A current work around (if your DB supports it) would be to use a temporary procedure or function to perform the more complicated logic that's hard to do in SQL. e.g. for postgres, there's: https://www.postgresql.org/docs/current/server-programming.html

[denismakogon]

Plugins are still quite raw technology, let's be honest. With plugins, you'd have a problem when your plugin is built for another OS, etc. I'd really stick with *.go migrations (more safe)

I'm not familiar with Go plugins (yet), so I'm not sure if plugins will work across platforms. e.g. built on Mac OS but used/run on Linux.

no, linux plugins for linux, mac plugins for mac

Also, I'm not sure about the requirements for this feature. Do people want to run Go migrations with SQL migrations? e.g. run 5 SQL migrations, then 1 Go migration, followed by 3 SQL migrations.

No, there's no point having both, only *.go matters and gives a lot of options around the migration process.

[eldad87]

@denismakogon can you please share an example on how to run it along this package? Thanks!

[ahammond]

Everything runs in docker containers these days. A user-interactive experience for these tools should be focused around debugging migrations, and to me, that means keeping the environment as close as possible to the production runtime environment. And that means a linux container.

[eldad87]

@ahammond, That's great, exactly as I thought. Nevertheless, a real life example will become handy.

[euskadi31]

see: Yaegi for dynamic plugin :)

[l0nax]

Is there any work going on implementing this feature?

[dhui]

None AFAIK, feel free to pick it up!

[mainpart]

I've ended up using this simple library that suit my needs https://github.com/xakep666/mongo-migrate

[eidins]

What about simply making it easy to extend the CLI so that I could do,

func main() {
  custom_migrations = map[string]Migration{
    "183747version": my_function_handle
  }

  migration.CustomCLI{
    migrations: custom_migrations
  }.run()
}

Then you build this, it works identical to the normal CLI, but the migrations take into account this function as one of the steps.

Seems like it would be fairly simple and no importing .so files. If people like this idea, I could look at implementing.

[kecci]

Is there any updates for migrations with go function / extension ? because I'm using goose right now.

[dangkaka]

hello any updates on this?