go-bindata support

[hsyed]

I am embedding migrations into the product I am working on. I could do with a go-bindata support in the Migrator. go-bindata generated files have no dependencies -- which fits with pgx / tern.

[jackc]

Have you looked at Migrator.AppendMigration? With that you wouldn't need go-bindata or to modify tern/migrate to have your migration data embedded in your program. I'm not totally opposed to making the file system loader more abstract, but I think what you want can be done without it.

[hsyed]

We are working on a multi-tenant meta-schema based access layer / incremental event processing engine for very specific use-cases -- the business logic is runtime installable into postgres (plv8).

There is a thin layer of DDL / stored procedures and JS framework that needs to be installed into the database. I say thin, but I suspect there will be quite a bit of it !

The JS framework will be written by a colleague who will be comping online shortly, I don't yet know what approach he has in mind for packaging the JS.

I have given it some thought and it might make more sense for us to generate go files with AppendMigration calls -- but that does mean being careful with versioning !

[hsyed]

I am almost done with getting go-bindata working -- what do you think of this (this depends on the refactoring I did in my PR) ? I can commit this into the PR tomorrow. Gonna hit the sack now !

// MigrationDir is a function that follows the semantics defined by the go-bindata AssetDir function:
// AssetDir returns the file names below a certain
// returns the file names below a certain
// directory embedded in the file by go-bindata.
// For example if you run go-bindata on data/... and data contains the
// following hierarchy:
//     data/
//       foo.txt
//       img/
//         a.png
//         b.png
// then AssetDir("data") would return []string{"foo.txt", "img"}
// AssetDir("data/img") would return []string{"a.png", "b.png"}
// AssetDir("foo.txt") and AssetDir("notexist") would return an error
// AssetDir("") will return []string{"data"}.
type MigrationDir func (string) ([]string, error)

// MigrationAsset is a function that follows the semantics defined by the go-bindata Asset function:
// Asset loads and returns the asset for the given name.
// It returns an error if the asset could not be found or
// could not be loaded.
type MigrationAsset func (string) ([]byte, error)

// recursive function to harvest the shared sql templates.
func traverseShared(l migrationLoader, dir MigrationDir, asset MigrationAsset, cwd string, sps []string) error {
    for _, name := range sps {
        filePath := filepath.Join(cwd, name)
        if sharedDirs, err := dir(name); err == nil {
            traverseShared(l, dir, asset, filePath, sharedDirs)
        } else if body, err := asset(filePath); err != nil {
            return err
        } else if err := l.loadShared(name, body); err != nil {
            return err
        }
    }
    return nil
}

type kv struct {
    key string
    value []byte
}

func (m * Migrator) LoadMigrationsFromGoBindata(base string, dir MigrationDir, asset MigrationAsset) error {
    if as, err := dir(base); err != nil {
        return err
    } else {
        toResolve := []kv{}
        loader := m.newMigrationLoader(base+"/")
        for _, name := range as {
            filePath := filepath.Join(base, name)
            // first check if the entry is a subdirectory, if it is harvest shared sql
            if shared, err := dir(filePath); err == nil {
                if err := traverseShared(loader, dir, asset, filePath, shared); err != nil {
                    return err
                }
            // otherwise it must be a migration at the root, in which case defer processing it till all
            // shared sql has been loaded
            } else if body, err := asset(filePath); err != nil {
                return err
            } else {
                toResolve = append(toResolve, kv{ key: name, value: body})
            }
        }
        // now resolve the actual migrations
        for k, v := range toResolve {
            if err := loader.load(k, v); err != nil { return err }
        }
        return nil
    }
}

[jackc]

I'm still mostly of the opinion that Migrator.AppendMigration is a sufficient integration point for non-file system migration sources, but that said, if you really feel the need for more integrated support there is an alternative abstraction that may be much cleaner than directly adding go-bindata support to the Migrator type.

A quick look at migrate.go indicates that FindMigrations and LoadMigrations are the only methods that directly work with the file system. Further inspection of those methods shows that the only operations they perform are directory listings and reading files. What if we had a pseudo-file system interface that had those two methods? The default implementation could read the real file system. An alternative go-bindata implementation could read from there. In this way the only changes to the Migrator type would be to use that interface instead of directly using the Go file system functions. This also shows the advantage of using a options struct for configuring Migrator. A file system interface value can be added to that struct without changing the API at all.

[hsyed]

I think I could use append migration as things stand now. However, I am storing the migrations in the bindata according to the naming conventions of tern, and I have put in the above / below string. Plus It would be nice to be able to rely on the template support if needed.

I did pursue the implementation you suggested at first since I didn't understand exactly what the two methods were doing. I'll give creating the filesystem abstraction a whirl.

[hsyed]

Hmm How about ?

type MigratorFS interface {
    ReadDir(dirname string) ([]os.FileInfo, error)
    ReadFile(filename string) ([]byte, error)
    Glob(pattern string) (matches []string, err error) // <-- easy enough to implement
}

Turns out it's literally a copy paste -- I love the interfaces in go.

[hsyed]

My colleague has come online and we have decided on bundling the JS framework as a single plv8_init function. So my needs are currently met by LoadMigration. I'd like to get the FS abstraction in though !

[bojanz]

If anyone is wondering how to integrate tern and vfsgen: https://github.com/runbilliam/billiam/commit/635b810074ce72c87027ed954f3454708590076c

The main part being:

import (
    "net/http"
    "os"

    "github.com/shurcooL/httpfs/path/vfspath"
    "github.com/shurcooL/httpfs/vfsutil"
)

// migratorFS wraps a http.Filesystem for usage with jackc/tern.
type migratorFS struct {
    fs http.FileSystem
}

// ReadDir implements the migrate.MigratorFS interface.
func (m migratorFS) ReadDir(dirname string) ([]os.FileInfo, error) {
    return vfsutil.ReadDir(m.fs, dirname)
}

// ReadFile implements the migrate.MigratorFS interface.
func (m migratorFS) ReadFile(filename string) ([]byte, error) {
    return vfsutil.ReadFile(m.fs, filename)
}

// Glob implements the migrate.MigratorFS interface.
func (m migratorFS) Glob(pattern string) (matches []string, err error) {
    return vfspath.Glob(m.fs, pattern)
}