Starting small

[gowalk0012]

I'm checking how practical it would be to convert an existing code base to Go and the DB interface is the most challenging part by far.

YouTube no doubt appears as an excellent source to rely on in that space but Vitess addresses an advanced scenario which might be an overkill for a young system.

Would it be possible to start using parts of the package like the mysql db adapter with connection pool and be ready to add further components along as the system grows?

[enisoc]

If you're writing in Go, it's certainly possible to re-use our mysql package (a thin CGO wrapper around the C mysql client) and connection pools. These are both pretty generic tools for interacting with mysqld from Go.

Because they're so generic though, it probably won't make it any easier if you later decide to move to Vitess proper. The minimum setup to be ready to easily scale later with Vitess would be one instance each of etcd (or zk), vtgate, vttablet, and mysqld -- vtctld is also helpful but not strictly necessary. When you're small, these could all be on the same machine.

You would then use the vtgate client to send queries, which would insulate you from any future scaling -- you could add replicas, keyspaces, shards, HA failovers, etc. as needed without any changes in your app (and with no maintenance downtime). In the meantime, vttablet's connection pooling and rowcache (automatic row-based memcache) will help you get the most out of that single instance before you need to add more.

[gowalk0012]

Thanks. I'll probably test both "CGO/Connection Pools" and the "all-in-one" setup. Is there already an independent package that provides a simple query api which abstracts the CGO/connection pool management or would I need to write this myself?

[enisoc]

You'll probably want to use the dbconnpool package.

There's an example of how to get started with it in the docs for DBConnectionCreator. Here it is with some better formatting and a few more details filled in:

// Create a connection pool and open it with a set of connection params.
pool := dbconnpool.NewConnectionPool("name", 10, 30*time.Second)
mysqlStats := stats.NewTimings("Mysql")
connParams := &sqldb.ConnParams{
  Host: ...
  Port: ...
  Uname: ...
  Pass: ...
  ...
}
pool.Open(dbconnpool.DBConnectionCreator(connParams, mysqlStats))

// Get a connection from the pool.
conn, err := pool.Get()
// Do something with it.
conn.ExecuteFetch(...)
// Return it to the pool.
conn.Recycle()

[gowalk0012]

Thanks again. I noticed package db which appears to expose an even simpler interface in particular Exec taking parameters. Is there an additional package that abstracts prepared statements and dealing with query results?

[enisoc]

The go/db package actually should be deleted. It's just an interface, and it isn't imported or implemented by any other package anymore. It was used for the old V1 API where the app talks directly to vttablet. In the current V2 API, the app talks to vtgate (with package vtgateconn), which routes to the vttablets.

Unfortunately, I don't think our version of prepared statements (bindvars) is abstracted enough to be used separately from vttablet.

[gowalk0012]

I understand... I'll try to figure it out.

It's just that with Go database/sql you need to deal with two packages that are independently maintained (e.g framework + driver) and I don't see the scale-out path, where with Vitess I get a one-stop shop and starting to work with the underlying database interface logic can at least help to understand and support a bigger configuration.

So just to put my two cents in https://github.com/youtube/vitess/issues/819 I consider Youtube Open Source to be "Radio.. oops,, CodeShack" and I prefer it stays that way.. just let me pick one kit at a time :)