Exposed and more customisable golang generator

[popescu-af]

What do you want to change?

Recently I had to write a plugin for slight alterations of the default behavior of the golang generator for postgres. To build the plugin, I needed to

• copy over some of the source files from the internals of sqlc and adapt the imports

    internal
    ├── codegen
    │   ├── golang
    │   │   ├── driver.go
    │   │   ├── enum.go
    │   │   ├── field.go
    │   │   ├── gen.go
    │   │   ├── go_type.go
    │   │   ├── imports.go
    │   │   ├── mysql_type.go
    │   │   ├── postgresql_type.go
    │   │   ├── query.go
    │   │   ├── result.go
    │   │   ├── sqlite_type.go
    │   │   ├── struct.go
    │   │   ├── template.go
    │   │   └── templates
    │   │       ├── pgx
    │   │       │   ├── batchCode.tmpl
    │   │       │   ├── copyfromCopy.tmpl
    │   │       │   ├── dbCode.tmpl
    │   │       │   ├── interfaceCode.tmpl
    │   │       │   └── queryCode.tmpl
    │   │       ├── stdlib
    │   │       │   ├── dbCode.tmpl
    │   │       │   ├── interfaceCode.tmpl
    │   │       │   └── queryCode.tmpl
    │   │       └── template.tmpl
    │   └── sdk
    │       ├── sdk.go
    │       ├── utils.go
    │       └── utils_test.go
    ├── debug
    │   └── dump.go
    ├── inflection
    │   └── singular.go
    ├── metadata
    │   └── meta.go
    ├── opts
    │   ├── debug.go
    │   └── parser.go
    └── pattern
        └── match.go

• modify the internal/codegen/golang/templates/pgx/queryCode.tmpl to suit my needs
• add some code to insert some extra imports that I needed for the additional template code

In my opinion, although this works, it would be great if the sqlc's generators would be

• exported (not under internal/, which renders themt inaccessible) so that one could use them in their plugins

• more configurable - receive GeneratorOpts

  • e.g. func Generate(ctx context.Context, req *plugin.CodeGenRequest) (*plugin.CodeGenResponse, error) would become func Generate(ctx context.Context, req *plugin.CodeGenRequest, opts ...plugin.GenerateOption) (*plugin.CodeGenResponse, error)
  • some useful options, at least for me, would be
    • override one of the templates (queryCode, dbCode, etc.)
    • pass custom template functions, to be used by one's custom template
    • pass custom imports

  I'd be more than happy to implement this if it was to be selected. I need it for golang/pg, but I'd be willing to look into the rest of the combinations too.

  Thanks!

What database engines need to be changed?

PostgreSQL

What programming language backends need to be changed?

Go

[jwc-clinnection]

Do you mean something like this? https://github.com/kyleconroy/sqlc/pull/2302

[kyleconroy]

The majority of the code is kept in the internal package so that it's easy for us to change the internals of sqlc without causing backwards incompatible changes. Plugins have been supported for a few versions now. You can learn how to author a plugin and see two of our plugins here:

• https://github.com/sqlc-dev/sqlc-gen-kotlin
• https://github.com/sqlc-dev/sqlc-gen-python

I'm very hesitant to expose the templates publicly, as they're really difficult to maintain. There was little to no thought into how they're structured. I've talked to @jwc-clinnection about creating a plugin that makes it easy to bring your own templates, and it sounds like that would be helpful for you.

[popescu-af]

The majority of the code is kept in the internal package so that it's easy for us to change the internals of sqlc without causing backwards incompatible changes. Plugins have been supported for a few versions now. You can learn how to author a plugin and see two of our plugins here:

• https://github.com/sqlc-dev/sqlc-gen-kotlin
• https://github.com/sqlc-dev/sqlc-gen-python

I'm very hesitant to expose the templates publicly, as they're really difficult to maintain. There was little to no thought into how they're structured. I've talked to @jwc-clinnection about creating a plugin that makes it easy to bring your own templates, and it sounds like that would be helpful for you.

Makes sense. I checked the plugin section, but from what I could see the generator has to be written entirely for each plugin. Which in my case means implementing the default sqlc behavior with slight alterations. But I think the plugin you are mentioning would be perfect for my use case, looking forward to it.

[popescu-af]

Do you mean something like this? #2302

Yes seems to be similar.

[justinrixx]

I would find this useful as well.

My use-case is putting metric emission for timings into the generated code. This allows me to see db query timings easily on my operations dashboard.

Example:

func (c *Client) GetCatalogEntry(ctx context.Context, componentID string) (CatalogEntry, error) {
    var entry CatalogEntry

    defer instrument.NewMetricTimer("db.get-catalog-entry").Record()
    result := c.db.QueryRowContext(ctx, queryGetCatalogEntry, componentID)

    err := result.Scan(&entry.ProjectID, &entry.Active, &entry.CreatedAt)
    if err != nil {
        return CatalogEntry{}, errors.Wrap(err)
    }

    return entry, nil
}

The defer line starts a timer, then records it when this function completes. A workaround exists: give instrumentation responsibility to the callers, but I've been bitten before by forgetting to instrument every single call. The other workaround is to add a wrapper function for all of the generated functions which handles the metric emission and then calls the generated function, but this is also less than ideal.

[fdietze]

I just created a plugin to support custom templates: https://github.com/fdietze/sqlc-gen-from-template

Feedback appreciated.

For example, I'm wondering if there should be a template to generate one file per query. Or a giant template file which gets access to the whole GenerateRequest data structure?

edit: I setteled with giving the template access to the full GenerateRequest struct.