Thinking: inference of import paths and package names

[zombiezen]

I don't have a good solution for this yet, but it has been suggested to me that capnpc-go can/should be inferring the import path and package name for the generated code. This can certainly be introduced in a backward-compatible way, since we currently require import paths and package names to be explicitly annotated, so this is just relaxing the rule. Note that solving this would make #41 obsolete.

Experience has shown me that finding the appropriate Go package boundaries is challenging. Finding a good solution should address the following three use-cases:

1. You have a bunch of schemas in one directory, but they're probably separate packages. For example, the schemas included in capnproto (e.g. schema.capnp, persistent.capnp, rpc.capnp). If foo.capnp is located at $root/foo.capnp, the generated Go file should be placed at $root/foo/foo.capnp.go.
2. You have a bunch of schemas in one directory, but they're tightly coupled and should be one package. If foo.capnp is located at $root/foo.capnp, the generated Go file should be placed at $root/foo.capnp.go.
3. You are writing a Go project from scratch and so you place one schema per directory. If foo.capnp is located at $root/foo.capnp, the generated Go file should be placed at $root/foo.capnp.go. Note that this is a strict subset of the second use-case.

[zenhack]

Proposal: extend the code generator to accept a few command line flags. We can then distinguish the above cases by passing e.g. -onepkg or -manypkgs or such.

This would require inverting the control flow with the capnp compile command, but it's easy enough to just invoke it as capnp compile -o - and then read from its stdout.

We can derive the import path from wherever the generated files end up, relative to $GOPATH. The package name can be derived from the last path segment, with some mangling rules to ensure it's a valid name. We can deal with duplication by having the generated packages always give their imports names, so they don't have to care what the "real" package name is. The way those names are generated is less critical, since developers won't have to type them very much.

Inverting the control flow with the capnp compiler would also open the door for some other nice things, like adding -I $GOPATH/src/zombiezen.com/go/capnproto2/std by default.

[zombiezen]

The problem with the command-line flag approach is that other files need to know the file's import path. In seeing the complexity from Go's protoc interaction, I'm thinking that the explicit import path is a good idea. Perhaps just making the output file path logic smarter would help here.

[zenhack]

Yeah, maybe the way to go is to output the files to $GOPATH/src/$specified_import_path. We could still infer a package name, and deriving the output dir from the import path solves the various use cases you have above.

It would still be nice to not have to patch every schema to use it with Go.

[zenhack]

Was thinking about this again recently, having tackled the path inference problem in the Haskell implementation I'm working on. I'm dubious of the importance of (2); all of the schema I've seen have been variants (1) or (3). And I think people doing (3) will fall into one of two cases:

• Compatibiltiy with other languages is not important, so adding an annotation is no big deal.
• Compatibility with other languages matters. In this case they can just put all of their schema in the parent directory. This seems like it would be less annoying to people than having to wrangle patching the schema for other languages.

I'm tempted to suggest that we just require an annotation to do anything but (1). Since we currently require import paths for everything, this is backwards compatible. We require schema to be under $GOPATH, and the import path to $GOPATH/path/to/schema/foo.capnp is $GOPATH/path/to/schema/foo. We derive the package name from the last component of the import path.

Thoughts?

[zenhack]

The discussion in #122 got me thinking again about this issue, but I'd mis-remebered the state of the discussion. I'm still curious as to your thoughts on the above.

[zombiezen]

With the upcoming Go Modules and removal of GOPATH, I'm less sure of the inference now. If there's a way to get your plan to work in the context of the new source layout, I think I support it, but I haven't thought through this in much depth.

[zenhack]

I think it works with modules without much modification -- just instead of looking for where you are inside $GOPATH, you find the nearest go.mod and assume you're relative to the module specified there.