matiasinsaurralde
commented
1 year ago @helderco I've been playing with the following idea:
- Extend
Generatorinterface to incorporate common methods. - Provide a
BaseGeneratordata structure that can implement "default" functionality for specific helpers, etc. - Make every generator embed
BaseGeneratorwhile also keeping generator-specific implementation in the context of their own data structure. This allows the following behavior:Generatorinterface establishes a method calledFormatName(string) string, this method gets implemented atBaseGeneratorlevel so that when a generator is introduced we rely on the default implementation of it. During the code generation process we callGenerator.FormatName, let's say the Rust generator needs a custom implementation for it, then it can be defined as(g *RustGenerator) FormatName(string) string, all other implementations that don't require customized code will rely on theBaseGenerator.FormatNameimplementation. I think this is good for most helpers and internal functionality. - I've kept
CommonFunctionsfor compatibility but I think we could migrate it completely or not. - A more complex
template.FuncMaphelper is implemented, it can take existingCommonFunctionsand generator-specific functions like:funcMap := generator.FuncMap(g, commonFunc, template.FuncMap{ "FieldOptionsStructName": g.fieldOptionsStructName, "FieldFunction": g.fieldFunction, "IsListOfObject": g.isListOfObject, "GetArrayField": g.getArrayField, "ToLowerCase": g.toLowerCase, "ToUpperCase": g.toUpperCase, "FormatArrayField": g.formatArrayField, "FormatArrayToSingleType": g.formatArrayToSingleType, }) - From the template execution side the function map we end up passing would look like:
template.FuncMap{ "FieldOptionsStructName": g.fieldOptionsStructName, // Go specific method - func(g *GoGenerator) fieldOptionsStructName(...) "FieldFunction": g.fieldFunction, // Go specific method - func(g *GoGenerator) fieldfunctio(...) "FormatEnum": g.FormatEnum, // Generic method, part of BaseGenerator - func(b *baseGenerator) FormatEnum(...) } - The code from the experiment is here, it does generate code that looks 100% like the original implementation. I think it could be a nice starting point to improve the common layer, let me know your thoughts. Still doesn't tackle the template improvements though.
helderco
wingyplus
TomChv
Summary
Refactor the Go based codegen tooling to make it easier to maintain the codegen feature in a growing list of SDKs.
Background
The Go SDK launched with a custom implementation of GraphQL introspection to generate the Dagger client based on the API schema. This is similar to the GraphQL Code Generator project, but uses Go code rather than Javascript.
Python needed to be released shortly after but the Go introspection wasn’t ready yet to be shared with other SDKs. As Python’s GraphQL client already provides the introspection, it was faster to do the code generation in Python which was done in a day.
With Node.js right after, we were still trying to decide if it was easier to do it with Javascript, but sharing the workload allowed another Go developer to help out by modularizing the Go introspection to serve both Go and Node.js.
Motivation
While adding support for
Container.syncin the API I noticed that It makes more sense now for every SDK to use the same Go based codegen:Converge to using the Go based codegen
At the time the SDKs were launched, codegen was simpler. However, features kept being added over time, requiring more maintenance. Now, expanding to more languages like Rust and Elixir, there’s even more different implementations of codegen, making it harder to keep feature parity in sync.
While initially doing the Python codegen in Python allowed for a faster iteration, and allowed access to some Python APIs like detecting if a name is a reserved keyword, it’s now a bigger problem to not share common Dagger specific functionality requiring unnecessary duplication.
While adding enum support requires knowledge in each language to extend the SDK’s client, there’s also decisions like knowing when to convert an ID into an object or vice versa, knowing when a field should make a request vs be lazy, and even possibly implementing custom error types. Putting these in a common abstraction layer reduces the burden on each SDK.
Improve the common layer
Even between Go and Node.js, since both use Dagger’s Go codegen, there’s still unnecessary duplication. That’s because Node.js SDK’s functions and templates were added without much change to Go SDK’s.
I saw several places where this can be improved, by moving everything that has to be in all SDKs to the common layer, as much as possible and in an agnostic and granular way. It’s the same idea as our own SDK clients, that are dagger specific abstractions on top of more general query builders. In the same way, this common layer should be a dagger specific abstraction on top of the general GraphQL introspection layer.
There’s already a few common functions but I suggest taking it to another level.
Simplify templates
I think working with the templates is a bit difficult at the moment. There’s too much logic in some, or two spread out in different files.
There’s too much noise with indentation and general whitespace, with a lot of
ifs within afor, within aif... Some templating languages terminate with{% endfor %}and{% endif %}, but Go uses the same{{ end }}. At least there could be a comment on every{{ end }}that references what’s the corresponding opening statement when it’s not immediately obvious due to the distance between opening and closing.Templates should be as dumb as possible and easy to read and maintain. Sometimes even not worth the trouble.
High-level design
There’s no formal specification for this yet, but at a high-level:
[^1]: This is not directly related to the Go codegen tooling but it helps with the codegen maintainability in all SDKs individually.
I want to migrate Python’s codegen to Go either as a part of this process or as a consequence of it, bringing all supported SDKs under the same tooling. Other SDKs to hopefully follow, especially when considering maturing out of experimental, or as a new addition to the repo.
Common abstraction layer
The Go codegen implements the visitor pattern as used in the GraphQL Code Generator project (but using Go instead of Javascript, and not caring about operations). Between the GraphQL introspection and the templates however, there’s a lot of Dagger specific decisions and data manipulation that can be abstracted away in a Dagger abstraction layer.
So rather than using the visitor pattern at the GraphQL introspection layer, do as much data processing as possible for anything that’s shared between all SDKs, and visit the processed data instead.
Templates
Some top-of-mind improvements:
TypeReforFieldin the template, process the data in code and send the ready to use values to the template. Most of this should already be handled by the common abstraction layer though.Boilerplate
Add helpers for common patterns, like delegating to templates, hooking in the needed naming conventions or anything else that helps reduce the boilerplate for an SDK.
Rationale
An advantage for using Go here is that any change that requires re-generating clients comes more often from changes to the API which is implemented in Go. There’s been a lot of cases where a new feature is added to the API which requires a change in all codegen solutions. If the same developer adapts the common abstraction layer accordingly, it should be much simpler to add support in each language.
The “entrypoint” for an SDK should be defining the functions that are to be visited. Sometimes it’s much simpler to just produce the needed output in the function rather than delegating to a template. In other words, using templates is not a requirement, only use them if they add clarity and are easier to maintain. Similarly, if something else instead of Go templates or functions is easier to maintain, we could also have a helper to hook that into the visitor “life-cycle”.
The most important idea here is to centralize as much as possible the common parts between every SDK, and help make it easier to maintain in a growing list of SDKs.