austinvalle
commented
9 months ago Hey there @remyleone 👋🏻 ,
I'm going to transfer this over to the codegen-spec repository for now, but having some guidance documentation for creating tools that work well in this design is definitely something we can do once we get further along.
I can expand a little more on some of the documentation we have about the codegen design that may be able to spark some discussion.
So to recap first, the general high level design for provider code generation looks like below:
So for this diagram, we are talking about how to create "Source Spec Generators", which as of today, there only exists the OpenAPI Spec Generator, which I will use as a reference for expanding on what a source spec generator does.
The overall goal of a "Source Spec Generator" is to convert one or more sources into one or more provider code specifications (which are just JSON files that satisfy this JSON schema). It's completely up to the design of the "Source Spec Generator" on what those sources are, it could be 1 proto file, multiple proto files, multiple proto files and a config, etc.
For the OpenAPI generator, the sources are 1 OpenAPI 3.x spec and 1 generator config
When you go to convert an IDL like OpenAPI, protobuf, or thrift into a provider code specification, you'll come across some common problems, how the "Source Spec Generator" solves those problems depends on who's writing the tool, who's using the tool, how many use-cases is the tool trying to solve, etc. Some things to consider:
- How will your generator determine what a Terraform resource or data source is? Naming conventions? Annotations/comments on the IDL? CLI arguments? Config file?
For the OpenAPI generator, we chose to go with a config file (generator config). We did try a POC that automatically discovered Terraform resources and data sources using RESTful naming conventions, but we weren't satisfied with the accuracy, so we opted to start with the config approach.
- How will your generator convert an IDL schema into a Terraform resource/data source schema? What IDL types map to the provider code spec types? How will you determine if an attribute is computed vs. optional vs. required?
The OpenAPI generator has an "overly detailed" 😆 design document that details exactly how it merges request bodies with response bodies, how OpenAPI types map to provider code types, how it determines computed/optional/required, etc.
- How will you handle edge cases that might arise in an IDL, like if an IDL description needs to be overridden to better fit a Terraform attribute description, or if an IDL supports multi-types, which the Terraform SDKs currently do not, but OpenAPI can 😆
The OpenAPI generator handles a few edge cases now using the generator config, but there are surely plenty more to handle.
These are just a few of the things we ran into building the OpenAPI spec generator, but depending on how generic you want your "Source Spec Generator" to be, some of these may not be difficult to solve, or may not be problems at all.
Once you've determined how to convert your IDL into a provider code spec, creating it is just creating a JSON file that satisfies this JSON schema. We provide Go bindings, but there is nothing stopping you from creating a "Source Spec Generator" in another language (javascript, python, etc.). You can always just use our JSON schema to ensure what you create is valid.
Once you've created a provider code spec JSON file, now you can use that with any tool that consumes that interface, like the Plugin Framework Code Generator.
These were just my off-the-head thoughts, loosely formatted 😆 , maybe it will be helpful for you or others to get some insight into our initial thoughts.
I think a more refined guidance document for creating "source spec generators" and consuming "provider code generators" will become clearer to write as we identify what works and what doesn't. 👍🏻
At the moment, only openapis is supported, while it is enough to explore the potentiality of this generator code, we would like to explore how complicated it would be to bring support for protobuf for this. Once you feel confident enough in your design choices for openapi, could you add documentation so that community/contributor could start exploring coding a support for other IDL ? (Protobuf, thrift, ...)