Protobuf generators are not compatible with new resources

[hugoShaka]

Everything was designed around the protobuf messages as the source of truth.

Generating CRDs from proto now involves an increasing pile of hacks and manual overrides that is not maintainable. Generating terraform schemas from proto is not possible with the latest created resources.

The generator must use the source of truth as input. Currently, the source of truth is the Teleport go type.

Details

Protobuf does not match the internal go types anymore

We have two kinds of structure holding a resource:

• the protobuf ones, used solely for transport between the client and the server
• the "real" ones, passed to the teleport client, the one we use for most of the logic in Teleport, usually we also unmarshal the user input in it directly

With gogo, we generated the "go" struct from the protobuf. This is not true anymore; since the change, the "real" structure schema does not match the protobuf schema. This can be because the names are different (Header vs ResourceHeader), because some formats are easier to serialize and send over protobuf (map[string]X becomes []struct{key string, value X}), or by choice of the developer implementing the resource.

Both CRD and terraform schema generators read the protobuf to generate their schemas. However, those integrations receive input from the user and pipe it to a Teleport client. This is the role of the "real" structures, not the Proto-generated ones. In the best case we have to maintain a lot of manual overrides to correct the drifts between the proto and the "real" structs. In the worst case we can't use the generated code at all.

The terraform schema generator expects specific structures

The protoc-gen-terraform plugin generates code that take care of the copy and conversion from the terraform schema to the teleport types. It makes assumptions like the teleport types representing a message will be a pointer to a struct (this is what the protoc-gen-go does). Since we're not generating the teleport types with protoc-gen-go anymore, there's no reason to have pointers anymore.

For example the gogo-generated struct was

type MyResource struct {
    Spec *MyResourceSpec
}

but now we write

type MyResource struct {
    Spec MyResourceSpec
}

The generator produces the following code:

if obj.ResourceHeader == nil {
    obj.ResourceHeader = &v1.ResourceHeader{}
} 

This code is working for the proto-generated struct but is illegal for the new hand-crafted struct.

[hugoShaka]

cc @marcoandredinis @tigrato

[zmb3]

@rosstimothy does RFD 153 cover this? Is there any other work left to do or should we close this?

[hugoShaka]

RFD 153 does not solve this issue. It makes it more outstanding now that every resource must have its IaC but we don't have the tooling to generate it properly.