Generated 0 resources

[waldner]

What happened?

Trying to generate a provider based on https://registry.terraform.io/providers/vmware/vcd/3.7.0. Following the guide at https://github.com/upbound/upjet/blob/main/docs/generating-a-provider.md, here's the relevant Makefile section, which looks correct to me:

export TERRAFORM_PROVIDER_SOURCE := vmware/vcd
export TERRAFORM_PROVIDER_REPO := https://github.com/vmware/terraform-provider-vcd
export TERRAFORM_PROVIDER_VERSION := 3.7.0
export TERRAFORM_PROVIDER_DOWNLOAD_NAME := terraform-provider-vcd
export TERRAFORM_NATIVE_PROVIDER_BINARY := terraform-provider-vcd_v3.7.0
export TERRAFORM_DOCS_PATH := website/docs/r

I customized internal/clients/vcd.go with the configuration keys as follows:

                ps.Configuration = map[string]any{
                        "user":     creds["user"],
                        "password": creds["password"],
                        "org":      creds["org"],
                        "url":      creds["url"],
                }

                // optional values
                for _, n := range []string{"vdc", "auth_type", "allow_unverified_ssl", "max_retry_timeout"} {
                        if v, ok := creds[n]; ok {
                                ps.Configuration[n] = v
                        }
                }

                return ps, nil

I want to generate all resources, to I just ran: make generate. While no error is produced, no resource is generated, either, see the attached logfile.

I was able to do the same thing above with terrajet, so perhaps I'm missing something obvious here. Thanks.

log.txt

[ulucinar]

Hi @waldner, I believe it's expected that no resources are generated because my understanding is that you still need to supply external-name configurations for your managed resources to be generated here.

Upjet has the notion of a provider configuration, which you can provide via the provider configuration API. In that API, we have the config.Provider.IncludeList that specifies a list of regular expressions. If a Terraform resource from the Terraform provider's schema matches any regex from this list, then we attempt to generate a corresponding Crossplane managed resource. And if you provide an external-name configuration for a Terraform resource via config. ExternalNameConfigs, then it's included and a corresponding managed resource is generated.

You may also want to check our resource configuration guide here. And here's another one detailing the resource configuration API.

[waldner]

Hi @ulucinar ,

thanks for your reply. It turns out that, coming from terrajet, my understanding of external-name was wrong. I now see that resources do get generated when I add ExternalNameConfigs.

Any way to use regular expressions or patterns in ExternalNameConfigs? For example, if you know in advance that all resources will use the config.NameAsIdentifier as naming schema.

Thanks!

[ulucinar]

Hi @waldner, It's my pleasure! Yes, it's possible to specify a default configuration for your resources. But even if you do so, you will still need to specify the IncludeList. For example, in upbound/provider-aws, we are utilizing the config.ExternalNameconfigs map to populate the IncludeList.

However, you do not need to know and specify each Terraform resource's name at compile time. config.Provider.IncludeList expects an array of regular expressions, so something like the following in your config.GetProvider function should do the job:

    pc := config.NewProvider([]byte(providerSchema), resourcePrefix, modulePath, []byte(providerMetadata),
        config.WithIncludeList([]string{`.*`}),
...

And if you need to skip generation of any resources (for example, you may hit some bugs), then you can specify a SkipList with the config.WithSkipList, which again expects an array of regular expressions.

And in order to define a default resource configuration, you can use the config.WithDefaultResourceOptions, though which you can configure the default resource configuration options to be applied to all generated resources. This is how you can configure a default external name configuration to be applied to all resources. Looks like config.NameAsIdentifier is the default external name configuration.

[waldner]

Hi @ulucinar,

many thanks! I just have another question...the autogenerated code in provider.go does this:

func GetProvider() *ujconfig.Provider {
        pc := ujconfig.NewProvider([]byte(providerSchema), resourcePrefix, modulePath, []byte(providerMetadata),
                ujconfig.WithIncludeList(ExternalNameConfigured()),       // <<--------------
                ujconfig.WithDefaultResourceOptions(
                        ExternalNameConfigurations(),
                ))
...

(comment is mine) where, as you said, ExternalNameConfigured() explicitly includes all resources declared in the ExternalNameConfigs in external_name.go. But then, it looks to me that by default, the include list is already .+ (see https://github.com/upbound/upjet/blob/main/pkg/config/provider.go#L206), so to include all resources I could just remove the marked ujconfig.WithIncludeList(ExternalNameConfigured()) call in provider.go, right? Is my understanding correct?

EDIT: it looks like I'm correct, as I just tried it and I get:

...
12:18:10 [ .. ] go generate linux_amd64

Generated 70 resources!
12:18:26 [ OK ] go generate linux_amd64
12:18:26 [ .. ] go mod tidy
12:18:26 [ OK ] go mod tidy

EDIT2: Obviously, this was just for understanding purposes. It's clear that as soon as even one resource needs custom config, this is not a viable solution.

[ulucinar]

Hi @waldner, Thank you for checking it. As you mentioned, we still generally need custom (resource) configuration for the resources being generated (at least the external name configuration). Our resource configuration guide gives an overview of the configuration process, and here you can find a more detailed account.

Upjet provides a default configuration for all the generated resources but sometimes with that default configuration, a generated managed resource may not be functioning. And at some other times, we would like to have higher levels of conformance to the Crossplane resource model (e.g., managed resource API normalization, cross resource references, etc.) and thus, we do some custom configuration.