Improve docs on how to `shim` a provider hidden by `/internal/`

TieyanFu commented 3 months ago

What happened?

Trying to create a bridged terraform provider for the BTP terraform provider

https://github.com/SAP/terraform-provider-btp/

I pushed my latest changes here

https://github.com/TieyanFu/pulumi-btp/tree/issue_111

In step make tfgen, I got following error:

pulumi plugin install resource random 4.3.1
(cd provider && go build -o /Users/d049740/Work/k8s/workspace/TerraformProvider/pulumi-btp/bin/pulumi-tfgen-btp -ldflags "-X github.com/tieyanfu/pulumi-btp/provider/pkg/version.Version=0.0.1-alpha.1710887702+9ff41a2f.dirty" github.com/tieyanfu/pulumi-btp/provider/cmd/pulumi-tfgen-btp)
package github.com/tieyanfu/pulumi-btp/provider/cmd/pulumi-tfgen-btp
    imports github.com/tieyanfu/pulumi-btp/provider
    resources.go:31:2: use of internal package github.com/SAP/terraform-provider-btp/internal/provider not allowed
make: *** [tfgen] Error 1

The error is understandably correct, since the BTP terraform plugin provider is inside folder internal. However, the folder structure is following the latest terraform provider scaffolding framework template where provider is also inside internal folder.

I had look into a few bridged provider examples:

terraform-provider-xyz, has put the provider package under the root folder of its repo.
pulumi-azure: has put the provider package inside internal folder, but in https://github.com/pulumi/pulumi-azure/blob/master/provider/resources.go, it did not references provider package, instead only references two packages, and yet shim package is also under internal folder internal/services/storage/shim

    "github.com/hashicorp/terraform-provider-azurerm/helpers/azure"
    "github.com/hashicorp/terraform-provider-azurerm/shim"

pulumi-splunk, its "provider" pacakge is named splunk which is also not under internal folder

A quick note: in provider/resources.go it seems the pulumi is trying to reference datasource definition, not necessarily provider itself, not sure if I understand it correctly.

So the questions are:

What are the recommended project structure for terraform-plugin in order to use pulumi bridge tf provider?
If the provider package in the terraform-plugin is already inside internal folder, how should it be configured in resources.go as described in below step?

Example

see above

Output of `pulumi about`

CLI Version 3.111.1 Go Version go1.22.1 Go Compiler gc

Host OS darwin Version 14.4 Arch x86_64

Backend Name pulumi.com URL https://app.pulumi.com User Unknown Organizations Token type personal

Pulumi locates its logs in /var/folders/xp/w_dp2zwj1v31p7rsl_qr0zbr0000gn/T/ by default warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from /Users/dxxxxx/Work/k8s/workspace/TerraformProvider/pulumi-btp). If you have not created a project yet, use pulumi new to do so: no project file found warning: Failed to get information about the current stack: no Pulumi.yaml project file found (searching upwards from /Users/xxxxx/Work/k8s/workspace/TerraformProvider/pulumi-btp). If you have not created a project yet, use pulumi new to do so: no project file found

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

TieyanFu commented 3 months ago

Just found out that there is a section discuss the shim pattern. However, there are still some hoops to jump through. https://github.com/pulumi/pulumi-random/blob/master/provider/go.mod provides a good example.

iwahbe commented 3 months ago

Hey @TieyanFu. Thanks for raising this. There are definitely hoops to jump through. It sounds like you were able to get unblocked. I'm going to leave this issue as a reminder that we need to improve our documentation here.

It might be worth checking out https://github.com/tmeckel/pulumi-tf-provider-cookiecutter here. It's not officially supported, but it will handle this type of setup for you. It's much closer to "it just works" then the boilerplate is right now.

pulumi / pulumi-tf-provider-boilerplate