search

crossplane / terrajet

Generate Crossplane Providers from any Terraform Provider
https://crossplane.io
Apache License 2.0
289 stars 38 forks source link

Issues of generating provider-jet-alibaba #227

Closed zzxwill closed 2 years ago

zzxwill commented 2 years ago

What happened?

I tried to generate a Terraform provider in the repo https://github.com/zzxwill/provider-jet-alibaba and it failed.

➜  /Users/zhouzhengi/Programming/golang/src/github.com/zzxwill/provider-jet-alibaba git:(main) ✗ make generate
16:42:06 [ .. ] verify dependencies have expected content
all modules verified
16:42:21 [ OK ] go modules dependencies verified
16:42:22 [ .. ] go generate darwin_amd64
go: downloading github.com/turkenh/terraform-plugin-sdk v1.17.2-patch1
# github.com/hashicorp/terraform-plugin-sdk/internal/configs/configload
../.work/pkg/pkg/mod/github.com/turkenh/terraform-plugin-sdk@v1.17.2-patch1/internal/configs/configload/loader_snapshot.go:52:29: cannot use fs (type snapshotFS) as type afero.Fs in argument to configs.NewParser:
    snapshotFS does not implement afero.Fs (missing Chown method)
../.work/pkg/pkg/mod/github.com/turkenh/terraform-plugin-sdk@v1.17.2-patch1/internal/configs/configload/loader_snapshot.go:57:28: cannot use fs (type snapshotFS) as type afero.Fs in field value:
    snapshotFS does not implement afero.Fs (missing Chown method)
../.work/pkg/pkg/mod/github.com/turkenh/terraform-plugin-sdk@v1.17.2-patch1/internal/configs/configload/loader_snapshot.go:200:5: cannot use snapshotFS{} (type snapshotFS) as type afero.Fs in assignment:
    snapshotFS does not implement afero.Fs (missing Chown method)
apis/generate.go:32: running "go": exit status 2
16:42:39 [FAIL]
make[1]: *** [go.generate] Error 1
make: *** [generate] Error 2

How can we reproduce it?

  1. Clone https://github.com/zzxwill/provider-jet-alibaba.git
  2. run make generate
turkenh commented 2 years ago

@zzxwill adding the following replace to your go.mod should help:

replace github.com/spf13/afero => github.com/spf13/afero v1.2.2
zzxwill commented 2 years ago

@turkenh Thanks. I have added the line. But I hit another issue.

➜  provider-jet-alibaba git:(main) ✗ make generate
17:59:58 [ .. ] verify dependencies have expected content
all modules verified
18:00:42 [ OK ] go modules dependencies verified
18:00:45 [ .. ] go generate darwin_amd64
panic: interface conversion: interface {} is schema.ValueType, not schema.ValueType (types from different packages)

goroutine 1 [running]:
github.com/crossplane/terrajet/pkg/types/conversion.toV2Schema(0xc000a077c0, 0xc000bfd020)
    /Users/zhouzhengxi/Programming/golang/src/github.com/zzxwill/provider-jet-alibaba/.work/pkg/pkg/mod/github.com/crossplane/terrajet@v0.3.2/pkg/types/conversion/conversion.go:82 +0x5c5
github.com/crossplane/terrajet/pkg/types/conversion.toV2Resource(0xc00085abe0, 0xc0010c0c90)
    /Users/zhouzhengxi/Programming/golang/src/github.com/zzxwill/provider-jet-alibaba/.work/pkg/pkg/mod/github.com/crossplane/terrajet@v0.3.2/pkg/types/conversion/conversion.go:41 +0xf7
github.com/crossplane/terrajet/pkg/types/conversion.GetV2ResourceMap(0x7ff35f8, 0xc000454280, 0xc000ab4180)
    /Users/zhouzhengxi/Programming/golang/src/github.com/zzxwill/provider-jet-alibaba/.work/pkg/pkg/mod/github.com/crossplane/terrajet@v0.3.2/pkg/types/conversion/conversion.go:32 +0x117
main.main()
    /Users/zhouzhengxi/Programming/golang/src/github.com/zzxwill/provider-jet-alibaba/cmd/generator/main.go:48 +0x7d
exit status 2
apis/generate.go:32: running "go": exit status 1
18:02:31 [FAIL]
make[1]: *** [go.generate] Error 1
make: *** [generate] Error 2
zzxwill commented 2 years ago

@zzxwill The repo for Alibaba Cloud https://github.com/zzxwill/provider-jet-alibaba.git is the latest with your suggested fix.

bretagne-peiqi commented 2 years ago

@turkenh hello, could we fix this bug in this days. we are also evaluating this solution to generate provider-jet-alibaba

bretagne-peiqi commented 2 years ago

@zzxwill The repo for Alibaba Cloud https://github.com/zzxwill/provider-jet-alibaba.git is the latest with your suggested fix.

hello, which terraform-provider for alicloud you use as the original plugin ?

bretagne-peiqi commented 2 years ago
panic

I meet the same problem

turkenh commented 2 years ago

Sorry, I missed this issue probably due to the title is no longer relevant.

Just reproduced and debugged on my side a bit and it seems to be the problem is the terraform provider using a very old SDK version, v1.4.0, where there is v1.17.2 available as the latest v1 version.

I believe this issue could still be fixed with some workaround without the terraform provider upgrading the version, however, I don't think it is worth spending time on it.

Importing terraform provider codebase is already something problematic and something that we are about to deprecate. So, I will hold off trying to find a workaround for this issue and will close this once I verify generating the Alibaba provider with the new approach is possible.

turkenh commented 2 years ago

@zzxwill and @bretagne-peiqi just merged the said changes. I've also tested the code generation part with provider-alibaba but couldn't test at runtime as I don't have an account.

I would suggest you start fresh following the updated (simplified) provider generation guide and the latest provider-jet-template repo.

Also, I would love to hear any issues/feedback regarding the new experience as you will be the first ones testing this approach other than me 😊

zzxwill commented 2 years ago

@turkenh Great, I will try. Thanks.

bretagne-peiqi commented 2 years ago

@zzxwill Hi, have you tested the new version. does it work well? we are planning to use it as our solution

bretagne-peiqi commented 2 years ago

@turkenh, hello, what's the purpose for config/schema.json; I compared Crossplane/alibaba-provider with generated Alibaba-provider using terrajet. I don't understand how the generated provider works. It lacks update/create/delete methods ?? besides, I don't understand how clients connect and communication with alicloud APIs, I don't see the methods.

thanks.

turkenh commented 2 years ago

@turkenh, hello, what's the purpose for config/schema.json; I compared Crossplane/alibaba-provider with generated Alibaba-provider using terrajet. I don't understand how the generated provider works. It lacks update/create/delete methods ?? besides, I don't understand how clients connect and communication with alicloud APIs, I don't see the methods.

thanks.

Does this mean you could successfully generate? 🙂

I would suggest you check our Terrajet deep dive series to learn more about how it works. For example, you can check this section to understand the controller part.

zzxwill commented 2 years ago

@bretagne-peiqi @turkenh Everything looks fine except failing to generate resources (CRD) for newly added Terraform resources alicloud_vpc and alicloud_oss_bucket. Did I miss any configuration?

The new test repo is https://github.com/zzxwill/provider-jet-alibaba2.git, and the result of make generate is as below.

➜  provider-jet-alibaba2 git:(main) ✗ make generate
10:56:51 [ .. ] generating provider schema for hashicorp/null 3.1.0
10:56:54 [ OK ] generating provider schema for hashicorp/null 3.1.0
10:56:54 [ .. ] verify dependencies have expected content
all modules verified
10:56:59 [ OK ] go modules dependencies verified
10:56:59 [ .. ] go generate darwin_amd64

Generated 0 resources!
10:57:20 [ OK ] go generate darwin_amd64
10:57:20 [ .. ] go mod tidy
10:57:23 [ OK ] go mod tidy
turkenh commented 2 years ago

@zzxwill did you follow all steps in the guide till make generate? Asking because these lines should have been replaced with your provider.

zzxwill commented 2 years ago

@zzxwill did you follow all steps in the guide till make generate? Asking because these lines should have been replaced with your provider.

No, I thought you made a mistake in the guide as the syntax is wrong for export and I was about to raise an issue.

export TERRAFORM_PROVIDER_SOURCE=aliyun/alicloud
export TERRAFORM_PROVIDER_VERSION=1.157.0
export TERRAFORM_PROVIDER_DOWNLOAD_NAME=terraform-provider-alicloud
export TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX=https://releases.hashicorp.com/terraform-provider-alicloud/1.157.0

Sorry, let me change the makefile. Thanks.

turkenh commented 2 years ago

No worries. I agree that it looks like it is expected to be pasted to the terminal unless you read instructions carefully, I was also caught in the same trap couple of times :)

Feel free to open a PR if you have any idea to make it more explicit.

zzxwill commented 2 years ago

@turkenh I successfully created resources with your guide, thanks. By testing with Alibaba Cloud VPC, it succeeded.

➜  providerconfig git:(main) ✗ k get vpcs.vpc.alicloud.jet.crossplane.io
NAME      READY   SYNCED   EXTERNAL-NAME               AGE
example   True    False    vpc-2zegc9hgtyeyis96d7bov   21m
➜  providerconfig git:(main) ✗ aliyun vpc DescribeVpcAttribute --VpcId vpc-2zegc9hgtyeyis96d7bov
{
    "CidrBlock": "172.16.0.0/12",
    "CloudResources": {
        "CloudResourceSetType": [
            {
                "ResourceCount": 1,
                "ResourceType": "VRouter"
            },
            {
                "ResourceCount": 1,
                "ResourceType": "RouteTable"
            }
        ]
    },
    "CreationTime": "2022-02-25T07:44:24Z",
    "Description": "",
    "Ipv4GatewayId": "",
    "Ipv6CidrBlock": "",
    "IsDefault": false,
    "NetworkAclNum": 0,
    "OwnerId": 1874279259696164,
    "RegionId": "cn-beijing",
    "RequestId": "F8AAAB31-75FA-5DBB-B86F-1599AC96A408",
    "ResourceGroupId": "rg-acfm3s5kf5r6yja",
    "SecondaryCidrBlocks": {
        "SecondaryCidrBlock": []
    },
    "Status": "Available",
    "SupportIpv4Gateway": false,
    "UserCidrs": {
        "UserCidr": []
    },
    "VRouterId": "vrt-2zekj3mo5ryh2kqp0jvu8",
    "VSwitchIds": {
        "VSwitchId": []
    },
    "VpcId": "vpc-2zegc9hgtyeyis96d7bov",
    "VpcName": "vpc-2zegc9hgtyeyis96d7bov"
}
bretagne-peiqi commented 2 years ago

@turkenh @zzxwill do we have a fix format for resource template ? for example alicloud_instance, alicloud_cs_kubernetes.

Screen Shot 2022-02-25 at 4 32 39 PM

how can we create these config files quickly without looking into apis...

zzxwill commented 2 years ago

@bretagne-peiqi Generally, these docs/tables can help crossplane/crossplane#2962.

But in fact, it's pretty difficult to set those values for complex cloud resources, like ecs/rds/ack, even we checked the usage docs/tables very carefully. In Terraform, we use preconfigured Terraform modules to simply the properties set.

turkenh commented 2 years ago

@turkenh I successfully created resources with your guide, thanks. By testing with Alibaba Cloud VPC, it succeeded.

➜  providerconfig git:(main) ✗ k get vpcs.vpc.alicloud.jet.crossplane.io
NAME      READY   SYNCED   EXTERNAL-NAME               AGE
example   True    False    vpc-2zegc9hgtyeyis96d7bov   21m
➜  providerconfig git:(main) ✗ aliyun vpc DescribeVpcAttribute --VpcId vpc-2zegc9hgtyeyis96d7bov
{
  "CidrBlock": "172.16.0.0/12",
  "CloudResources": {
      "CloudResourceSetType": [
          {
              "ResourceCount": 1,
              "ResourceType": "VRouter"
          },
          {
              "ResourceCount": 1,
              "ResourceType": "RouteTable"
          }
      ]
  },
  "CreationTime": "2022-02-25T07:44:24Z",
  "Description": "",
  "Ipv4GatewayId": "",
  "Ipv6CidrBlock": "",
  "IsDefault": false,
  "NetworkAclNum": 0,
  "OwnerId": 1874279259696164,
  "RegionId": "cn-beijing",
  "RequestId": "F8AAAB31-75FA-5DBB-B86F-1599AC96A408",
  "ResourceGroupId": "rg-acfm3s5kf5r6yja",
  "SecondaryCidrBlocks": {
      "SecondaryCidrBlock": []
  },
  "Status": "Available",
  "SupportIpv4Gateway": false,
  "UserCidrs": {
      "UserCidr": []
  },
  "VRouterId": "vrt-2zekj3mo5ryh2kqp0jvu8",
  "VSwitchIds": {
      "VSwitchId": []
  },
  "VpcId": "vpc-2zegc9hgtyeyis96d7bov",
  "VpcName": "vpc-2zegc9hgtyeyis96d7bov"
}

@zzxwill I am closing this issue based on your comment above. Feel free to create another issue if there are any other problems you're observing.

Thanks!