Migration from Terraform errors

[KevinvOosterhout]

What happened?

Whilst trying to migrate from Terraform to Pulumi we encountered an issue trying to migrate our HCL to Pulumi Python. We've got a complicated repo with our own modules and the following dependencies. Using terraform we're creating resources on DigitalOcean such as K8s cluster, loadbalancer, DNS, executing helm charts for ArgoCD, Traefik, cert-manager, etc.

Dependencies/Providers

terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "~> 2.39.2"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~>2.19.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.9.0"
    }
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.14.0"
    }
  }
}

Example

Console output

pulumi convert --from terraform --language python
...
================================================================================
The Pulumi CLI encountered a code generation error. This is a bug!
We would appreciate a report: https://github.com/pulumi/pulumi/issues/
Please provide all of the below text in your report.
================================================================================
Pulumi Version:   v3.122.0
warning: argocd.pp:24,30-54: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: argocd.pp:29,31-50: the first argument to 'element' must be a list or tuple; 
warning: argocd.pp:29,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: argocd.pp:30,3-20: unsupported attribute 'overrideNamespace'; unsupported attribute 'overrideNamespace'
warning: argocd.pp:33,27-51: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: argocd.pp:39,31-47: the first argument to 'element' must be a list or tuple; 
warning: argocd.pp:39,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: argocd.pp:40,3-20: unsupported attribute 'overrideNamespace'; unsupported attribute 'overrideNamespace'
warning: argocd.pp:43,33-57: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: argocd.pp:49,31-53: the first argument to 'element' must be a list or tuple; 
warning: argocd.pp:49,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: argocd.pp:50,3-20: unsupported attribute 'overrideNamespace'; unsupported attribute 'overrideNamespace'
warning: argocd.pp:53,37-61: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: argocd.pp:59,31-57: the first argument to 'element' must be a list or tuple; 
warning: argocd.pp:59,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: argocd.pp:60,3-20: unsupported attribute 'overrideNamespace'; unsupported attribute 'overrideNamespace'
warning: cert-manager.pp:6,32-52: unknown package 'helm'; rpc error: code = Unknown desc = could not find latest version for provider helm: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-helm/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
warning: cert-manager.pp:11,3-7: unsupported attribute 'name'; unsupported attribute 'name'
warning: cert-manager.pp:12,3-13: unsupported attribute 'repository'; unsupported attribute 'repository'
warning: cert-manager.pp:13,3-8: unsupported attribute 'chart'; unsupported attribute 'chart'
warning: cert-manager.pp:14,3-12: unsupported attribute 'namespace'; unsupported attribute 'namespace'
warning: cert-manager.pp:15,3-6: unsupported attribute 'set'; unsupported attribute 'set'
warning: cert-manager.pp:40,43-67: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: cert-manager.pp:46,22-54: the first argument to 'element' must be a list or tuple; 
warning: cert-manager.pp:46,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: cert-manager.pp:55,46-70: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: cert-manager.pp:61,22-57: the first argument to 'element' must be a list or tuple; 
warning: cert-manager.pp:61,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: traefik.pp:6,27-47: unknown package 'helm'; rpc error: code = Unknown desc = could not find latest version for provider helm: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-helm/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
warning: traefik.pp:11,3-7: unsupported attribute 'name'; unsupported attribute 'name'
warning: traefik.pp:12,3-12: unsupported attribute 'namespace'; unsupported attribute 'namespace'
warning: traefik.pp:13,3-8: unsupported attribute 'chart'; unsupported attribute 'chart'
warning: traefik.pp:14,3-13: unsupported attribute 'repository'; unsupported attribute 'repository'
warning: traefik.pp:15,3-10: unsupported attribute 'version'; unsupported attribute 'version'
warning: traefik.pp:16,3-9: unsupported attribute 'values'; unsupported attribute 'values'
warning: traefik.pp:31,31-55: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: traefik.pp:37,22-55: the first argument to 'element' must be a list or tuple; 
warning: traefik.pp:37,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: traefik.pp:46,28-52: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: traefik.pp:52,22-39: the first argument to 'element' must be a list or tuple; 
warning: traefik.pp:52,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
warning: traefik.pp:61,25-49: unknown package 'kubectl'; rpc error: code = Unknown desc = could not find latest version for provider kubectl: 404 HTTP error fetching plugin from https://api.github.com/repos/pulumi/pulumi-kubectl/releases/latest. If this is a private GitHub repository, try providing a token via the GITHUB_TOKEN environment variable. See: https://github.com/settings/tokens
error: traefik.pp:67,22-36: the first argument to 'element' must be a list or tuple; 
warning: traefik.pp:67,3-11: unsupported attribute 'yamlBody'; unsupported attribute 'yamlBody'
error: could not generate output program

Output of pulumi about

pulumi about CLI
Version 3.122.0 Go Version go1.22.4 Go Compiler gc

Host
OS darwin Version 15.0 Arch arm64

Backend
Name pulumi.com URL https://app.pulumi.com/kevinvanoosterhout User kevinvanoosterhout Organizations kevinvanoosterhout, nuxnova Token type personal

Pulumi locates its logs in /var/folders/c2/3z599gmj5pn0k4gx92mcfdpw0000gn/T/ by default warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from /Users/kevin.oosterhout/Projects/NuxNova/Infrastructure/terraform). If you have not created a project yet, use pulumi new to do so: no project file found warning: Failed to get information about the current stack: no Pulumi.yaml project file found (searching upwards from /Users/kevin.oosterhout/Projects/NuxNova/Infrastructure/terraform). If you have not created a project yet, use pulumi new to do so: no project file found

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[justinvp]

Thanks for opening the issue. Moving this issue to the terraform converter repo.

The main issue here is that we don't have a bridged Pulumi providers available for the helm and kubectl providers. There are some things we may want to consider doing here:

1. Provide a way to convert from TF helm provider to Pulumi's Kubernetes provider's helm support
2. Dynamically bridge providers that aren't already bridged, for cases like kubectl
3. Alternative to (2), allow the code gen to succeed even when there isn't a bridged Pulumi kubectl... just generate "fake" Pulumi code in the target language, that won't actually work (because there isn't a provider available) but could help migrating anyway, in this case converting the code to use the Pulumi Kubernetes provider's YAML support.

In the meantime, to get the converter to succeed, you may need to hack-up your TF code a bit, removing uses of kubectl and helm providers, and then manually converting those to Pulumi python.