Adhoc Provider - Custom Resource / Data Source

skorfmann commented 1 year ago

Feature Spec

With Wing Custom Resources, it's now simple to interface with bespoke APIs and extend the Terraform inflight behaviour with custom code snippets.

bring resource

class GithubIssue impl resource.ICustomResource {
  create(title: str) {
    // ...
  }
  update(id: num, title: str) {
    // ...
  }
  read(id: num) {
    // ...
  }
  destroy(id: num) {
    // ...
  }
}

or for data fetching / manipulation something like this (inspired by this)

interface CustomDockerImageArgs {
  repositoryUrl: str;
  username: str;
  password: str;
}

class CustomDockerImage impl resource.ICustomDataSource {    
  sha256Digest: str?;  
  reposoitryUrl: str;
  username: str;
  password: str;

  init(config: CustomDockerImageArgs) {
    this.repositoryUrl = config.repositoryUrl;
    this.username = config.username;
    this.password = config.password;
  }

  onRead(): {    
    // For complex cases, this should be JS code. So maybe external?
    const drc = require('docker-registry-client')
    return new Promise((resolve, reject) => {
      var rar = drc.parseRepoAndRef(this.repositoryUrl);
      var client = drc.createClientV2({
        repo: rar,
        insecure: false,
        username: this.username,
        password: this.password,
        maxSchemaVersion: 2
      });
      var tagOrDigest = rar.tag || rar.digest;
      client.getManifest({ref: tagOrDigest}, function (err:any, _manifest:any, _res:any, manifestStr:any) {
        client.close();
        if (err) {
          reject(err)
        }
        this.sha256Digest = drc.digestFromManifestStr(manifestStr);
        resolve(this);
      });
    });
  }
}

Both, resources and data sources, are build ad-hoc as a Terraform provider when comping this with the Wing CLI transparent to to the user.

In a Wing application, this could be used like:

let image = new CustomDockerImage(...)

which would render down to a Terraform HCL (JSON) data source, something like (pseudo HCL code)

provider "wing-adhoc" {}

data "wing-adhoc_custom-docker-image" {
  repository_url = "https://abc"
  username = "hello"
  password = "world"
}

output "sha56" {
  value = data.wing-adhoc_custom-docker-image.sha56
}

Use Cases

build ad hoc resources to interface with APIs not covered by a Terraform provider
build ad hoc data sources, to fetch and / or manipulate data as needed
perform custom computations at Terraform run time
build test cases and bake them into Terraform
enable running any JS code at Terraform run time

Implementation Notes

I proposed the same for the CDK for Terraform project a while ago
There's an incomplete and unmaintained nodejs port of the Go Plugin Framework https://github.com/lukekaalim/terraform-plugin-node-SDK/
There is something similar in Pulumi with Dynamic Providers
I believe Deno would be a great fit as the runtime enabling this, primarily due to the dependency management mechanics of url imports, the possibility to entirely sandbox a Deno process and creating binaries is a direct feature of Deno.
This could be an open source project on its own and quite useful beyond Wing
There is a Javascript Terraform provider for data wrangling https://github.com/lewispeckover/terraform-provider-javascript (based on https://github.com/dop251/goja).

There are two components to make this work at a high level:

Build a solid implementation of the Provider Plugin Framework in Javascript or find a way to utilize parts the Go implementation under the hood.
Build abstractions in Wing which can use this with minimal overhead

Component

Other

Community Notes

Please vote by adding a 👍 reaction to the issue to help us prioritize.
If you are interested to work on this issue, please leave a comment.

github-actions[bot] commented 1 year ago

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

skorfmann commented 1 year ago

Here's a use case for fly.io - while there's a Terraform provider, the readme states:

This project is not currently maintained, and is not a recommended method of deployment to Fly.io.

we want to do create a project via Terraform, right now this is shelling out via null_resource and local-exec. While this works for creation, it's pretty much a dead-end for destroying - see https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax#destroy-time-provisioners

   let resource = new nullProvider.resource.Resource(triggers: { "changed": util.nanoid() }) as "create"
    resource.addOverride("provisioner.local-exec.environment", {"FLY_APP_NAME": appName});
    resource.addOverride("provisioner.local-exec.command", "
flyctl status -a \$FLY_APP_NAME || flyctl launch --copy-config --no-deploy --name \$FLY_APP_NAME -o ${props.org} -r iad -y
    ");

  // FIXME: We'll have to find another way to do this. Can't reference other resources here. Also, there are drawbacks
    // see https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax#destroy-time-provisioners

   let destroy =  new nullProvider.resource.Resource() as "destroy";
    destroy.addOverride("provisioner.local-exec.when", "destroy");
    destroy.addOverride("provisioner.local-exec.environment", {"FLY_APP_NAME": appName});
    destroy.addOverride("provisioner.local-exec.command", "flyctl status -a \$FLY_APP_NAME && flyctl apps destroy \$FLY_APP_NAME -y");

so, if it was possible to do something like:

bring http;

struct FlyProjectProps {
  name: str;
  org: str;
  region: str;
}

class FlyProject extends CustomResource {
  var name: str;
  var region: str;
  var bucket: str;

  init(props: FlyProjectProps) {
    this.name = props.name;
    this.region = props.region;
    this.bucket = props.bucket;
  }

  // get invoked during terraform apply as normal Terraform resource
  pub create(): void {
    // Add code here to create the project via http
    http.post("https://api.fly.io/v2/projects", ...)
  }

  pub read(): void {
    // Add code here to read the resource
    http.get("https://api.fly.io/v2/projects", ...)
  }

  pub update(): void {
    // Add code here to update the resource
    http.put("https://api.fly.io/v2/projects", ...)
  }

  pub delete(): void {
    // Add code here to delete the resource
    http.delete("https://api.fly.io/v2/projects", ...)
  }
}

then this could become

bring fly

new fly.Project(
  name: "foo",
  region: "iad",
  org: "bar"
);

github-actions[bot] commented 8 months ago

Hi,

This issue hasn't seen activity in 90 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

github-actions[bot] commented 5 months ago

Hi,

This issue hasn't seen activity in 90 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!

winglang / wing