search

PrefectHQ / terraform-provider-prefect

Terraform Provider for Prefect Cloud
https://registry.terraform.io/providers/PrefectHQ/prefect/latest/docs
Apache License 2.0
29 stars 13 forks source link

feat(blocks): add Block data source #209

Closed mitchnielsen closed 3 weeks ago

mitchnielsen commented 3 weeks ago

Summary

Closes https://github.com/PrefectHQ/terraform-provider-prefect/issues/175

To do

Testing

First, start up the local instance:

docker compose up -d

Create a main.tf file with these contents:

terraform {
  required_providers {
    prefect = {
      source = "registry.terraform.io/prefecthq/prefect"
    }
  }
}

provider "prefect" {
  endpoint = "http://localhost:4200/api"
}

resource "prefect_block" "secret" {
  name      = "foo"
  type_slug = "secret"

  data = jsonencode({
    "value" : "bar"
  })
}

data "prefect_block" "my_existing_secret" {
  id = prefect_block.secret.id
}

output "block_info" {
  value     = data.prefect_block.my_existing_secret
  sensitive = true
}

Test it:

Click to expand ``` Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create <= read (data resources) Terraform will perform the following actions: # data.prefect_block.my_existing_secret will be read during apply # (config refers to values not yet known) <= data "prefect_block" "my_existing_secret" { + block_schema_id = (known after apply) + block_type_id = (known after apply) + block_type_name = (known after apply) + created = (known after apply) + data = (sensitive value) + id = (known after apply) + name = (known after apply) + updated = (known after apply) } # prefect_block.secret will be created + resource "prefect_block" "secret" { + created = (known after apply) + data = (sensitive value) + id = (known after apply) + name = "foo" + type_slug = "secret" + updated = (known after apply) } Plan: 1 to add, 0 to change, 0 to destroy. Changes to Outputs: + block_info = (sensitive value) Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes prefect_block.secret: Creating... prefect_block.secret: Creation complete after 0s [id=b98844a2-39a5-4ba3-9562-0f104cc4c33f] data.prefect_block.my_existing_secret: Reading... data.prefect_block.my_existing_secret: Read complete after 0s [id=b98844a2-39a5-4ba3-9562-0f104cc4c33f] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Outputs: block_info = ```

Get the specific value:

$ tf output -json block_info | jq .data
"{\"value\":\"bar\"}"
mitchnielsen commented 3 weeks ago

Currently, some of the fields in the schema are omitted because I'm not yet sure how to represent types other than string, UUID, or timestamp. Here's the full response from the API Read document by ID:

Click to expand ```json { "block_documents": { "operator": "and_", "id": { "any_": [ "497f6eca-6276-4993-bfeb-53cbbbba6f08" ] }, "is_anonymous": { "eq_": false }, "block_type_id": { "any_": [ "497f6eca-6276-4993-bfeb-53cbbbba6f08" ] }, "name": { "any_": [ "string" ], "like_": "my-block%" } }, "block_types": { "name": { "like_": "marvin" }, "slug": { "any_": [ "string" ] } }, "block_schemas": { "operator": "and_", "block_type_id": { "any_": [ "497f6eca-6276-4993-bfeb-53cbbbba6f08" ] }, "block_capabilities": { "all_": [ "write-storage", "read-storage" ] }, "id": { "any_": [ "497f6eca-6276-4993-bfeb-53cbbbba6f08" ] }, "version": { "any_": [ "2.0.0", "2.1.0" ] } }, "include_secrets": false, "sort": "NAME_ASC", "offset": 0, "limit": 0 } ```

So the current fields omitted are:

I'm also omitting is_anonymous, but I don't think we need to expose that (yet?).

@parkedwards two quick questions:

mitchnielsen commented 3 weeks ago

Looking into querying by name:

In the Read Block Documents endpoint, we could allow a user to pass a name value and then interpolate it to fit into the block_documents.name.like or block_documents.name.any field:

image

This seems doable, but will require a more detailed implementation to do that translation. At the moment, I'm thinking that querying by ID should cover most use cases (especially since users would likely reference the object by the friendly data.prefect_block.my_block.id identifier).

What do you think @parkedwards?

mitchnielsen commented 3 weeks ago

I drafted a test, but looking at existing Datasource tests it seems like they read existing objects in Prefect. Wondering which project we create these in, since I can't read the encrypted variables in the GitHub project once they're written.

In this case I'd need to go create a block named my_block before committing the test (following the example from the variable_test.go).

package datasources_test

import (
    "fmt"
    "testing"

    "github.com/hashicorp/terraform-plugin-testing/helper/resource"
    "github.com/prefecthq/terraform-provider-prefect/internal/testutils"
)

func fixtureAccBlockByID(name string) string {
    return fmt.Sprintf(`
data "prefect_block" "test" {
    id = prefect_block.%s.id
}
`, name)
}

//nolint:paralleltest // we use the resource.ParallelTest helper instead
func TestAccDatasource_block(t *testing.T) {
    datasourceName := "data.prefect_block.test"
    blockName := "my_block"
    blockValue := "{\"value\":\"bar\"}"

    resource.ParallelTest(t, resource.TestCase{
        ProtoV6ProviderFactories: testutils.TestAccProtoV6ProviderFactories,
        PreCheck:                 func() { testutils.AccTestPreCheck(t) },
        Steps: []resource.TestStep{
            {
                Config: fixtureAccBlockByID(blockName),
                Check: resource.ComposeAggregateTestCheckFunc(
                    resource.TestCheckResourceAttrSet(datasourceName, "id"),
                    resource.TestCheckResourceAttr(datasourceName, "name", blockName),
                    resource.TestCheckResourceAttr(datasourceName, "data", blockValue),
                ),
            },
        },
    })
}
parkedwards commented 3 weeks ago

Do you see a need to add these currently-omitted fields to the data source? If yes, do you happen to know how to do it? I'm seeing some references to nested objects as a type but seems a little verbose.

@mitchnielsen you're right, i think we can omit those, and i added a couple more in the comments above that we might think about skipping. the nested object typing is a little nasty, and if we dont believe users to need to actually access those property values (which im not sure they do as of right now - that could change, at which point we can always take on the incremental work to support those), then we can omit

parkedwards commented 3 weeks ago

This seems doable, but will require a more detailed implementation to do that translation. At the moment, I'm thinking that querying by ID should cover most use cases (especially since users would likely reference the object by the friendly data.prefect_block.my_block.id identifier).

@mitchnielsen great idea, and totally do-able. i think we should do this (in this PR or a followup). the general idea would be - allow the user to input either an id or a name, and based on the presence of either (you'd have to give precedence to one of those, so it'd probably be id) - you'd fire off a request to either the Get(id uuid.UUID) client method that we already have OR we can add a List(names []string) method to the client (similar to what you did with BlockSchema client)

check out the variable.go datasource, we do something similar there

https://github.com/PrefectHQ/terraform-provider-prefect/blob/main/internal/provider/datasources/variable.go#L121-L164

parkedwards commented 3 weeks ago

I drafted a test, but looking at existing Datasource tests it seems like they read existing objects in Prefect. Wondering which project we create these in, since I can't read the encrypted variables in the GitHub project once they're written.

@mitchnielsen yeah that was an approach i started with back then but dont love. maybe we can create a new block resource in the same fixture string (right before your datasource) and then you can compare values against your input resource and the queried datasource

mitchnielsen commented 3 weeks ago

maybe we can create a new block resource in the same fixture string (right before your datasource) and then you can compare values against your input resource and the queried datasource

Works for me. I was originally wondering if it should be cleaned up after the test run, but it's probably alright for now that it stays around since even concurrent tests won't try to modify it.