POST creates attributes that PUT expects

[arun-a-nayagam]

Hi,

I have an API that is creating few attributes when the POST call is made, /Applications is where the POST call is made and at that time it creates an apiKey and apiSecret attributes.

Now whenever an update is made to the resource - /Applications/{id}, it is expecting the apiKey and apiSecret to be passed.

Is there a way to conditionally say when a PUT is made pass these extra attributes from the POST calls response?

Love the simplicity of this provider but stuck with this issue. Hopefully this is possible.

[jgrumboe]

Hi @arun-a-nayagam, maybe this PR https://github.com/Mastercard/terraform-provider-restapi/pull/182 could help you.

[arun-a-nayagam]

Hi @jgrumboe - Thank you for the reply, yes an update_data will be very helpful. Oh I just saw your comment "I published my changes interim to TF registry under https://registry.terraform.io/providers/jgrumboe/restapi/1.18.0-dev" I will definitely try it out and let you know, thank you so much!

[jgrumboe]

Don't know honestly, but see here: https://github.com/Mastercard/terraform-provider-restapi/pull/182#issuecomment-1216278310

[arun-a-nayagam]

@jgrumboe, I did try to test your provider. I refer to the ApiKey and KeySecret that the POST call is creating using api_data

update_data = jsonencode({ "Uuid": random_uuid.app_scripted_demo_client_uuid.id, "Name": "Test", "ApiKey": restapi_object.app_scripted_demo_client.api_data.ApiKey, "KeySecret": restapi_object.app_scripted_demo_client.api_data.KeySecret })

But I get this error,

Error: Self-referential block │ │ on apps.tf line 31, in resource "restapi_object" "app_scripted_demo_client":
│ 31: "ApiKey": restapi_object.app_scripted_demo_client.api_data.ApiKey,
│ │ Configuration for restapi_object.app_scripted_demo_client may not refer to itself.

How do I refer to the values that's returned by the original POST call?

[jgrumboe]

Have you tried using self. reference? But probably that's also not possible, because the value cannot be evaluated at plan phase. Sounds like a Terraform problem/design itself and not a provider issue. Maybe it's possible via a local variable, which would store the value interim.

[arun-a-nayagam]

Hi @jgrumboe I had raised the same issue on hashicorp support site, https://discuss.hashicorp.com/t/ignore-circumvent-self-referential-block/43342

This was their reply,

No, it’s not possible for a resource’s inputs to refer to its own outputs. That would be a circular dependency in the general case.

In this specific case the provider itself would have to implement a different way to express merging supplied values with values from a GET request.

I am going to try implementing a "data http" call to get the api key and secret and assign it to variables. Hopefully after that it's not seen as a self-referential block or a circular dependency.

[jgrumboe]

Thanks for asking support. Yeah, probably a separate data http source could work. Regarding provider support for this use case, I want to say, that 1. I'm also just a normal user of this provider and no maintainer and 2. wouldn't have a glue how to implement that as I'm also no real go programmer. 😊

[arun-a-nayagam]

Hi @jgrumboe while I was trying to implement data http I realised, I need the bearer token to be able make that call. I know restapi internally uses the token, is there a way to access it so it can be used in the data "http" call?

[jgrumboe]

Could you post an example resource block how you call restapi_object?

[arun-a-nayagam]

Hi @jgrumboe, a bit like this,

resource "restapi_object" "app_scripted_demo_client" {
  provider = restapi.restapi_oauth
  path     = "/Applications"
  data = jsonencode({
    "Uuid": random_uuid.app_scripted_demo_client_uuid.id,
    "Name": "Scripted Demo Client"
    }
  })

  update_data = jsonencode({
    "Uuid": random_uuid.app_scripted_demo_client_uuid.id,
    "Name": "Scripted Demo Client 1",
    "ApiKey": try(restapi_object.app_scripted_demo_client.api_data.ApiKey, "dummykey"),
    "KeySecret": try(restapi_object.app_scripted_demo_client.api_data.KeySecret, "dummysecret")
    }
  })

  id_attribute = "Uuid"
  read_path    = "/Applications('{id}')"
  create_path  = "/Applications"
  destroy_path = "/Applications('{id}')"
  update_path  = "/Applications('{id}')"
}

Provider block is like this,

provider "restapi" {
  alias                = "restapi_oauth"
  uri                  = "https://portal.com/portal"
  debug                = true
  write_returns_object = true

  oauth_client_credentials {
    oauth_client_id = "cccccc"
    oauth_client_secret = "ssssss"
    oauth_token_endpoint = "https://token.com/auth/oauth/v2/token"
    oauth_scopes = ["opeind"]
  }
}

[jgrumboe]

I see. AFAIK it's not possible in Terraform to access provider attributes/arguments directly. There would need to be something implemented in a data source way, like the aws-provider did with the default_tags https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/default_tags

[arun-a-nayagam]

@jgrumboe I used provider http-full and got the access token explicitly. That works fine.

But even a data http to get the key and secret and use that in update_data block is having the same cyclic issue. Because even the data http depends_on restapi_object.

Here's the code,

locals {
  app_scripted_demo_client_key    = jsondecode(data.http.app_scripted_demo_client.body).results[0].apiKey
  app_scripted_demo_client_secret = jsondecode(data.http.app_scripted_demo_client.body).results[0].keySecret
}

data "http" "app_scripted_demo_client" {
  url = "https://portal.com/applications/${random_uuid.app_scripted_demo_client_uuid.id}/api-keys"
  request_headers = {
    Authorization = "Bearer ${local.access_token}"
    Accept        = "application/json"
  }

  depends_on = [data.http.papi, random_uuid.app_scripted_demo_client_uuid, restapi_object.app_scripted_demo_client]
}

resource "restapi_object" "app_scripted_demo_client" {
  provider = restapi.restapi_oauth
  path     = "/Applications"
  data = jsonencode({
    "Uuid": random_uuid.app_scripted_demo_client_uuid.id,
    "Name": "Scripted Demo Client"
    }
  })

  update_data = jsonencode({
    "Uuid": random_uuid.app_scripted_demo_client_uuid.id,
    "Name": "Scripted Demo Client 1",
    "ApiKey": local.app_scripted_demo_client_key,
    "KeySecret": local.app_scripted_demo_client_secret
    }
  })

  id_attribute = "Uuid"
  read_path    = "/Applications('{id}')"
  create_path  = "/Applications"
  destroy_path = "/Applications('{id}')"
  update_path  = "/Applications('{id}')"
}

If I remove the depends_on on restapi_object.app_scripted_demo_client, data resource fails as that gets executed before restapi_object.app_scripted_demo_client

[jgrumboe]

You could try to use https://registry.terraform.io/providers/SvenHamers/oauth instead of http data source which would just give an oauth_token not depending on restapi_object. (Haven't tested on my own)

[jgrumboe]

So the update_data routine would use a different Bearer token, then createor destroy. But that shouldn't be a problem IMO.

[arun-a-nayagam]

The issue is not them using the same oauth_token, in fact I generate a different token using http-full provider. The issue is the data resource having to depends_on on restapi_object.

depends_on = [data.http.papi, random_uuid.app_scripted_demo_clientuuid, restapi_object.app_scripted_demo_client_]

Because it needs to wait for restapi_object to be created before it can query for key and secret and that creates the circular dependency.

[jgrumboe]

So, just that I get it right: the ApiKey and Keysecret are available after the restapi_object was created and are specific to the object itself? Then restapi is probably not the solution for you. Don't know how to fix that cyclic dependency in Terraform code.

Just as side note: you don't need to add a resource to depends_on block, if you're already referencing an attribute of it in the resource block. Terraform automatically orders the resources accordingly.

[arun-a-nayagam]

Ok, I was able to solve the circular dependency issue by using a data external and wrap the API call error with a dummy response. So here's my terraform and python codes, if anybody faces a similar issue,

locals {
  app_scripted_demo_client_key    = data.external.external_resource.result.key
  app_scripted_demo_client_secret = data.external.external_resource.result.secret
}

data "external" "external_resource" {
  program = ["python", "${path.module}/get_key_details.py"]

  query = {
    token = local.access_token
    api_id = random_uuid.app_scripted_demo_client_uuid.id
  }
  depends_on = [data.http.papi, random_uuid.app_scripted_demo_client_uuid]
}

resource "random_uuid" "app_scripted_demo_client_uuid" {
}

resource "restapi_object" "app_scripted_demo_client" {
  provider = restapi.restapi_oauth
  path     = "/Applications"
  data = jsonencode({
    "Uuid" : random_uuid.app_scripted_demo_client_uuid.id,
    "Name" : "Scripted Demo Client 1",
    "Description" : "Demo Client App 1"
  })

  update_data = jsonencode({
    "Uuid" : random_uuid.app_scripted_demo_client_uuid.id,
    "Name" : "Scripted Demo Client 1",
    "Description" : "Demo Client App 1 blah blah",
    "ApiKey" : try(local.app_scripted_demo_client_key, "dummykey"),
    "KeySecret" : try(local.app_scripted_demo_client_secret, "dummysecret")
  })

  id_attribute = "Uuid"
  read_path    = "/Applications('{id}')"
  create_path  = "/Applications"
  destroy_path = "/Applications('{id}')"
  update_path  = "/Applications('{id}')"
}

get_key_details.py

import json
import sys
import requests

input = sys.stdin.read()
input_json = json.loads(input)
token = input_json.get("token")
api_id = input_json.get("api_id")
endpoint = "https://portal.com/applications/"+api_id+"/api-keys"
headers = {"Authorization": "Bearer "+token}
r = requests.get(endpoint, headers=headers, verify=False)
if r.status_code == 200:
    r_json = r.json()
    key = r_json["results"][0]["apiKey"]
    secret = r_json["results"][0]["keySecret"]
else:
    key = 'dummykey'
    secret = 'dummysecret'

output = {
    "key": key,
    "secret": secret
}

output_json = json.dumps(output)
print(output_json)

Would be nice to see the update_data feature merged into the master code.

@jgrumboe thanks for all your suggestions and help.

[jgrumboe]

Hi @arun-a-nayagam I just thought a little bit more about your solution/workaround: would a terracurl data source (https://registry.terraform.io/providers/devops-rob/terracurl/latest/docs/data-sources/request) be a good replacement for your external_source and python script?

[arun-a-nayagam]

Hi @jgrumboe, The main reason for including a python, is to alter the API response with a dummy/default response when the resource hasn't been created. like so,

else:
    key = 'dummykey'
    secret = 'dummysecret'

When the POST happens using data block the resource is null. Whereas when the PUT happens using update_data block those values would have been initialized properly.

I don't think, with terracurl I will be able to give some default response when there's an error.

[jgrumboe]

@arun-a-nayagam Just ICYMI : my PR was merged and you can switch to the official restapi provider version >= 1.18.0. No need to fetch mine anymore, as I won't update it.

[arun-a-nayagam]

Thank you for the update, yes already using 1.18.0