Requesting a list of vcd_ipsets results in an error

[erikmouw]

Requesting a list of vcd_ipsets results in a HTTP 500 error. Alternatively, requesting vcd_nsxv_ip_set or vcd_nsxt_ip_set results in an "Unhandled resource type" error.

Terraform Version

$ ~/bin/terraform -v
Terraform v1.0.10
on linux_amd64
+ provider registry.terraform.io/vmware/vcd v3.4.0

Affected Resource(s)

Please list the resources as a list, for example:

• vcd_resource_list --> vcd_ipset

Terraform Configuration Files

terraform {
  required_providers {
    vcd = {
      source = "vmware/vcd"
      version = "~> 3.4.0"
    }
  }
}

provider "vcd" {
    user    =  "MYUSERNAME"
    password = "MYPASSWORD"
    auth_type   = "integrated"
    org     = "TEST"
    vdc     = "TEST-ERIK-TST-AZ1"
    url     = "https://cloud-az1.example.com/api"
    allow_unverified_ssl = true
}

/* This works fine and prints a list of resources */
data "vcd_resource_list" "list_of_resources" {
  provider = vcd
  name          = "list_of_resources"
  resource_type = "resources"
  list_mode     = "name_id"
}

output "resource_list" {
  value = data.vcd_resource_list.list_of_resources.list
}

/* This should output a list of ipsets, but results in a HTTP 500 error */
data "vcd_resource_list" "list_of_vcd_ipsets" {
  provider      = vcd
  name          = "vcd_ipset_list"
  resource_type = "vcd_ipset"
  list_mode     = "name_id"
}

output "vcd_ipset_list" {
  value = data.vcd_resource_list.list_of_vcd_ipsets.list
}

Debug Output

Debug output

Panic Output

None

Expected Behavior

A list of ipsets as output.

Actual Behavior

$ TF_LOG_PATH=tf.log TF_LOG=TRACE GOVCD_LOG_FILE=go-vcloud-director.log GOVCD_LOG=true ~/bin/terraform plan
|
│ Error: unable to read IP sets for scope a5a21dda-0c9b-441b-a068-d0057c6346df: 500 Server Error
│
│   with data.vcd_resource_list.list_of_vcd_ipsets,
│   on main.tf line 13, in data "vcd_resource_list" "list_of_vcd_ipsets":
│   13: data "vcd_resource_list" "list_of_vcd_ipsets" {
│

Steps to Reproduce

1. terraform plan

User Access rights

My account has admin rights for all tenants, so I don't think it's an access rights issue.

Important Factoids

Cloud Director version is 10.2.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here?

• Not that I know of

[Didainius]

Hello @erikmouw, First query - is your VDC backed by NSX-T or NSX-V?

[erikmouw]

As far as I remember (not at work) it’s NSX-T as it was installed this spring. I‘ll verify tomorrow and let you know.

[erikmouw]

@Didainius Verified, it is NSX-T.

[Didainius]

@erikmouw , ok - so far the vcd_resource_list item doesn't support listing NSX-T IP sets. But that is noted to also improve the error message.

Just want to also understand your case? Is your final goal just to list all IP sets?

[emielkuiper]

@Didainius: I am working with Erik on this. Sorry for the delay, but the end goal here is to be able to list and provision IP sets and FW rules for edge gateway and distributed firewall. However, we already run into issues listing existing items using vcd_resource_list, hence the initial question about being able to list those types of resource types.

[Didainius]

@emielkuiper, Thanks for answer - what do you intend to do? Create a firewall rule for each IP set? I am trying to understand the need for listing. We have a data source to lookup single IP set - https://registry.terraform.io/providers/vmware/vcd/latest/docs/data-sources/nsxt_ip_set so far

[emielkuiper]

@Didainius: thanks, will retry looking up IP set with nsxt_ip_set. The need for listing is mostly to be able to run a job to quickly show 'what has already been configured'. I personally wanted that ability to check the actual presence of items after running create jobs for them. Also: I wanted to use that mechanism to get a list of edge gateways (settings) that were available to me to import as the names I got from the VCD webportal resulted in an ENF after using them in terraform code.

[emielkuiper]

Hello Didainius,

I enabled logging in the provider block like:

  logging              = true
  logging_file         = "terraform_provider_az1.log"

When I request a certain edge gateway, the following shows up in the logfile mentioned:

2021/11/23 10:34:19 Request caller: schema.(*Resource).read-->vcd.datasourceVcdNsxtEdgeGatewayRead-->govcd.(*Vdc).GetNsxtEdgeGatewayByName-->govcd.(*Vdc).GetAllNsxtEdgeGateways-->govcd.getAllNsxtEdgeGateways-->govcd.(*Client).OpenApiGetAllItems-->govcd.(*Client).openApiGetAllPages-->govcd.(*Client).newOpenApiRequest
2021/11/23 10:34:19 GET https://<SOME_URL>/cloudapi/1.0.0/edgeGateways/?filter=name%3D%3D<SOME_NAME>%3BorgVdc.id%3D%3Durn%3Avcloud%3Avdc%3A<SOME_ID>&pageSize=128
2021/11/23 10:34:19 --------------------------------------------------------------------------------
2021/11/23 10:34:19 Req header:
2021/11/23 10:34:19     Authorization: [********]
2021/11/23 10:34:19     X-Vmware-Vcloud-Token-Type: [Bearer]
2021/11/23 10:34:19     Accept: [application/json;version=34.0]
2021/11/23 10:34:19     Content-Type: [application/json]
2021/11/23 10:34:19     User-Agent: [terraform-provider-vcd/v3.4.0 (linux/amd64; isProvider:false)]
2021/11/23 10:34:19     X-Vmware-Vcloud-Access-Token: [********]
2021/11/23 10:34:19 ################################################################################
2021/11/23 10:34:19 Response caller schema.(*Resource).read-->vcd.datasourceVcdNsxtEdgeGatewayRead-->govcd.(*Vdc).GetNsxtEdgeGatewayByName-->govcd.(*Vdc).GetAllNsxtEdgeGateways-->govcd.getAllNsxtEdgeGateways-->govcd.(*Client).OpenApiGetAllItems-->govcd.(*Client).openApiGetAllPages-->govcd.decodeBody
2021/11/23 10:34:19 Response status 200 OK
2021/11/23 10:34:19 ################################################################################
2021/11/23 10:34:19 Response header:
2021/11/23 10:34:19     Date: [Tue, 23 Nov 2021 09:34:19 GMT]
2021/11/23 10:34:19     X-Vmware-Vcloud-Ceip-Id: [xxxxx-xxxxx-xxxxx-xxxxx-xxxxx]
2021/11/23 10:34:19     Cache-Control: [no-store, must-revalidate]
2021/11/23 10:34:19     X-Vmware-Vcloud-Request-Id: [xxxxx-xxxxx-xxxxx-xxxxx-xxxxx]
2021/11/23 10:34:19     Content-Type: [application/json;version=34.0]
2021/11/23 10:34:19     Content-Location: [https://<SOME_URL>/cloudapi/1.0.0/edgeGateways/]
2021/11/23 10:34:19     Link: [<https://<SOME_URL>/cloudapi/>;rel="up";type="*/*"]
2021/11/23 10:34:19     Vary: [Accept-Encoding, User-Agent]
2021/11/23 10:34:19 Response text: [112]
{
  "resultTotal": 0,
  "pageCount": 0,
  "page": 1,
  "pageSize": 128,
  "associations": null,
  "values": []
}

When I try to access the VCD API directly with Curl (tested with versions 33.0, 34.0 and 35.2):

curl --request GET \
  --url https://<SOME_URL>/cloudapi/1.0.0/edgeGateways \
  --header 'Accept: application/json;version=34.0' \
  --header 'Authorization: Bearer xx.yy.zz' \
  --header 'Content-Type: application/json'

I do get an actual gateway back:

{
  "resultTotal": 1,
  "pageCount": 1,
  "page": 1,
  "pageSize": 25,
  "associations": null,
  "values": [
    {
      "status": "REALIZED",
      "id": "urn:vcloud:gateway:aa-bb-cc-dd-ee",
      "name": "SOME_NAME",
      "description": "",
      "edgeGatewayUplinks": [
(...)

When reproducing the literal TF call however:

curl --request GET \
  --url 'https://<SOME_URL>/cloudapi/1.0.0/edgeGateways/?filter=name%3D%3D<SOME_NAME>%3BorgVdc.id%3D%3Durn%3Avcloud%3Avdc%3A<SOME_ID>&pageSize=128' \
  --header 'Accept: application/json;version=34.0' \
  --header 'Authorization: Bearer xx.yy.zz' \
  --header 'Content-Type: application/json' \
  --header 'User-Agent: [terraform-provider-vcd/v3.4.0 (linux/amd64; isProvider:false)]' \
  --header 'X-Vmware-Vcloud-Access-Token: xx.yy.zz' \
  --header 'X-Vmware-Vcloud-Token-Type: Bearer'

I get no results:

{
  "resultTotal": 0,
  "pageCount": 0,
  "page": 1,
  "pageSize": 128,
  "associations": null,
  "values": []
}

I checked the values used for name and orgVdc.id and they are the correct values.

In my working curl request I get a.o. the following returned:

.values[].name:  an actual name 
.values[].id:  an actual id like urn:vcloud:gateway:aa-bb-cc-dd-ee 

I reproduced the filter section in the TF generated url with both name==NAME and id==ID but in both cases I get no results.

I really want to clarify the issue and be able to reproduce the situation as well as I can. Is there anything else I can do to clarify the matter further and provide you with more useful debug output?

Thanks in advance,

Emiel Kuiper

[Didainius]

I really want to clarify the issue and be able to reproduce the situation as well as I can. Is there anything else I can do to clarify the matter further and provide you with more useful debug output?

Thanks for this informationo. Are you able to share the name of your edge gateway (maybe not exact, but I suspect you have something in your name that breaks filtering). Alternatively - are you able to try and lookup an edge gateway with a really simple name - letters only - no spaces and specials characters.

[emielkuiper]

Hello Didainius,

I tested some more; copied the TF generated query and ran as-is with the name filter but without the orgVdc.id filter, and I got results. It seems that the orgVdc.id filter is causing the 0 results to be returned.

By the way, the name of the edge gateway is only capitals and dashes, like:

STRING-STRING-STRING-STRING-T-AZ1

This is the naming scheme for the gateways available to me so I can unfortunately not test with a name consisting only of letters.

When I tested the output without the orgVdc.id filter, this is roughly the place in the return JSON where the name and orgVdc id are mentioned.

{
    "resultTotal": 1,
    "pageCount": 1,
    "page": 1,
    "pageSize": 128,
    "associations": null,
    "values": [
        {
            "status": "REALIZED",
            "id": "urn:vcloud:gateway:xxxxxx,
            "name": "STRING-STRING-STRING-STRING-T-AZ1",
            "description": "",
            "orgVdc": null,
            "orgRef": {
                "name": "STRING-STRING-STRING",
                "id": "urn:vcloud:org:xxxxxxx"
            }
        }]
        (...)
}

Hope this helps you. Please note the null value in .values[].orgVdc. Not really sure how to work around that behaviour.

Best regards,

Emiel Kuiper

[Didainius]

Hope this helps you. Please note the null value in .values[].orgVdc.

This brings me to some interesting question - what role does your user have? Ar some of the rights adjusted in any way? Is the Edge Gateway in a VDC (no VDC group)?

Also - could you give a more precise version of your VCD 10.2.X? Maybe a build number? I could looko into having the same.

[emielkuiper]

My roles are (as reported by the /roles api endpoint):

"Organization Administrator" "Catalog Author" "vApp Author" "vApp User" "Console Access Only" "Defer to Identity Provider" "Tenant Administrator" "Tenant Read Only" "Tenant Server" "Tenant Network DFW" "Tenant Network EGW"

My VCD version is:

VMware Cloud Director version: 10.2.2.17855680

Wether or not the Edge Gateway is in a group or not, will have get that answer from someone else. Will get back to you on that.

[emielkuiper]

@Didainius: sorry for the delay but I just got the confirmation that edge gateways apparently are indeed part of a DC group. As far as I can tell now my user was assigned to abovementioned groups with no further adjustments to my rights.

[Didainius]

Ok, so this is probably the problem. We're catching up in this game and full VDC group support is yet to be done, so these resources and data sources are not tested with VDC groups yet.

This is the task tracking VDC group resource / data source support rollout, but we will still have to adjust all the resources/data sources to cover with tests and make sure they work. I suspect they won't out of the box but we need to pave the way.

https://github.com/vmware/terraform-provider-vcd/issues/659

[vbauzys]

@emielkuiper Just to be sure: vdc = "TEST-ERIK-TST-AZ1" Is VDC or VDC group?

[emielkuiper]

@vbauzysvmware

To be sure, here is the translation of the answer I got from the people who maintain the VCD environment:

"Every gateway is also part of a datacenter group. Every VDC is part of a DC Group (1-to-1 mapping). The DC group also holds the Edge gateway of the VDC."

When Erik/I specify:

vdc = "TEST-ERIK-TST-AZ1"

I assume we're targeting a specific VDC instead of a VDC group. Could however be that I don't understand your question well enough. If you have any more questions, please ask and I will have our VCD maintainers take a look at them.

[vbauzys]

Thank you for your answer.

[Didainius]

@emielkuiper, 3.6.0 should have resolved your issue. https://registry.terraform.io/providers/vmware/vcd/latest/docs/guides/vdc_groups

I am closing this issue for now - if you find something new - feel free to reopen.