panos_device_group is not returning the list of devices of the group name provided (Panorama)

[nirav18]

link: https://registry.terraform.io/providers/PaloAltoNetworks/panos/latest/docs/data-sources/device_group (Panorama only)

Versions: Terraform version: v1.2.2 Panos Version: v1.10.1

data "panos_device_group" "get_list_of_device" { name = "location_1" }

Output:

• get_list_of_devices = {
  • description = ""
  • device = []
  • id = "location_1"
  • name = "location_1"

There are two devices in this group. but it is showing empty list.

[welcome-to-palo-alto-networks[bot]]

:tada: Thanks for opening your first issue here! Welcome to the community!

[shinmog]

What version of PAN-OS are you running against?

Also, if it's ok, could you add "receive" logging to your provider block and do TF_LOG=debug terraform refresh so I can see the XML PAN-OS is sending back to you? I don't need everything in the debug output, just the XML PAN-OS is sending back to you.

[nirav18]

PAN - OS Version: 10.0

I enabled the provider logging to "receive",

Got response as below:

Output TF block

XML response screenshot

[shinmog]

In that output, do you see a message that looks like this?

[WARN] Error setting 'device' for "(REDACTED-DEVICE-GROUP-NAME-HERE)": (ERROR MESSAGE HERE)

[nirav18]

@shinmog - No, I don't see anything like that in output. I saw panorama sys info like, hostname, ip, uptime, certificate, version, device name, family, sys mode, op mode etc. and the xml response as shown in above ss.

after that xml response, ''' provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" provider: plugin process existed: path ="xxxxxxxxxxxxxx" provider: plugin existed building apply graph to check for errors ProviderTransformer: "data.panos_device_group.get_device_from_group (expand)" All DEBUG lines, no [WARN] lines

[nirav18]

Just for reference, the output is like below xml block. provider.terraform-provider-panos_v1.10.1: 2022/06/17 14:26:50 Response =

`

`

[shinmog]

As far as I can tell, what you're describing is impossible...?

I ran some unit tests on pango (the SDK library that the panos provider uses under the hood) itself to make sure that the parsing is happening correctly and it is:

// Added the following to PaloAltoNetworks/pango/pnrm/dg/pano_test.go
// then execute using "go test ."
func TestDevices(t *testing.T) {
    mc := &testdata.MockClient{}
    ns := PanoramaNamespace(mc)

    mc.Version = version.Number{10, 0, 0, ""}
    mc.AddResp(`
<entry name="foo">
    <description>war</description>
    <devices>
        <entry name="first"/>
        <entry name="second"/>
        <entry name="third">
            <vsys>
                <entry name="vsys2"/>
            </vsys>
        </entry>
    </devices>
</entry>`)

    obj, err := ns.Get("foo")
    if err != nil {
        t.Fatalf("Error in get: %s", err)
    }
    if obj.Name != "foo" {
        t.Fatalf("Name is not foo, but %q", obj.Name)
    }
    if obj.Description != "war" {
        t.Fatalf("Description is %q, not war", obj.Description)
    }
    if len(obj.Devices) != 3 {
        t.Fatalf("Devices is len %d, not 2", len(obj.Devices))
    }

    val, ok := obj.Devices["first"]
    if !ok {
        t.Fatalf("first not in devices")
    } else if val != nil {
        t.Fatalf("first val is not nil")
    }

    val, ok = obj.Devices["second"]
    if !ok {
        t.Fatalf("second not in devices")
    } else if val != nil {
        t.Fatalf("second val is not nil")
    }

    val, ok = obj.Devices["third"]
    if !ok {
        t.Fatalf("third not in devices")
    } else if len(val) != 1 || val[0] != "vsys2" {
        t.Fatalf("third vsys list is %v, not [vsys2]", val)
    }
}

I reviewed the code for the data source, it seems to be in order.. I don't see anything that should be causing the device struct to be empty...

Can you try using the panos_device_group resource to create a device group that has a serial number in it, and then terraform plan shows that it's configured correctly..?

[shinmog]

Oh, I think I see the issue...

Let me fix this and do a new build.

[shinmog]

This should be fixed in 1.10.2

[nirav18]

It seems the output list is missing the last element. there are two devices in device group but output device (target) list contains one device only.

potentially array/list length issuu I am no expert in go, just saying if you’re iterating over a length of list in util.go, m+1 (m = len(list)) might do the trick.

[shinmog]

No, there's no off-by-one error in that code.

Try a few things for me?

1) Remove the device that does show up, then refresh the data source 2) Remove the device that doesn't show up, then refresh the data source 3) Add both in, then refresh the data source

[nirav18]

That didn't do much, Still getting one device.

I've total 80 device groups in panorama with two devices in each group, still the terraform data source return one device.

[nirav18]

Just to make clear, debug log has all two devices but final output shows only one device.

[shinmog]

Sorry, I said v1.10.2. I meant v1.10.3.

Can you confirm that you tried in v1.10.3 and it is not working?

[nirav18]

Yes I tried with 1.10.3 The output shows one device not two.

[shinmog]

I cannot reproduce what you're describing using the code as it is right now.

So, in my previous comment I've shown that pango is doing the correct thing, this leaves the possibility that there is an issue with the dumpTargets function and it's handling of map[string] []string. So I created a dummy data source and played with various contents of devices, and nothing I do is reproducing what you're describing.

The rest of this comment is showing my setup and providing you with the attempted reproduction steps I tried to take.

Clone this repo locally:

$ git clone https://github.com/PaloAltoNetworks/terraform-provider-panos.git

Add the following to ./panos/provider.go:

            "panos_device_check": dataSourceDeviceCheck(),

Create this new file ./panos/check.go (the filename doesn't matter as long as it's in the panos directory):

package panos

import (
    "log"

    "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
)

func dataSourceDeviceCheck() *schema.Resource {
    return &schema.Resource{
        Read: dataSourceDeviceCheckRead,
        Schema: map[string] *schema.Schema{
            "device": targetSchema(true),
        },
    }
}

func dataSourceDeviceCheckRead(d *schema.ResourceData, meta interface{}) error {
    con, err := panorama(meta, "")
    if err != nil {
        return err
    }

    m := map[string] []string{
        "006230": nil,
        "006226": nil,
        // If you want to play with serial numbers that have vsys specified, you can uncomment the
        // one or more of these lines, or make new ones that look like it.
        //"010": []string{"vsys1"},
        //"020": []string{"vsys2", "vsys3"},
    }

    d.SetId(con.Hostname)
    if err = d.Set("device", dumpTarget(m)); err != nil {
        log.Printf("[WARN] Error setting 'device' for %q: %s", d.Id(), err)
    }
    return nil
}

Set a development override for this provider like so:

provider_installation {
    dev_overrides {
        "registry.terraform.io/paloaltonetworks-local/panos" = "/the/path/to/the/cloned/repo/terraform-provider-panos"
    }

    direct {}
}

Then use the provider in a plan by itself like so (I did this is the cloned repo directory itself so everything is self contained), I just named the file ./plan.tf:

terraform {
    required_providers {
        panos = {
            source = "paloaltonetworks-local/panos"
            version = "1.0.0"
        }
    }
}

provider "panos" {
    hostname = "127.0.0.1"
    username = "admin"
    password = "password"
}

data "panos_device_check" "x" {}
output "ans" {
    value = data.panos_device_check.x
}

Then finally:

$ go build
$ terraform apply -auto-approve

Continue making changes to the ./panos/check.go file device map, then redo go build / terraform apply -auto-approve to see that the device map has the appropriate contents.

[jsimonetti]

Just to chime in, I have the same issue with 1.11.0. I'll see if I can create a reproducer.

[chris3ware]

This is still an issue with v1.11.1 of the provider. Please see the terraform output below:

provider "panos" {
  # PANOS_HOSTNAME
  # PANOS_API_KEY
  logging = ["send", "receive"]
}

terraform {
  required_providers {
    panos = {
      source  = "PaloAltoNetworks/panos"
      version = "~> 1.11.1"
    }
  }
}

data "panos_device_group" "all" {
  name = "dg-ingress01-${terraform.workspace}-all"
}

output "device_group" {
  value = data.panos_device_group.all
}

❯ terraform output
device_group = {
  "description" = ""
  "device" = toset([
    {
      "serial" = "007955000413181"
      "vsys_list" = toset([
        "vsys1",
      ])
    },
  ])
  "id" = "dg-ingress01-dev-all"
  "name" = "dg-ingress01-dev-all"
}

I have run TF_LOG=debug terraform apply with logging = "receive" set in the provider configuration which produces the following:

2023-11-01T08:59:24.562Z [DEBUG] ReferenceTransformer: "data.panos_device_group.all" references: []
2023-11-01T08:59:24.566Z [DEBUG] provider.terraform-provider-panos_v1.11.1: 2023/11/01 08:59:24 Sending data: url.Values{"action":[]string{"get"}, "key":[]string{"########"}, "type":[]string{"config"}, "xpath":[]string{"/config/devices/entry[@name='localhost.localdomain']/device-group/entry[@name='dg-ingress01-dev-all']"}}
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1: 2023/11/01 08:59:24 Response = <response status="success" code="19"><result total-count="1" count="1">
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:   <entry name="dg-ingress01-dev-all">
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:     <devices>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:       <entry name="007955000403464">
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:         <vsys>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:           <entry name="vsys1"/>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:         </vsys>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:       </entry>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:       <entry name="007955000413181">
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:         <vsys>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:           <entry name="vsys1"/>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:         </vsys>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:       </entry>
2023-11-01T08:59:24.931Z [DEBUG] provider.terraform-provider-panos_v1.11.1:     </devices>
... <output_omitted> ...
2023-11-01T08:59:24.935Z [DEBUG] provider.terraform-provider-panos_v1.11.1: </result></response>
2023-11-01T08:59:24.939Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-11-01T08:59:24.941Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/paloaltonetworks/panos/1.11.1/darwin_arm64/terraform-provider-panos_v1.11.1 pid=87872

As you can see the device group has two entries, but only one is displayed in the terraform output.

Two devices are also returned when querying the xml API directly with curl:

curl -k "${PANOS_HOSTNAME}/api/?type=op&cmd=<show><devicegroups><name>dg-ingress01-dev-all</name></devicegroups></show>&key=${PANOS_API_KEY}"

See this gist for the xml output: https://gist.github.com/chris3ware/4b5685c04b14e1825c241f3280e506a2