Error Unmarshaling into provided interface: EOF

[a30004053]

Describe the bug

Error Unmarshaling into provided interface: EOF error when creating large number of address_objects on Panorama

Expected behavior

Terraform plan executes successfully and provides a valid plan output

Current behavior

Error Unmarshaling into provided interface: EOF error occurs at different stages while refreshing state, see screenshot below

Possible solution

N/A

Steps to reproduce

1. Write a simple deploy.tf containing a large >1000 number of address_objects, for instance in our context we are creating an address_object for each azure service subnet specific to a region. see: azure service tags
2. Run terraform plan

Screenshots

Fails while refreshing a subnet under guestandhybridmanagement azure service

next run, fails while processing a subnet used by the logicappsandmanagement service

Context

We intend to create an address_object for each subnet pertinent to our specific region as returned in the output of the command described here Azure Service Tags. Since this list changes overtime, we have developed a pipeline which generates a tf file containing a panos_address_object resource for each subnet.

Your Environment

• Version used: v1.6.3
• Environment name and version: Terraform v0.13.3
• Operating System and version (desktop or mobile): Ubuntu 18.04 and CentOS 7.9

[shinmog]

Suspecting PAN-OS itself is having problem with the huge number of concurrent connections...

terraform apply has an optional flag, -parallelism=n. Try using this, setting it to 5, and see if the terraform apply works without issue.

[shinmog]

Note: the 5 is coming from our PAN-OS documentation (last bullet point in the note at the top of this page): https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-panorama-api/pan-os-xml-api-request-types/apply-user-id-mapping-and-populate-dynamic-address-groups-api.html

[a30004053]

Thanks @shinmog.

[shinmog]

@a30004053

Just curious, how long did it take to successfully terraform apply those thousands of address objects?

[a30004053]

@shinmog - To be honest, we took a step back and consolidated the fragmented addresses (lots of /29s /28s) into larger /16 blocks, bringing that number down to a few hundred.

To answer your question, when we did run it the first time the terraform apply created 3000 objects in roughly 10 mins

[jubrad]

Limit the number of concurrent API calls to five. This limit ensures that there is no performance impact to the firewall web interface as the management plane web server handles requests from both the API and the web interface.

@shinmog I've been setting the limit to 4 with reasonable success. Thanks for pointing out this issue, and the above bullet in the PA docs.

If this is a limitation of PAN-OS API and ~5 threads is a general limit in reliability not just performance slow downs then it might be good to call this out on the README. Further the above excerpt doesn't really imply the level of degradation experienced. If the server is only able to process 5 requests concurrently it'd be great to see that spelled out more explicitly.

Lastly, Is there any ticket to investigate this issue on the server side or should the standard solution moving forward be to reduce the parallel requests? I'm fine with the latter, and knowing this personally I can plan for other projects that use the api, or pango/tf module, but this does potentially impact all pa client libraries and downstream modules that could be run with relatively high concurrency, ie pango, pa salt/ansible modules, or custom libraries/apis talking to PA.

[shinmog]

Yeah, I'll add the thread limitation to the docs, as well as lifetime.create_before_destroy needing to be on everything.

Yes, this is a PAN-OS limitation, it's just that Terraform really exposes that limit (and the consequences of exceeding it) front and center.

It doesn't really impact downstream stuff as much as you imply, tho. pan-os-python and pango don't thread, so it's up to the users of those libs to do concurrency if that's what they want. But ansible playbooks are serially executed, so there's nothing to do there.