Closed a30004053 closed 3 years ago
Suspecting PAN-OS itself is having problem with the huge number of concurrent connections...
terraform apply
has an optional flag, -parallelism=n
. Try using this, setting it to 5, and see if the terraform apply
works without issue.
Note: the 5
is coming from our PAN-OS documentation (last bullet point in the note at the top of this page): https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-panorama-api/pan-os-xml-api-request-types/apply-user-id-mapping-and-populate-dynamic-address-groups-api.html
Thanks @shinmog.
@a30004053
Just curious, how long did it take to successfully terraform apply
those thousands of address objects?
@shinmog - To be honest, we took a step back and consolidated the fragmented addresses (lots of /29s /28s) into larger /16 blocks, bringing that number down to a few hundred.
To answer your question, when we did run it the first time the terraform apply created 3000 objects in roughly 10 mins
Limit the number of concurrent API calls to five. This limit ensures that there is no performance impact to the firewall web interface as the management plane web server handles requests from both the API and the web interface.
@shinmog I've been setting the limit to 4 with reasonable success. Thanks for pointing out this issue, and the above bullet in the PA docs.
If this is a limitation of PAN-OS API and ~5 threads is a general limit in reliability not just performance slow downs then it might be good to call this out on the README. Further the above excerpt doesn't really imply the level of degradation experienced. If the server is only able to process 5 requests concurrently it'd be great to see that spelled out more explicitly.
Lastly, Is there any ticket to investigate this issue on the server side or should the standard solution moving forward be to reduce the parallel requests? I'm fine with the latter, and knowing this personally I can plan for other projects that use the api, or pango/tf module, but this does potentially impact all pa client libraries and downstream modules that could be run with relatively high concurrency, ie pango, pa salt/ansible modules, or custom libraries/apis talking to PA.
Yeah, I'll add the thread limitation to the docs, as well as lifetime.create_before_destroy
needing to be on everything.
Yes, this is a PAN-OS limitation, it's just that Terraform really exposes that limit (and the consequences of exceeding it) front and center.
It doesn't really impact downstream stuff as much as you imply, tho. pan-os-python and pango don't thread, so it's up to the users of those libs to do concurrency if that's what they want. But ansible playbooks are serially executed, so there's nothing to do there.
Describe the bug
Error Unmarshaling into provided interface: EOF
error when creating large number of address_objects on PanoramaExpected behavior
Terraform plan executes successfully and provides a valid plan output
Current behavior
Error Unmarshaling into provided interface: EOF
error occurs at different stages while refreshing state, see screenshot belowPossible solution
N/A
Steps to reproduce
Screenshots
Fails while refreshing a subnet under![image](https://user-images.githubusercontent.com/57163790/109461117-ceac4e00-7ab5-11eb-9eb4-6b2d3f24bb31.png)
guestandhybridmanagement
azure servicenext run, fails while processing a subnet used by the![image](https://user-images.githubusercontent.com/57163790/109461195-e97ec280-7ab5-11eb-91e0-b8fbc2de4c1b.png)
logicappsandmanagement
serviceContext
We intend to create an address_object for each subnet pertinent to our specific region as returned in the output of the command described here Azure Service Tags. Since this list changes overtime, we have developed a pipeline which generates a tf file containing a
panos_address_object
resource for each subnet.Your Environment