Open vabagaria opened 3 years ago
Your provider timeout
is set to 60 right now - if you increase that (say 120 or whatever) does it finish ok?
Your provider
timeout
is set to 60 right now - if you increase that (say 120 or whatever) does it finish ok?
Setting it to any higher than 60 results in an error as it is the maximum timeout value.
I'll increase the max from 60 to 600 in the next provider release.
Your provider
timeout
is set to 60 right now - if you increase that (say 120 or whatever) does it finish ok?Setting it to any higher than 60 results in an error as it is the maximum timeout value.
I've seen this issue as well.
Ok, v1.7.0 is out now. I set the timeout to 600 sec now, IIRC, so increase that and let me know the results.
For those still having this issue with 1.7, instead of using username and password try generating an API key using the Go code provided in the TF provider documentation and pass that to TF. This seems to have solved the issue for me.
I slightly misspoke above: I didn't increase the default timeout to 600 sec, I increased the max timeout to 600 sec. I'm uncertain if having a max timeout is a useful thing or not.. Will reassess after this issue is resolved.
With regards to what @vabagaria said above - I actually removed that chunk of documentation because I wasn't sure if it was a useful thing to have or not. So instead of asking people to install pango and compile code to get a simple answer, I've just released another provider version, 1.8.0, which has panos_api_key
that you can use to find out the API key for the provider auth you've supplied.
@vabagaria Seems like this issue is resolved..?
@shinmog I haven't been able to reproduce the issue on tf-plan with 1.8 but I have gotten session time out errors on tf-apply.
So you have the timeout set to 600 seconds and it times out..? Does it actually sit around and take those 10 minutes or..?
@shinmog I'm also seeing this issues or at least something very similar.
So you have the timeout set to 600 seconds and it times out..? Does it actually sit around and take those 10 minutes or..?
I've been setting the following in my env to work around this: TF_CLI_ARGS_destroy="-parallelism=2"
Since I see this issue with a few local exec provisioners running python pa libraries which cover configure resources not supplied by this terraform provider I can only assume this is an issue with the PA server or a general issue with both client modules.
I think parallel requests are causing a Session timed out
, not any http or tcp/network issues. The Session timed out
issue oddly enough happens with a server HTTP response of 200 ok...
To figure out if this was really Parallel requests I setup a test environment which creates many static entries in the PA DNS proxy, this uses python, but again I think the issues are related. When run in series I've completed over 4k requests with no timeout issues, however, running this test in parallel ~10-20 threads, I'm able to occasionally get a session timeout, upping to ~60 threads concurrently creating 10 records I'm able to repro the timeout consistently.
I have created a pr for pan-python which immediately reads and closes the http connection in a context manager.. With that in place I don't see the python issue anymore, but I saw it far less frequently in python than pango/tf to begin with. I believe closing the connections just lowers the likelihood of hitting the problem and I'm seeing it go because there are more resources defined with terraform panos resources.
My guess now is that there's something up with the server, it either struggles with responding to high volumes of requests, or there's some random collision that can occur.
If this is a PA server error would something like the following be possible? Perhaps with exponential backoffs? https://github.com/PaloAltoNetworks/pango/compare/master...jubrad:retry-on-timeout?expand=1
Ok, if this is a parallelism issue, then I've come across this already in the provider:
https://github.com/PaloAltoNetworks/terraform-provider-panos/issues/264#issuecomment-788118806
Describe the bug
Frequently receive 'Error: Session timed out' when trying to run plan or apply.
Expected behavior
Should finish plan and display changes or apply planned changes, every time.
Current behavior
Terraform quits with error 'Session timed out' frequently but not always.
Possible solution
This problem can be resolved by forcing terraform to only perform one operation (using -parallelism=1 to disable parallel requests) which is not a practical fix. Other possible solutions could include:
Steps to reproduce
The following is from a redacted/example plan where config matches production but values have been replaced with dummy data. Problem can be reproduced in lab with separate firewall and terraform host. It is not re-producible when Ansible is used with PanOS Collection with the same firewall and host.
Terraform code:
Screenshots
Context
Trying to configure a number of firewalls with security rules and address objects for multiple apps and running plan/apply multiple times a day. The pipeline is failing frequently and needs to be re-run manually.
Your Environment
Version used: