Open JerradGit opened 2 years ago
Hi there,
Thanks for the detail and reproduction info. Super useful.
Yikes. Ok. That's not good. In all of my testing, I came across this once and it seemed to level off at about 5-10 connections. The system kind of self-regulated as parts of the tree converged etc. However, as to your test, you've found an unhappy path with the way the transport layer works. Thanks for flagging this.
I'll tag this as an enhancement and work on it. Historically, the TF transport layer was locked to a single transport instance which limited the connection to exactly one, but it took forever to complete.
Would a configurable worker pool of connections work for you? Something we could put in the provider config as input?
I think a worker pool would be fine, I can test locally on a dev branch when something is ready and report back.
@davedotdev
I have created a PR on the go-netconf
library that addresses the transport part of this issue. Using it in conjunction with this provider the time it takes to deploy has been greatly reduced.
Previously we were seeing upwards of 10+ minutes for a deployment on the same scale, this is also performing a commit check
before we fully commit it.
Apply complete! Resources: 721 added, 0 changed, 0 destroyed.
terraform apply -auto-approve 298.09s user 5.12s system 329% cpu 1:32.02 total
Like I mentioned in the PR on the other library, I actually went with a worker pool initially and that just ends up killing the netconf subsystem regardless of any settings I tried.
Summary
We want to use JTAF to provision 100% of device configurations which results in hundreds if not thousands of resources which causes the plugin to fail randomly.
After investigation we have determined that the JTAF plugin is establishing and terminating a TCP sockets towards the device for each resource which causes Junos to enforce connection limits and refuse new connections. This issue can occur very randomly depending on the performance of your terraform client device. e.g. Fast CPU with no competition for resources you can expect this to occur more frequently.
e.g. vqfx messages output during a few seconds of an apply
We have found the following work around for now which allows us to get through the apply, however 700+ resources can take over 10 minutes to apply against a vqfx on the same device as the terraform client.
You can re-produce this with this sample HCL
I don't know if the new socket per resource is by design or if it can be changed. While the limit values can be adjusted, this creates a larger DoS surface which would be nice to avoid.