We are using this HTTP provider to make multiple requests to the same API in a single Terraform sync.
We have found that when these requests are made, a new TCP connection is created for each request instead of the connection re-use/pooling that I would typically expect in Go. This causes a problem for us - we unavoidably have to make several hundred requests to that API and the connections limit on the remote server is exhausted as a result of not having connection reuse.
Proposal
The reason for the problem I believe is that the HTTP client transport is being cloned each time in the provider's Read() method instead of relying directly on the default shared transport http.DefaultTransport (in order to guard against changes to the shared value).
I propose that we move the HTTP client and transport creation to the initialisation of the data source so we can take advantage of connection reuse. Consecutive calls to Read() can use the same instance of the client instead of building its own each time.
Incidentally, the go-retryablehttp client whose transport is replaced by the clone above itself relies on a library that creates its own instance of a http client and transport (go-cleanhttp), so if you were to use the client by itself you would have the same issue.
How much impact is this issue causing?
Medium
Additional Information
No response
Code of Conduct
[X] I agree to follow this project's Code of Conduct
Terraform CLI and Provider Versions
% terraform --version Terraform v1.5.7 on darwin_arm64
Use Cases or Problem Statement
We are using this HTTP provider to make multiple requests to the same API in a single Terraform sync.
We have found that when these requests are made, a new TCP connection is created for each request instead of the connection re-use/pooling that I would typically expect in Go. This causes a problem for us - we unavoidably have to make several hundred requests to that API and the connections limit on the remote server is exhausted as a result of not having connection reuse.
Proposal
The reason for the problem I believe is that the HTTP client transport is being cloned each time in the provider's
Read()method instead of relying directly on the default shared transporthttp.DefaultTransport(in order to guard against changes to the shared value).I propose that we move the HTTP client and transport creation to the initialisation of the data source so we can take advantage of connection reuse. Consecutive calls to
Read()can use the same instance of the client instead of building its own each time.Incidentally, the
go-retryablehttpclient whose transport is replaced by the clone above itself relies on a library that creates its own instance of a http client and transport (go-cleanhttp), so if you were to use the client by itself you would have the same issue.How much impact is this issue causing?
Medium
Additional Information
No response
Code of Conduct