Open jd opened 6 years ago
sileht
commented
6 years ago urllib3 connection pool size is handled by python-request and can be configured like.
a = requests.adapters.HTTPAdapter(pool_maxsize=N)
s = requests.Session()
s.mount('https://', a)
s.mount('https://', a)But the request.Session() used by swiftclient is not customisable: https://github.com/openstack/python-swiftclient/blob/master/swiftclient/client.py#L384
swiftclient have to be fixed first...
jd
commented
6 years ago There is an upstream bug opened here https://bugs.launchpad.net/python-swiftclient/+bug/1671949
zaitcev
commented
6 years ago Please pardon my lack of understanding, but I do not follow how the underlying symptom ended with a desire to make the connection pool larger. I say that because, I think, BadStatusLine('') means that the client opens the TCP connection, possibly establishes TLS, sends PUT down, but a read() from the socket ends with EOF. Either Swift's own proxy or its LB (HAproxy in case of RHOS) closed the connection without returning anything.
If we look into urllib3/connectionpool.py:HTTPConnectionPool.urlopen(), then indeed there's no telling who's raising the HTTPException - _get_conn() or _make_request(). Both of them end with "Connection aborted." But it seems to me that only _make_request() can produce BadStatusLine('').
But if so, sizing the pool isn't going to do Gnocchi any good. The Swift or HAproxy which it is using must be tuned for larger number of connections. Heck I'd have a look if someone's segfaulting. As long as it's only a traceback, Swift is good at returning at least 500. But in this case, see the scenario above.
Julien, is there something that am I missing here?
jd
commented
6 years ago Either Swift's own proxy or its LB (HAproxy in case of RHOS) closed the connection without returning anything.
I did not have time to dig into that back then, but the swift CLI tool was working properly, and gnocchi-upgrade ran properly. So Swift was functioning.
I don't know what kind of deployment it was and @Prophidys could give us a hint here. We may be able to access the system to take a look
The thing is that as you say, there is a high number of connections opened by Gnocchi due to the way it parallelizes the tasks. Maybe one the source of the problem is that Swift proxy was not configured to handle that many connections (though I kinda doubt it), though the connection pool not being configurable is still a problem :)
When Swift is used as an incoming measures driver, the number of connections opened by httplib3 is too small to succeed. On large batches, the error occurs: