search

exoscale / cs

A simple, yet powerful CloudStack API client for python and the command-line.
BSD 3-Clause "New" or "Revised" License
88 stars 36 forks source link

Support user-provided Requests HTTP session #107

Closed falzm closed 5 years ago

falzm commented 5 years ago

This change adds support for user-provided Requests HTTP session attribute, allowing the caller for example to implement a retry policy for HTTP calls.

falzm commented 5 years ago

To test, I've added the following code to the cs/__init__.py file:

diff --git a/cs/__init__.py b/cs/__init__.py
index 5a3583c..76c1baa 100644
--- a/cs/__init__.py
+++ b/cs/__init__.py
@@ -96,7 +96,32 @@ def main(args=None):
         config['method'] = 'post'
     if options.trace:
         config['trace'] = True
+
+    ###
+    import requests
+    from requests.adapters import HTTPAdapter
+    from requests.packages.urllib3.util.retry import Retry
+    max_retry = 5
+    retry_backoff_factor = 0.3
+
+    adapter = HTTPAdapter(
+        max_retries=Retry(
+            total=max_retry,
+            backoff_factor=retry_backoff_factor,
+            status_forcelist=None,
+        )
+    )
+
+    session = requests.Session()
+    session.mount("http://", adapter=adapter)
+    session.mount("https://", adapter=adapter)
+
+    config["session"] = session
+    ###
+
     cs = CloudStack(**config)
+
+

We can see the retry policy in effect by specifying a bogus API endpoint in the configuration:

Without retry policy (commenting config["session"] = session)

$ time cs listApis
Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x10f708f50>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /?apiKey=EXOxxxxxxxxxxxxxxxxxxxxxxxx&command=listApis&response=json&signatureVersion=3&expires=2019-08-27T16%3A08%3A25%2B0000&signature=QhD1WJyQRKScRcScLlWAoW9S12I%3D (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10f708f50>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/bin/cs", line 11, in <module>
    load_entry_point('cs', 'console_scripts', 'cs')()
  File "/Users/marc/Documents/Exoscale/git/cs/cs/__init__.py", line 130, in main
    **kwargs)
  File "/Users/marc/Documents/Exoscale/git/cs/cs/client.py", line 213, in handler
    return self._request(command, **kwargs)
  File "/Users/marc/Documents/Exoscale/git/cs/cs/client.py", line 276, in _request
    cert=self.cert)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /?apiKey=EXOxxxxxxxxxxxxxxxxxxxxxxxx&command=listApis&response=json&signatureVersion=3&expires=2019-08-27T16%3A08%3A25%2B0000&signature=QhD1WJyQRKScRcScLlWAoW9S12I%3D (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10f708f50>: Failed to establish a new connection: [Errno 61] Connection refused'))

real    0m0.234s
user    0m0.196s
sys 0m0.032s

With retry policy

$ time cs listApis
Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x10906d710>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    **response_kw)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    **response_kw)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    **response_kw)
  [Previous line repeated 2 more times]
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /?apiKey=EXOxxxxxxxxxxxxxxxxxxxxxxxx&command=listApis&response=json&signatureVersion=3&expires=2019-08-27T16%3A06%3A37%2B0000&signature=2V03wk1Q3PNfgDbv4PiT2QDCwbE%3D (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10906d710>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/bin/cs", line 11, in <module>
    load_entry_point('cs', 'console_scripts', 'cs')()
  File "/Users/marc/Documents/Exoscale/git/cs/cs/__init__.py", line 130, in main
    **kwargs)
  File "/Users/marc/Documents/Exoscale/git/cs/cs/client.py", line 213, in handler
    return self._request(command, **kwargs)
  File "/Users/marc/Documents/Exoscale/git/cs/cs/client.py", line 276, in _request
    cert=self.cert)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/marc/Documents/Exoscale/git/cs/.venv/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /?apiKey=EXOxxxxxxxxxxxxxxxxxxxxxxxx&command=listApis&response=json&signatureVersion=3&expires=2019-08-27T16%3A06%3A37%2B0000&signature=2V03wk1Q3PNfgDbv4PiT2QDCwbE%3D (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10906d710>: Failed to establish a new connection: [Errno 61] Connection refused'))

real    0m9.265s
user    0m0.207s
sys 0m0.037s
greut commented 5 years ago

The risk I see is that you reuse a closed session.

https://github.com/psf/requests/blob/3e7d0a873f838e0001f7ac69b1987147128a7b5f/requests/sessions.py#L428