search

msabramo / requests-unixsocket

Use requests to talk HTTP via a UNIX domain socket
Apache License 2.0
207 stars 29 forks source link

mounting retry adapter #37

Closed fantamiracle closed 5 years ago

fantamiracle commented 6 years ago

I am wondering if there is a way to mount retry adapter to unix_socket session.

Here is the code snippet:

        from requests.adapters import HTTPAdapter
        from requests.packages.urllib3.util.retry import Retry

        sess = requests_unixsocket.Session()

        HTTP_REQUEST_RETRIES = 3
        HTTP_RETRY_BACKOFF_FACTOR = 0.3
        HTTP_RETRY_FORCELIST = (500, 502, 504) 

        retry = Retry(total=HTTP_REQUEST_RETRIES,
                      read=HTTP_REQUEST_RETRIES,
                      connect=HTTP_REQUEST_RETRIES,
                      backoff_factor=HTTP_RETRY_BACKOFF_FACTOR,
                      status_forcelist=HTTP_RETRY_FORCELIST)

        adapter = HTTPAdapter(max_retries=retry)
        sess.mount('http+unix://', adapter)

and here is there error:

   requesting POST http+unix://%2Frun%2Fmqerd%2Fnginx.sock/apicall
   error: 'http+unix' - Traceback (most recent call last):
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/proton/tools/client.py", line 294, in handle_untrusted_msg
     response = self._handle_cloud_unlock(data)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/proton/tools/client.py", line 146, in _handle_cloud_unlock
     response = self.sess.request("POST", url, headers=headers, data=encoded_body)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/requests/adapters.py", line 405, in send
     conn = self.get_connection(request.url, proxies)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/requests/adapters.py", line 309, in get_connection
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/urllib3/poolmanager.py", line 279, in connection_from_url
     pool_kwargs=pool_kwargs)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/urllib3/poolmanager.py", line 227, in connection_from_host
     return self.connection_from_context(request_context)
   File "/opt/venvs/op-proton/lib/python3.5/site-packages/urllib3/poolmanager.py", line 237, in connection_from_context
     pool_key_constructor = self.key_fn_by_scheme[scheme]
 KeyError: 'http+unix'

Any help is appreciated!

keith-bennett-gbg commented 5 years ago

I would prefer to pass the retry object to the requests_unixsocket.Session constructor. But we can install the retry object into the UnixAdapter instead of mounting a new custom adapter.

import requests_unixsocket
from requests.packages.urllib3.util.retry import Retry

def TimeoutRetry(session = None):
    # Set up your own Retry object. This one fits my scenario.
    # I'm connecting to a forward-proxy via UNIX socket. The forward-proxy runs
    # on a different user, validates and logs requests, and then forwards the
    # request to another server. Sometimes that server is paged-out and will
    # cause a 504 gateway timeout. In that case, I want to just retry the
    # request. My specific back-end server uses POST inappropriately so I set 
    # method_whitelist to False to allow all methods to retry.
    retries = Retry(
        total=5,
        method_whitelist=False,
        status_forcelist=[504],
        backoff_factor=1
    )

    session = session or requests_unixsocket.Session()
    adapter = session.get_adapter(requests_unixsocket.DEFAULT_SCHEME)
    adapter.max_retries = retries
    return session
msabramo commented 5 years ago

Have a look at https://github.com/msabramo/requests-unixsocket/pull/42 and let me know if that works for you.

TomGoBravo commented 5 years ago

@msabramo Hi Marc, I came here to report exactly the same problem and was happy to see the fix I had in mind merged by https://github.com/msabramo/requests-unixsocket/pull/42/files . Could you release a new version with the fix? https://files.pythonhosted.org/packages/4d/ce/78b651fe0adbd4227578fa432d1bde03b4f4945a70c81e252a2b6a2d895f/requests-unixsocket-0.2.0.tar.gz doesn't seem to contain kwargs in UnixAdapter.__init__. Thank you!