aristanetworks / cvprac

Other
46 stars 48 forks source link

Timeout during image upload #139

Closed freedge closed 1 year ago

freedge commented 3 years ago

While uploading an image, I get a timeout as below. It looks like the connection timeout (default of 10s) is used instead of the request timeout (default of 30s).

After further checking, in fact it's caused by this issue https://github.com/psf/requests/issues/5263

For POST requests, is it possible to set something higher than 10s as default timeout? Cheers

uploading new EOS
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'XXX. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.6/ssl.py", line 934, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.6/ssl.py", line 903, in send
    return self._sslobj.write(data)
  File "/usr/lib64/python3.6/ssl.py", line 601, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/app-root/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.6/ssl.py", line 934, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.6/ssl.py", line 903, in send
    return self._sslobj.write(data)
  File "/usr/lib64/python3.6/ssl.py", line 601, in write
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "upgrade.py", line 112, in <module>
    resp = clnt.api.add_image(targetImage)
  File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_api.py", line 1870, in add_image
    files={'file': image_data})
  File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 898, in post
    return self._make_request('POST', url, timeout, data=data, files=files)
  File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 664, in _make_request
    raise error
  File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 642, in _make_request
    data, files)
  File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 774, in _send_request
    raise error
  File "/opt/app-root/lib/python3.6/site-packages/cvprac/cvp_client.py", line 763, in _send_request
    files=files)
  File "/opt/app-root/lib/python3.6/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/opt/app-root/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/app-root/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/app-root/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out',))
mharista commented 3 years ago

Hi @freedge

There are two timeouts you can set via the connect method in the cvprac client. Have you tried increasing the connect_timeout and the request_timeout via the connect method? Also the client post method does allow a timeout parameter to be sent.

Connect method info:

def connect(self, nodes, username, password, connect_timeout=10,
                request_timeout=30, protocol='https', port=None, cert=False,
                is_cvaas=False, tenant=None, api_token=None, cvaas_token=None):
        Login to CVP and get a session ID and cookie.  Currently
            certificates are not verified if the https protocol is specified. A
            warning may be printed out from the requests module for this case.

            Args:
                nodes (list): A list of hostname/IP addresses for CVP nodes
                username (str): The CVP username
                password (str): The CVP password
                connect_timeout (int): The number of seconds to wait for a
                    connection.
                request_timeout (int): The default number of seconds to allow
                    api requests to complete before timing out.
                protocol (str): The protocol to use to connect to CVP.
                    THIS PARAMETER IS NOT USED AND WILL BE DEPRECATED.
                    ONLY INCLUDED TO NOT BREAK EXISTING CODE THAT HAS PROTOCOL
                    SPECIFIED IN CONNECTION.
                port (int): The TCP port of the endpoint for the connection.
                    If this keyword is not specified, the default value is
                    automatically determined by the transport type.
                    (http=80, https=443)
                cert (str or boolean): Path to a cert file used for a https
                    connection or boolean with default False. If a cert is
                    provided then the connection will not attempt to fallback
                    to http. The False default sets the request to not verify
                    the servers TLS certificate.
                is_cvaas (boolean): Flag for enabling connection to CVaaS.
                tenant: (string): Tenant/Org within CVaaS to connect to.
                    Required if is_cvaas is enabled.
                cvaas_token (string): API Token to use in place of UN/PW login
                    for CVaaS.
                api_token (string): API Token to use in place of UN/PW login
                    for CVP 2020.3.0 and beyond.

Post method info:

def post(self, url, data=None, files=None, timeout=30):

           Make a POST request to CVP.  If the request call raises an error
            or if the JSON response contains a CVP session related error then
            retry the request on another CVP node.

            Args:
                url (str): Portion of request URL that comes after the host.
                data (dict): Dict of key/value pairs to pass as parameters into
                    the request. Default is None.
                files (dict): Dict of file name to files for upload. Currently
                    only used for adding images to CVP. Default is None.
                timeout (int): Number of seconds the client will wait between
                    bytes sent from the server.  Default value is 30 seconds.
freedge commented 3 years ago

My point is that the timeout=30 option on the post method is useless (in the context of pushing a large image), as it will timeout after 10 seconds anyway. My understanding is that the code is done with sane defaults of 10s to connect to a service and 30seconds to perform requests such as POST, but practically the wrong timeout value is used.

freedge commented 3 years ago

(also I will try to play with the connect_timeout, thanks for the suggestion :) )

freedge commented 1 year ago

we added a connect_timeout in this fashion

-clnt.connect([os.environ['CVP_HOST']], 'cvpadmin', os.environ['CVP_PASS'])
+clnt.connect([os.environ['CVP_HOST']], 'cvpadmin', os.environ['CVP_PASS'], connect_timeout=120)

and it works great. Thanks