Connection timeout error

[antonhagg]

It seems that the jottaservers throw you out after some time of inactivity (and you have to authenticate again). I haven't checked how long the time is but it feels like it isn't more than 5-10 minutes. Anyway, I see this can cause some trouble, i.e. when big files are cecksum matched and no new request is sent to the server for a while. I thought of two solutions to fix this issues:

1. Have some kind of keep-alive timer that sends requests to the server every 5 minutes
2. In all requests to the server i.e. in the JFSfile() have some kind of fallback if there is a server timeout or authentication error where it re-authenticate again.

Any opinions and takes on this?

/Anton

[antonhagg]

It seems that this error was cause because of other reasons (connection dropped by ISP). So I guess it might not be an issue with jottalib.

[antonhagg]

Still gets the error sometimes... Don't know if its the ISP or jottalib...

Traceback (most recent call last):
  File "C:\Python27\Scripts\jotta-download-script.py", line 9, in <module>
    load_entry_point('jottalib==0.4.1.post1', 'console_scripts', 'jotta-download')()
  File "c:\python27\lib\site-packages\jottalib\cli.py", line 291, in download
    remote_object = jfs.getObject(abs_path_to_object)
  File "c:\python27\lib\site-packages\jottalib\JFS.py", line 867, in getObject
    o = self.get(url)
  File "c:\python27\lib\site-packages\jottalib\JFS.py", line 855, in get
    o = lxml.objectify.fromstring(self.raw(url))
  File "c:\python27\lib\site-packages\jottalib\JFS.py", line 831, in raw
    r = self.request(url, extra_headers=extra_headers)
  File "c:\python27\lib\site-packages\jottalib\JFS.py", line 823, in request
    r = self.session.get(url, headers=extra_headers)
  File "c:\python27\lib\site-packages\requests\sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "c:\python27\lib\site-packages\requests\sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\python27\lib\site-packages\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "c:\python27\lib\site-packages\requests\adapters.py", line 426, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(10060, 'A connection attempt failed because the connected party did not properly respond after a
 period of time, or established connection failed because connected host has failed to respond'))

[antonhagg]

Ok, so I have tried to find out when it timeouts and it seems that it is the case when a big file is hashed and no requests is made to the server for a while. So is there any way to have a fallback option where it re-authenticate when if it fails to connect.

[havardgulldahl]

That's interesting. How big is the file? Or - how long does it take between requests?

[antonhagg]

Its about 3 Gb, It's hard to say the time but i would guess its about 3-5 minutes or so.

[antonhagg]

Ok, so this is stretching my knowledge, but would it be possible to have something like this in JFS.py?

class JFS(object):
    def __init__(self, auth=None):
        from requests.auth import HTTPBasicAuth
        self.authenticate(self)

    def authenticate(self)
        self.apiversion = '2.2' # hard coded per october 2014
        self.session = requests.Session() # create a session for connection pooling, ssl keepalives and cookie jar
        self.session.stream = True
        if not auth:
            auth = get_auth_info()
        self.username, password = auth
        self.session.auth = HTTPBasicAuth(self.username, password)
        self.session.verify = certifi.where()
        self.session.headers =  {'User-Agent':'jottalib %s (https://github.com/havardgulldahl/jottalib)' % (__version__, ),
                                 'X-JottaAPIVersion': self.apiversion,
                                }
        self.rootpath = JFS_ROOT + self.username
        self.fs = self.get(self.rootpath)

    def close(self):
        self.session.close()

    def request(self, url, extra_headers=None):
        'Make a GET request for url, with or without caching'
        if not url.startswith('http'):
            # relative url
            url = self.rootpath + url
        log.debug("getting url: %s, extra_headers=%s", url, extra_headers)
        if extra_headers is None: extra_headers={}

        try:
            r = self.session.get(url, headers=extra_headers))

        except requests.ConnectionError:
            self.close()
            self.autenticate()
            r = self.session.get(url, headers=extra_headers))

        if r.status_code in ( 500, ):
            raise JFSError(r.reason)
        return r

[antonhagg]

Since i'm not sure the problem is because of that the server has de-authed the client or its a simple timeout. It might be worth just retrying the last request, which is a much simpler solution. By reading the documentation around requests this is what it suggests around "retries".

self.session.mount('https://',requests.adapters.HTTPAdapter(max_retries=5))

Is this a good place for the code?

  def __init__(self, auth=None):
        from requests.auth import HTTPBasicAuth
        self.apiversion = '2.2' # hard coded per october 2014
        self.session = requests.Session() # create a session for connection pooling, ssl keepalives and cookie jar
        self.session.mount('https://',requests.adapters.HTTPAdapter(max_retries=5)) # `mount` a custom adapter that retries failed connections for HTTP and HTTPS requests.
        self.session.stream = True
        if not auth:
            auth = get_auth_info()
        self.username, password = auth
        self.session.auth = HTTPBasicAuth(self.username, password)
        self.session.verify = certifi.where()
        self.session.headers =  {'User-Agent':'jottalib %s (https://github.com/havardgulldahl/jottalib)' % (__version__, ),
                                 'X-JottaAPIVersion': self.apiversion,
                                }
        self.rootpath = JFS_ROOT + self.username
        self.fs = self.get(self.rootpath)

[havardgulldahl]

So, @antonhagg, is this still a problem?

Did you try changing the code with max_retries=5?

[havardgulldahl]

This is in current HEAD, bound for 0.6. Please keep testing and reopen this if there are any issues.

see https://github.com/havardgulldahl/jottalib/commit/943c142e09dc2c72a42e45b7851ffa1dcaed3bd5