No checking for twitter over capacity before calling simplehson.loads(json)

[GoogleCodeExporter]

This sequence is the standard error checking procedure

json = self._FetchUrl(url[, parameters=parameters])
data = simplejson.loads(json)
self._CheckForTwitterError(data)

However, this does not compensate for capacity errors. When twitter api servers 
are overloaded, a html error is sent. See the attached html file for the 
format.  This needs to be checked before the json conversion otherwise an 
uncaught ValueError is propagate. The <title> tag in the html error message is 
set to "Twitter / Over capacity" and can easily be checked with a regex.

I propose that a twitter.TwitterError("Capacity Error...") should be generated 
in this case  so that client applications can sleep(n) and try again in a few 
moments.  Further, the above fetch, covert, check for error procedure should be 
built into a separate function, there is no reason to repeat it all over the 
code.

I am going to hack something together and post the proposed function.

Original issue reported on code.google.com by adam.a...@gmail.com on 27 Sep 2010 at 3:41

Attachments:

• overload.html

[GoogleCodeExporter]

I added this code to API class, and replace the procedure above with a single 
call to _Fetch_and_Check(), which takes the same arguments as _FetchUrl()

def _Fetch_and_Check(self,url,**kwargs):
    """Performs the FetchUrl and CheckForTwitterErrors in one call

    Args:
       url: the url to retrieve
       All other kwargs the same as _Fetchurl()
    Returns:
       A json object, or raises a TwitterError
    """

    json = self._FetchUrl(url,**kwargs)
    self._CheckForCapcityError(json)
    data = simplejson.loads(json)
    self._CheckForTwitterError(data)
    return data

  def _CheckForCapictyError(self,json):
    """Raises a TwitterError if twitter returns a capcity error url

    Args:
      json : the json returned from a _FetchUrl call
    Raises:
      TwitterError("Capcity Error") if one exists
    """
    if(re.search("<title>Twitter / Over capacity</title>",json)):
      raise TwitterError("Capcity Error")

Original comment by adam.a...@gmail.com on 27 Sep 2010 at 4:20

[GoogleCodeExporter]

of course, spelling mistakes abound ... should be

_CheckForCapacityError()

Original comment by adam.a...@gmail.com on 27 Sep 2010 at 4:32

[GoogleCodeExporter]

I've discovered that there is another html error response that will cause a 
value error, this one is differentiated by the title header "<title>Twitter / 
Error</title>"

Here is a more robust form of what I suggested above:

 def _Fetch_and_Check(self,url,**kwargs):
    """Performs the FetchUrl and CheckForTwitterErrors in one call

    Args:
       url: the url to retrieve
       All other kwargs the same as _Fetchurl()
    Returns:
       A json object, or raises a TwitterError
    """

    json = self._FetchUrl(url,**kwargs)
    self._CheckForHTMLError(json)
    try:
      data = simplejson.loads(json)
    except ValueError,e:
      print >>sys.stderr, "Error in decoding"
      print >>sys.stderr, json
      print >>sys.stderr, "-"*10
      traceback.print_exc()
      raise TwitterError("json decoding")

    self._CheckForTwitterError(data)
    return data

  def _CheckForHTMLError(self,json):
    """Raises a TwitterError if twitter returns a capcity error url

    Args:
      json : the json returned from a _FetchUrl call
    Raises:
      TwitterError("Capcity Error") if one exists
    """
    if(re.search("<title>Twitter / Over capacity</title>",json,re.M)):
      raise TwitterError("Capacity Error")

    if(re.search("<title>Twitter / Error</title>",json,re.M)):
      raise TwitterError("Techinical Error")

Original comment by adam.a...@gmail.com on 28 Sep 2010 at 3:00

[GoogleCodeExporter]

committed changeset 227:284cc18751db

Original comment by bear42 on 27 Feb 2011 at 5:58

• Changed state: Fixed