problem with decoding unicode

[lloyd]

(move over here from yajl proper)

https://github.com/lloyd/yajl/issues/20

[jerzyk]

for the completion, here is original ticket content:

while using unicode data library is throwing exceptions:

In [1]: a = u'[{"data":"Podstawow\u0105 opiek\u0119 zdrowotn\u0105"}]'
In [2]: import yajl
In [3]: yajl.loads(a)
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/home/cms/sobre-cms/<ipython console> in <module>()
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0105' in position 19: ordinal not in range(128)

In [4]: import simplejson
In [5]: simplejson.loads(a)
Out[5]: [{u'data': u'Podstawow\u0105 opiek\u0119 zdrowotn\u0105'}]

same code for different libraries (here simplejson) working fine

[teepark]

fix at least for Decoder.decode() in cafdd07ee4c32239c8043b508d3f2fc842db2cae

http://docs.python.org/c-api/arg.html the issue is using the z# formatter for argument parsing. the docs there don't say how it encodes a unicode object into a char buffer, but it's clearly not "use utf8", so the change just gets explicit about that.

[teepark]

aaaand that change isn't decrefing the pybuffer in the success case. wait for a formal pull request pls