search

DobyTang / LazyLibrarian

This project isn't finished yet. Goal is to create a SickBeard, CouchPotato, Headphones-like application for ebooks. Headphones is used as a base, so there are still a lot of references to it.
730 stars 71 forks source link

Charbase issue #429

Closed sugarfunk closed 8 years ago

sugarfunk commented 8 years ago

I am trying to add an author "Marie Kondō" but LL cant handle the U+014D: LATIN SMALL LETTER O WITH MACRON. Even if I serach "Marie Kondo" GoodReads is returning Kondō so the following occurs.

Traceback (most recent call last): File "C:\LazyLibrarian\LazyLibrarian\cherrypy_cprequest.py", line 645, in respond response.body = self.handler() File "C:\LazyLibrarian\LazyLibrarian\cherrypy\lib\encoding.py", line 188, in call self.body = self.oldhandler(_args, _kwargs) File "C:\LazyLibrarian\LazyLibrarian\cherrypy_cpdispatch.py", line 29, in call return self.callable(_self.args, _self.kwargs) File "C:\LazyLibrarian\LazyLibrarian\lazylibrarian\webServe.py", line 446, in search result['authorname'].encode(lazylibrarian.SYS_ENCODING)) File "C:\Python27\lib\encodings\cp1252.py", line 12, in encode return codecs.charmap_encode(input,errors,encoding_table) UnicodeEncodeError: 'charmap' codec can't encode character u'\u014d' in position 10: character maps to

philborman commented 8 years ago

Can't reproduce it, it's working here, debug log below. What system are you running on (windows/mac/linux), and any idea what your character set is? It's working here under utf-8, your log seems to indicate "charmap" which is a new one on me!

18-Aug-2016 17:40:48 - DEBUG :: ADDAUTHOR : Searching for author with name: Marie Kondō 18-Aug-2016 17:40:48 - DEBUG :: ADDAUTHOR : [Marie Kondō] Processing info for authorID: 5589647 18-Aug-2016 17:40:49 - DEBUG :: ADDAUTHOR : [Marie Kondō] Now processing books with GoodReads API 18-Aug-2016 17:40:49 - DEBUG :: ADDAUTHOR : url http://www.goodreads.com/author/list/5589647.xml?key=ckvsiSDsuqh7omh74ZZ6Q 18-Aug-2016 17:40:49 - DEBUG :: ADDAUTHOR : author name Marie Kondō 18-Aug-2016 17:40:49 - DEBUG :: ADDAUTHOR : Found cached language [eng] for isbn [160] 18-Aug-2016 17:40:49 - DEBUG :: ADDAUTHOR : Book found: The Life-Changing Magic of Tidying Up: The Japanese Art of Decluttering and Organizing 2014 ... ... ... 18-Aug-2016 17:40:51 - DEBUG :: ADDAUTHOR : Imported/Updated 3 books for author 18-Aug-2016 17:40:51 - INFO :: ADDAUTHOR : [Marie Kondō] Book processing complete: Added 3 books to the database 18-Aug-2016 17:40:51 - DEBUG :: ADDAUTHOR : [Marie Kondō] Author update complete

sugarfunk commented 8 years ago

Im on Windows 10.

philborman commented 8 years ago

ok, if you don't mind a bit of editing, in the file LazyLibrarian.py around line 47 it says

for OSes that are poorly configured I'll just force UTF-8

if not lazylibrarian.SYS_ENCODING or lazylibrarian.SYS_ENCODING in ('ANSI_X3.4-1968', 'US-ASCII', 'ASCII'): lazylibrarian.SYS_ENCODING = 'UTF-8'

alter that so it reads...

for OSes that are poorly configured I'll just force UTF-8

lazylibrarian.SYS_ENCODING = 'UTF-8'

so you remove all the line starting with "if" and move the lazylibrarian.SYS_ENCODING line to the left so that the 'l' of lazylibrarian is directly under the '#'

That should get it working for now, please report back. I will need to push a proper fix if this doesn't introduce any other problems. Thanks.

sugarfunk commented 8 years ago

Worked like a charm. Thanks for this temporary patch recommendation.

philborman commented 8 years ago

No problem

Would be useful to know what your default encoding is set to.

Could you open a python console and enter these 3 lines and let me know the result...

import locale

locale.getpreferredencoding()

locale.getdefaultlocale()[1]

Thanks

On 18/08/16 19:29, sugarfunk wrote:

Worked like a charm. Thanks for this temporary patch recommendation.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240796358, or mute the thread https://github.com/notifications/unsubscribe-auth/ALmHOTKRUCXBBevP6AtuG8gW0LsPxA2jks5qhJaVgaJpZM4JnnJU.

sugarfunk commented 8 years ago

CP1252 https://en.wikipedia.org/wiki/Windows-1252

philborman commented 8 years ago

Thanks for that, wonder why we haven't hit it before. Maybe not many windows users read books by authors with LATIN SMALL LETTER O WITH MACRON

I will push a fix in the next hour or so, thanks for your help

On 18/08/16 19:56, sugarfunk wrote:

CP1252 https://en.wikipedia.org/wiki/Windows-1252

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240803793, or mute the thread https://github.com/notifications/unsubscribe-auth/ALmHOSvQdC5L5jq6PLqH3-BY4aOXFT2Fks5qhJy-gaJpZM4JnnJU.

sugarfunk commented 8 years ago

No problem. Thank you for resurrecting LL.

On Thu, Aug 18, 2016, 2:50 PM philborman notifications@github.com wrote:

Thanks for that, wonder why we haven't hit it before. Maybe not many windows users read books by authors with LATIN SMALL LETTER O WITH MACRON

I will push a fix in the next hour or so, thanks for your help

On 18/08/16 19:56, sugarfunk wrote:

CP1252 https://en.wikipedia.org/wiki/Windows-1252

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240803793>,

or mute the thread < https://github.com/notifications/unsubscribe-auth/ALmHOSvQdC5L5jq6PLqH3-BY4aOXFT2Fks5qhJy-gaJpZM4JnnJU .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240819928, or mute the thread https://github.com/notifications/unsubscribe-auth/ABTIKhBV0sPLwpUdC58Zxbx-lwN1Mw0uks5qhKmHgaJpZM4JnnJU .

[image: --]

Bart Dyer [image: https://]about.me/bartdyer https://about.me/bartdyer?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links

philborman commented 8 years ago

I'm enjoying the challenge.

I retired about 18 months ago and was bored, so thought I'd learn python :-)

On 18/08/16 20:55, sugarfunk wrote:

No problem. Thank you for resurrecting LL.

On Thu, Aug 18, 2016, 2:50 PM philborman notifications@github.com wrote:

Thanks for that, wonder why we haven't hit it before. Maybe not many windows users read books by authors with LATIN SMALL LETTER O WITH MACRON

I will push a fix in the next hour or so, thanks for your help

On 18/08/16 19:56, sugarfunk wrote:

CP1252 https://en.wikipedia.org/wiki/Windows-1252

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <

https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240803793>,

or mute the thread <

https://github.com/notifications/unsubscribe-auth/ALmHOSvQdC5L5jq6PLqH3-BY4aOXFT2Fks5qhJy-gaJpZM4JnnJU

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub

https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240819928, or mute the thread

https://github.com/notifications/unsubscribe-auth/ABTIKhBV0sPLwpUdC58Zxbx-lwN1Mw0uks5qhKmHgaJpZM4JnnJU .

[image: --]

Bart Dyer [image: https://]about.me/bartdyer https://about.me/bartdyer?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/DobyTang/LazyLibrarian/issues/429#issuecomment-240821376, or mute the thread https://github.com/notifications/unsubscribe-auth/ALmHOUsM2cbMDfmBu_9YYxGhjqHAl59Aks5qhKqugaJpZM4JnnJU.