Encoding error downloading some of the chapters

dipu-bd / lightnovel-crawler

Generate and download e-books from online sources.

https://pypi.org/project/lightnovel-crawler/

GNU General Public License v3.0

1.49k stars 293 forks source link

Encoding error downloading some of the chapters #139

Closed miracle152005 closed 5 years ago

miracle152005 commented 5 years ago

Describe the bug While downloading the chapters for some of them I get the error "encoding error : input conversion failed due to input error, bytes 0x9D 0x3C 0x2F 0x70"

Let us know

Novel URL: http://novelfull.com/battle-through-the-heavens.html
App Version: 2.9.10
Package: [pip
OS: Ubuntu Budgie 18.04

Could you please take a look

Thanks.

Best regards miracle152005

yudilee commented 5 years ago

I have already try and success in getting error. after some checking on that chapter i found that chapter has mandarin/raw text in it like this so i think better to use other source. I recommend you to use this toc link https://www.wuxiaworld.com/novel/battle-through-the-heavens

dipu-bd commented 5 years ago

One possible solution is to cleanup these mandarin texts before processing chapter body. It should be better to implement this on app.py

dipu-bd commented 5 years ago

this should be fixed now