unreadable code in response

binux / pyspider

A Powerful Spider(Web Crawler) System in Python.

http://docs.pyspider.org/

Apache License 2.0

16.48k stars 3.69k forks source link

unreadable code in response #847

Open ghost opened 5 years ago

ghost commented 5 years ago

pyspider version:
Operating system:
Start up command:

Expected behavior

Actual behavior

How to reproduce

ghost commented 5 years ago

when I use requests.session().post(), response is readable: In altiore thema Amir Khusrau

Bug, when i use pyspider, response.text is readable code: à | Sa | Ÿà§ à | Puer | -§ ° | ¾à | ® a | | -à§§à§§ a | | Â§ † ‡ ° | ªà§ a | | ° F ","à¦šà¦Ÿà§ à¦Ÿà¦—à§ à¦°à¦¾à¦®-à§§à§§ à¦†à¦¸à¦¨à§‡à¦° à¦ªà§ à¦°

ghost commented 5 years ago

Setting content-type in heades worked. headers = {'content-type': 'utf-8'} self.crawl(url='', data=post_data, headers=headers, timeout=120, callback=self.index_page, save={'start': start}, method="POST")