search

scrapinghub / portia

Visual scraping for Scrapy
BSD 3-Clause "New" or "Revised" License
9.22k stars 1.41k forks source link

Dynamic encoding possible for open and reading file or contents? #900

Open donni106 opened 4 years ago

donni106 commented 4 years ago

We have an error for web pages that are not utf-8. Here is a code line (L325) which is affected:

https://github.com/scrapinghub/portia/blob/d079667f09d453b86d3d740ed042cdd03dc7fd3f/slybot/slybot/utils.py#L322-L328

Can this encoding be dynamic based on the web page or an option set for a web page in some configs?