search

gottfrois / link_thumbnailer

Ruby gem that fetches images and metadata from a given URL. Much like popular social website with link preview.
MIT License
512 stars 106 forks source link

If charset is not found in the header, find it from the body #139

Closed noellabo closed 5 years ago

noellabo commented 5 years ago

Older Japanese websites are encoded with Shift_JIS or EUC-JP instead of UTF-8 for historical reasons.

What's worse, often the web server doesn't return encoding information in HTTP response.

Therefore, you need to get the encoding from the HTML content's META tag.

This improvement may be useful for other language encodings with similar issues.

gottfrois commented 5 years ago

Great! Thanks