Wrong text encoding in preview text

rkfg commented 6 years ago

Description

Wrong text encoding is used in the preview text on some sites.

Steps to reproduce

post an url like http://www.interfax.ru/russia/600675
the preview text looks like "Ïåðâûé êàíàë" îïðîâåðã èíôîðìàöèþ îá óõîäå Åêàòåðèíû Àíäðååâîé èç ïðîãðàììû "Âðåìÿ" - Interfax.ru Àíäðååâà âåäåò ïðîãðàììó áîëåå 20 ëåò è âñå åùå îñòàåòñÿ åå âåäóùåé which is obviously wrong, it's CP-1251 text decoded as CP-1252. The page contains <meta http-equiv="Content-Type" content="text/html; charset=windows-1251" /> so browsers display it fine.
try another page from the same site, for example http://www.interfax.ru/russia/
the text is decoded correctly and reads В России - Новости Интерфакс Новости в России.

The difference seems to be in the response headers. Where the text is decoded correctly the Content-Type returned by the server is text/html; Charset=windows-1251. Where it's wrong it's just text/html. I think the http-equiv value should have precedence over the header (if it's parsed at all).

Version information

Homeserver: own homeserver
Version: Synapse/0.26.0
Install method: package manager
Platform: set up on Debian 9.3 amd64.

turt2live commented 6 years ago

The client bug is https://github.com/vector-im/riot-web/issues/5473

tleydxdy commented 6 years ago

also happends on android it seems

hawkowl commented 6 years ago

This ought to work as intended next version, although malformed pages may fail the heuristics we've put in to detect when the http-equiv meta tag is being used. For the given examples, though, it should all work fine.

rkfg commented 3 years ago

I tested this exact URL on matrix.org just now and it still shows wrongly encoded text, can we reopen this please?

squahtx commented 3 years ago

Synapse would detect the correct encoding, except the <meta http-equiv="Content-Type" ... declaration for that particular URL lies just outside of Synapse's 1024 byte cutoff.

matrix-org / synapse