Latin1 / UTF-8 issue

[GoogleCodeExporter]

Currently, when generating HTML, the board will output latin1 if all the
characters can be represented as such. But the web server sends the
Content-type: encoding=utf-8 header, which breaks things.

Preferred fix would be to make it default to utf-8 instead of defaulting to
latin1 and fallbacking on utf-8, but I suppose that setting the
Content-type header to latin1 when we're sending latin1 also works.

Original issue reported on code.google.com by eksopl on 6 Mar 2009 at 1:58

[GoogleCodeExporter]

EASIEST FIX I EVER WROTE

I'll try to research it in depth later, right now it works just fine.

Original comment by andrey.o...@gmail.com on 6 Mar 2009 at 4:31

• Changed state: Started

[GoogleCodeExporter]

Hahah, oh wow.

Also, I just realized both /a/ and /jp/ links at the top have "Japan/General" as
their title.

And oh wow, Firebug is pretty awesome, it even does what LiveHTTPHeaders does, 
minus
the replay POST requests thing.

Original comment by eksopl on 6 Mar 2009 at 4:41

[GoogleCodeExporter]

UTF-8 in perl sucks.

Making perl run with perl -C rather than just perl seems to make it use UTF-8 
when
reading and writing to streams. Or you can use binmode STDOUT, ":utf8"; to make 
stout
use utf-8.

-utf8 in CGI.pm is also weird as hell. When you enable -C, it works fine with
variables passed through GET, but it chokes on variables passed through POST. 
If you
don't enable -C and use binmode STDOUT, ":utf8", it seems to fuck up the "That 
was
VIP Quality" messages?

If all else fails, running with -C and Encode::decode_utf8'ing all CGI'd 
variables
seems to work. Which sucks.

http://ahinea.com/en/tech/perl-unicode-struggle.html

Original comment by eksopl on 7 Mar 2009 at 5:36

[GoogleCodeExporter]

Does the problem still persist?

I posted unicode stuff right now, and there

Anyway, CGI.pm reads GET through environments variables, and -utf8 is a hint 
that
they are in utf8. It reads POST from STDIN, so binmode *STDIN,":utf8" before 
use CGI
qw/-utf8/; would probably do the trick.

Original comment by andrey.o...@gmail.com on 7 Mar 2009 at 8:38

[GoogleCodeExporter]

I just reverted to the 〜 hack. I think I'll try that.

Original comment by eksopl on 7 Mar 2009 at 8:42

[GoogleCodeExporter]

Cannot decode string with wide characters at
/usr/local/lib/perl5/5.8.9/mach/Encode.pm line 170.

That's what happens when you use both binmode STDIN and use CGI qw/-utf8/. 
Using perl
-C or not using -C and then using binmode STDIN + binmode STDOUT seems to do 
the same
thing.

Can't really figure it out.

Original comment by eksopl on 7 Mar 2009 at 8:58

[GoogleCodeExporter]

Original comment by andrey.o...@gmail.com on 8 Mar 2009 at 8:46

• Changed state: Fixed