Foreign language slugs and titles produce - instead of the string used

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Login, publish, new blog post.
2. Use : титульную страницу as the title, then add text, tag, 
etc.
3. Visit front page, hover or click on the title, permalink or comments url
to see; that instead of the slug, Pixie outputs a - sign.

What is the expected output? What do you see instead?
Slug is correct and contains the title.
Slug not empty.

Additional information :

I tried replacing :
     $slug = preg_replace("/[^a-zA-Z0-9]/", "", $slug);
with :
     $slug = preg_replace("/[^a-zA-
ZĄĆĘŁŃÓŚŹŻVЁЙЦУКЕНГШЩЗХЪФЫВАПРОЛДЖЭЯЧСМИТ
ЬБЮÂÀÁÄÃÊÈÉËÎÍÌÏÔÕÒÓÖÛÙÚÜÇąćęłńóśźżvёйцу�
�енгшщзхъфывапролджэячсмитьбюâàáäãêèéëîí�
�ïôõòóöûùúüç0-9]/", 
"", $slug);

in line 327 of lib_misc with not much luck. Pixie then provides the right 
title when you hover over the links using clean urls but when you then 
click on the links, you goto a 404. The string looks like it's had too 
much htmlentities() function used upon it.

Please note that it will make more sense to try to find a solution to this 
problem using the code in trunk because trunk contains a fix for the way 
Pixie 1.03 was storing foreign characters as garbage in the database.

Original issue reported on code.google.com by tonywhit...@googlemail.com on 22 Feb 2010 at 4:42

[GoogleCodeExporter]

This problem occurs in both Pixie 1.03 and in trunk.

Original comment by tonywhit...@googlemail.com on 22 Feb 2010 at 5:01

[GoogleCodeExporter]

Further on this :

I have a method to patch this that I believe should work, although I've only 
just 
ascertained that it is mainly correct. The issue has exposed an extremely bad 
flaw 
in the Opera browser (All versions) Where the href tag of the url is output as 
garbage, using my proposed fix.
I will inform the Opera developers and I have attached a test case of how I 
think I 
can fix this. Please see opera-url-test.php and test it out in your favorite 
browser.
I have tested it in chromium, firefox and konqueror and it works fine, the href 
tag 
is output as expected, however in Opera browser it does not. Please test it in 
Opera 
to expose the horrible bug I have found in the Opera browser.

If it works out that this fix does work in trunk, I'll commit it regardless of 
how 
long it takes the Opera developers too fix their mess.

Thanks,
Tony

Original comment by tonywhit...@googlemail.com on 24 Feb 2010 at 5:12

Attachments:

• opera-url-test.php

[GoogleCodeExporter]

After further investigation, it turns out that my test case is not actually a 
bug 
and it works the same in all browsers, although the source of the href tag page 
differs in different browsers.

I was able to fix this in trunk but I can't push it as an upgrade for the 
following 
reasons :
My hacky fix relies on the mysql database storing foreign characters correctly 
and I 
can't commit the fix for that as an update because it will break Pixie sites 
already 
containing foreign characters in their database. Fixing that bug has no upgrade 
path.
The fix is hacky and only fixes Cyrillic characters found in most European 
languages 
like French, It wouldn't fix Japanese characters.
The fix only works with unclean urls, I had to hack to get the clean urls 
turned off 
for each url containing a foreign language string, just to make it work with 
clean 
urls turned on. It's something to do with the url that is output and the way 
the 
browser is handling it.

The only way to fix this bug is to completely rethink Pixie's url generation 
structure. It needs to basically convert any foreign characters into numbers 
instead 
of just removing them from the string.
It is a massive job and I'm not prepared to do that. With this bug as it is, it 
makes multi language support in Pixie pointless.

Original comment by tonywhit...@googlemail.com on 24 Feb 2010 at 8:11

[GoogleCodeExporter]

Fixed in trunk r478.
I will open a separate report for BIG5 languages because supporting those 
languages 
is way beyond this issue and may require a character set switch being added to 
Pixie.

Original comment by tonywhit...@googlemail.com on 26 Feb 2010 at 2:19

• Changed state: Fixed