boutell
commented
10 years ago You're right. We would like to accept a pull request to fix apos.slugify so that it does something rational with non-latin characters. The trouble is that JavaScript regular expressions are just so terrible. \w doesn't upgrade smoothly to deal with non-latin character sets, for instance.
It looks like xregexp is the answer:
http://xregexp.com/plugins/#unicode
This would allow the use of unicode categories that offer a much broader concept of "letters" and "numbers," which should result in a reasonable slugifier for more languages.
Right now in search.js the slugify method does this:
var r = "[^A-Za-z0-9" + RegExp.quote(options.allow) + "]"; var regex = new RegExp(r, 'g'); s = s.replace(regex, options.separator);
That clobbers all the characters that are not letters or digits and replaces them with the separator (-) and later we stomp consecutive separators.
This is a good approach, we just need to use xregexp and Unicode categories in place of "A-Za-z0-9", and use xregexp rather than vanilla regular expressions.
We also need to convert to lowercase in a more language-independent way (toLowerCase isn't very smart).
It might be best to exclude unwanted things specifically (like the "punctuation" unicode category) rather than allowing only a small number of things, since Chinese ideograms might be valid in slugs and they are not letters or numbers...
So you see it's a bit complicated.
(In Apostrophe 1.5, built in PHP, we got away with using the Unicode upper and lower case letter and number categories and no one complained about the Chinese thing. But that doesn't mean we shouldn't do better if we can.)
I hope this helps you down the road to patching apos.slugify. If not, it'll help me eventually. (:
On Sun, Jan 19, 2014 at 5:55 PM, Danny Povolotski notifications@github.comwrote:
Right now if the title is 100% non-latin characters, no slug is created. Also, no slug editing is permitted upon creation of a page.
I'm not sure this is the best approach, since the resulting slug is "none", and for each and every single page, an end-user would have to go back to "Page Settings" and add a slug.
I see two solutions: 1) Add slug field that would be auto-populated from the title, and would be validated to be non-empty 2) Enable non-latin characters in the automatically created slug 3) Both, with some sort of setting allowing or disallowing UTF8 in slugs globally (I'm a fan of this one ;))
Internationally it's more and more common to see slugs in non latin characters.
— Reply to this email directly or view it on GitHubhttps://github.com/punkave/apostrophe-pages/issues/6 .
Tom Boutell Lead Developer P'unk Avenue 215 755 1330 punkave.com window.punkave.com
etodanik
Right now if the title is 100% non-latin characters, no slug is created. Also, no slug editing is permitted upon creation of a page.
I'm not sure this is the best approach, since the resulting slug is "none", and for each and every single page, an end-user would have to go back to "Page Settings" and add a slug.
I see two solutions: 1) Add slug field that would be auto-populated from the title, and would be validated to be non-empty 2) Enable non-latin characters in the automatically created slug
and: 3) Both, with some sort of setting allowing or disallowing UTF8 in slugs globally (I'm a fan of this one ;))
Internationally it's more and more common to see slugs in non latin characters.