Problems with multi-byte UTF-8 characters

[michael-e]

The extension does not handle multibyte UTF-8 characters properly.

• When searching for a word containing multibyte characters (e.g. "döner"), results can be found. But in the excerpts the keyword will not be enclosed in a <strong/> tag.
• Furthermore there will be no search-suggestions if the search keyword contains a multibyte character. Searching for döner will not show any suggestions even if your field values contain döner or superdöner.
• In the search suggestions, multi-byte characters are displayed wrong. For example, when you search fpr superd and you have fields containing superdöner, the suggestion superdöner will actually be superdöner (which is a typically wrong multibyte representation).

So I guess the extension needs some utf-encoding/-decoding functions in the right places so that PHP can work with the strings correctly. Unfortunately I don't know enough about the inner workings to fix this myself, but I will do testing! (I really think that this extension is a big step forward for Symphony as a whole.)

[nickdunn]

Thanks mate. I must admit I haven't tested with characters outside of English so I'm not surprised it doesn't work with multibyte chars. Do you have any other examples I can test with? I'll take a look into this next week - I know a few places that are troublesome.

[michael-e]

I will try and select another example. But I am rather sure that the döner will reveal nearly everything! :-)

Do you know what a Döner is? Well, it's a doner. I like the word in German, it sounds so funny!

[nilshoerrmann]

Michael, search for kebap and everything will be fine ;)

[nickdunn]

In English it's a boring "donna" but I'm guessing German is more "durna"?

[michael-e]

Ah, no, the "ö" sounds more funny here. :-) Like in the word "schön".

[michael-e]

http://dict.leo.org/ende?lp=ende&lang=de&searchLoc=0&cmpType=relaxed&sectHdr=on&spellToler=&search=döner

Click on the loudspeaker symbol.

[nickdunn]

Pretty sure I've fixed these in: https://github.com/nickdunn/search_index/commit/68d2b394539f93c7be0d011d749cef262e4bc7a6

• use of mb_ string functions when they are available (didn't fix any of the bugs you discovered, but good practice nonetheless)
• a regex allowing only a-zA-Z0-9 was removing your ö character. Oops! Replaced with a better punctuation-stripping function that is used elsewhere
• malformed characters in the search suggestion DS was caused by the utf8_encode function which doesn't like encoding strings that are already UTF8. So I only now run strings through this when they're not already UTF8

All three of your bugs are fixed by the above commit.

To re-test you will need to:

• delete all rows in sym_search_index_keywords
• re-index your entries (Search Index > Indexes), select rows and "Re-Index" from the With Selected menu

[michael-e]

That looks very good now, Nick.

Well, one thing: The excerpt-length setting does not work anymore.

[nickdunn]

I'm pretty sure this didn't work before either :-/

[michael-e]

Well, at least the excerpts were a lot shorter than they are now. I haven't counted characters. I should have done that. :-)

[nickdunn]

Will raise it as another ticket as it doesn't seem related to the changes I made today. It's more down to the massive chunks of code that I lifted from Drupal 5. I'll see how Drupal 7/8 did it ;-)

[michael-e]

Thanks, Nick.

[nickdunn]

(I really think that this extension is a big step forward for Symphony as a whole

Cheers! Any other feedback or ideas you've got, please let me know.

[nickdunn]

This was caused by the mb string changes. There is a simple fix I have committed locally to rectify the excerpt length bug.

[michael-e]

Glad to hear this. You will push it soon, right?

Can you explain why entites in the excerpt are encoded (as posted here)? Is this an issue or a feature?

[nickdunn]

Glad to hear this. You will push it soon, right?

Pushed.

Can you explain why entites in the excerpt are encoded... Is this an issue or a feature?

Both ;-) The excerpt is run through General::sanitize to HTML-encode things like ampersands that break XML, hence everything is encoded. However this happens after the <strong> tags are wrapped around the highlighted keywords, so these tags also get encoded. The net result is that you have to use an xsl:value-of with disable-output-escaping so that the entities are rendered properly. It works, but is a bug ugly.

I have just changed the implementation so that the string is HTML-encoded and then the <strong> tags are added. This means that you now have to use an xsl:copy-of to get the excerpt (a value-of will work but will just get the plain text, not the strong elements too).

<xsl:copy-of select="excerpt/text() | excerpt/*" />

That itself is a bit messy, so I decided to wrap the excerpt in a <p> tag, so the XSLT becomes:

<xsl:copy-of select="excerpt/*" />

Nice and simple, just like a Markdown-formatted textarea.

[michael-e]

Thanks a lot for the fix! At first I was a little bit amazed at the the length difference from one excerpt to another (in the same search). But then I realized that there is some "intelligent logic" behind this, so the length of an excerpt depends on the position of a match in the string and the number of matches alike.

So: Well Dunn! :-)

I like the new implementation as well. It is clear and straightforward. And in case you don't like the excerpt to be wrapped in the <p> Element, you can still use

<xsl:copy-of select="excerpt/*/*|excerpt/*/text()"/>

[nickdunn]

so the length of an excerpt depends on the position of a match in the string and the number of matches alike

Yep. I did very little beyond modifying Drupal's search_excerpt function which does the hard work. It means, however, that the excerpt doesn't match exactly the length set in the config — it's just a rough length. But I don't ever see a need for it to be completely precise.