Vernacular scripts (utf8) must display properly

MrDys commented 12 years ago

CODEBASE-2: concerned about display of UTF8 due to Ruby being less UTF8 friendly.

Latest Ruby is supposed to make this easier in some way ...

We know Bob got the searching working (yay bob!) with solrmarc.

It's a deal breaker for us if we don't get the non-latin stuff displaying well. Our librarians have already tested our vufind searchworks for at least the following:

chinese hebrew arabic russian cyrillic all sorts of diacritics

right to left issues for hebrew arabic (and whatever else)

we have "display" fields in our index and can hammer this stuff into submission, but does ruby have any cool built in stuff to do it (or gems or something)

There is already some example data in the solrmarc svn project. It's currently in the 2.0 branch, under test/data

unicornWHoldings hebrew diacriticTests non-latin <-- I didn't put that there, so I don't know what's in it.

If we need more examples, I can certainly get them. I'm only confident there's already hebrew and chinese in there, and lots of diacritics. I was trying to write tests for the diacritics, but I couldn't get them to pass and fail at the right times.

Naomi

MrDys commented 12 years ago

Original reporter: ndushay

MrDys commented 12 years ago

ndushay: Bess is also going to send Stanford some specific examples of vernacular scripts (chinese at least) displaying in a blacklight UI.

MrDys commented 12 years ago

ndushay: careful with "composed" vs "decomposed" chars (Bob will know - I probably have terminology wrong)

MrDys commented 12 years ago

bess: Can you give us some specific records that it appears are not displaying correctly right now? Via either search works, or Blacklight?

MrDys commented 12 years ago

ndushay: This should probably be split into a couple of issues:

A. sorting of non-latin scripts. Within some languages (chinese? see below) and interfiled with latin scripts.

B. Right-to-Left vernacular content display.

On Mar 13, 2009, at 11:54 AM, Tom Cramer wrote:

Naomi,

I'm not sure what your citing with "broken UTF-8 display", but wrt supporting vernacular scripts in Unicode in the BL display, there are three separate classes of issues:

do the scripts display properly in search results and detailed record pages, especially for entries in right-to-left scripts? It is tricky to display fields that have entries in both roman script and Hebrew/Arabic in an intelligible fashion. Socrates does this now, though it took some work.

This may be an indexing issue. The index needs to have the appropriate fields as "display" fields. In our vufind implementation, I relied on java's BIDI and the ordering of MARC field concatenation. I got "mostly" there, as I recall. Bob Haschart may be able to improve this.

Indexing; is the vernacular script data indexed, and accessible for vernacular script search arguments? (I think this is already in place for BL, no?)

UVa needs to included fielded searching (as a pulldown "everything" "title" "author" "subject" "ISBN" ... in the plugin). Naomi needs to index vernacular fields properly and set up the fielded searches properly. Bob already did the work for solrmarc and it's wonderful.

Sorting: do the results of UTF-8 searches sort in the appropriate alphanumeric order (whatever that may mean on a language by language basis--in Chinese, e.g., I believe sort order is determined by number of strokes in the first pictographic character, not by the alphabet).

I don't know who should tackle this. Bob? We have a title_sort field ... that doesn't sort correctly. We have a way to map Latin diacritics to plain chars (borrowed from your code), but I'm not sure that's working ... and not sure what to do about non-latin chars.

Having just converted Symphony to Unicode and partnered with Sirsi to get iLink to display and sort results appropriately, we have a solid set of test cases and internal expertise to help get BL up to snuff on these. Lauren would be the key resource in coordinating this testing and feedback from the appropriate language experts.

There is at least one Arabic record with vernacular script in the blacklight demo data. Basically, use the language facet to find Hebrew and Arabic. If you don't have someone to look at Hebrew or Arabic results, we might be able to help.

MrDys commented 12 years ago

ndushay: This has been split into two separate issues, so this one is now closed.

MrDys commented 12 years ago

bess: I won't really be able to test this until we can do some more complex indexing, which will be enabled by using solrmarc.

MrDys commented 12 years ago

bess: The problems expressed here are covered separately in other tickets.

MrDys commented 12 years ago

projectblacklight / blacklight

Vernacular scripts (utf8) must display properly #29