Test diacritics and other non-Basic-Latin character search for all services

[maxkadel]

What maintenance needs to be done?

• We have found issues in some of the services for searches on diacritics. We should ensure that
  • We do not have a regression on this front
  • All existing services handle these characters gracefully
  • Any new services handle these characters gracefully

Level of urgency

• [x] High
• [ ] Moderate
• [ ] Low

Why is this maintenance needed?

• We need to ensure that we have good search access for all languages and character sets for equity

Acceptance criteria

• [x] There are consistent tests across services for non-Basic-Latin characters
• [x] These tests pass for all services, or further tickets are created for specific services and characters

  Stretch goal

• [ ] Newly created services include these tests

Implementation notes, if any

• Run schemathesis against staging - see instructions in PR
• Look for existing tests for non-Basic-Latin characters and and create shared examples for these characters and include them in the appropriate service tests (lib-jobs has some examples for shared examples in spec/support/sftp_shared_examples.rb

[maxkadel]

Examples from @tventimi:

https://allsearch.princeton.edu/?q=K%C5%8Dbuns%C5%8D%20Taika%20Koshomoku (works) vs https://allsearch.princeton.edu/?q=Kobunso+Taika+Koshomoku (doesn't work) vs https://allsearch.princeton.edu/?q=Ko%CC%84bunso%CC%84+taika+koshomoku (doesn't work) also https://allsearch.princeton.edu/?q=Chos%C5%8Fn+wangjo+sillok (works) vs https://allsearch.princeton.edu/?q=Choson+wangjo+sillok (doesn't work) vs https://allsearch.princeton.edu/?q=Choso%CC%86n+wangjo+sillok (doesn't work)

The DB entries themselves use the precomposed form of the accented characters. It appears that the matching fails when the search string uses the decomposed form or no diacritics at all. Blacklight is able to do this kind of normalization, which is why these searches have hits in the catalog but not the A-Z list. (edited)

[kevinreiss]

@sandbergja and I were talking a bit about this issue and Jane referenced this commit in OL that enabled matching to work for the precomposed, decomposed, and non-accented forms for the Browse lists: https://github.com/pulibrary/orangelight/pull/3894/files. We could potentially apply this to our local indexes in the allsearch to facilitate searching all forms.