Alphabetical ordering in results list incorrect

[cessda-bitbucket-importer]

Original report on BitBucket by Taina Jääskeläinen.

---

It seems that the alphabetical ordering in the results list is now incorrect in most languages.

• If there is no search term entered, the system should show the vocabularies in alphabetical ordering according to the vocabulary long name in the selected language.

For instance, Slovenian is otherwise in alphabetical order but Časovna metoda (TimeMethod) comes las in the production version. It should not.

‌

The list in English in the Editor in staging contains vocabularies

• not available in English (on page 2 of the results list) so should not be in the ‘English’ list
• they are not in alphabetical order but neither are the rest of the vocabularies in English

This is not a design issue so perhaps wait till the versioning and workflow issues for different roles have been resolved.

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

it looks like the sort can't deal with the used character encoding

[cessda-bitbucket-importer]

Original comment by Taina Jääskeläinen.

---

Using UTF-8 might solve this, what do you think?

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

my temporal notes:

• https://unicode-org.github.io/icu/userguide/collation/
• https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html
• https://www.elastic.co/guide/en/elasticsearch/plugins/6.8/analysis-icu-collation-keyword-field.html

# get field mapping - https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-get-field-mapping.html
GET vocabularypublish/_mapping/field/titleSl

# put field mapping (in general, the mapping for existing fields cannot be updated) - https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-put-mapping.html
PUT vocabularypublish
{
  "mappings": {
    "properties": {
      "name": {   
        "type": "text",
        "fields": {
          "sort_titleSl_Key": {  
            "type": "icu_collation_keyword",
            "index": false,
            "language": "sl",
            "country": "SL",
            "variant": "@‌collation=phonebook"
          }
        }
      }
    }
  }
}

‌

‌

1. before deploying the new version, which fixes this issue, either:

   1. delete indexes in running elasticsearch service

      • the mapping for existing fields cannot be updated (see https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-put-mapping.html)
      • curl -X DELETE http://cvs-elasticsearch:9203/vocabularypublish,vocabularyeditor

   2. stop cvs-elasticsearch service, remove it, delete its docker volume cvs-es and start the service again using the updated docker compose yaml (app.yml + elasticsearch.yml)

2. reindex indexes in CVS → Admin → Maintenance →

   • Indexing Vocabulary Publish → Index Published CVs
   • Indexing Vocabulary Editor → Indexing

‌

[cessda-bitbucket-importer]

Original comment by Taina Jääskeläinen.

---

I forgot to tell you that if the user has chosen (or left) the language as English, the vocabularies that are only in Finnish or German (e.g. in the Editor) should not be in the English list at all. They should only appear if the user has chosen ‘All languages’ which is at the bottom of the list.

If ‘All languages’ is not chosen, the alphabetical ordering is by the vocabulary long name in the specific language and follows the alphabetical order of that language. But for ‘all languages’ your notes are more than relevant.

‌

Specifying the problem in the staging (both in Home and Editor list)

• The English list contains vocabularies that are not in English (e.g. _KorkeakoulujenTieteenalaluokitus or Testi_87). Using short names for convenience here even though the alphabetical ordering is by long name.
• Even the ones that are in English only (e.g. test1, test_cv3) are not in alphabetical order.
• I wonder if the ordering has changed at some specific point of time as most are still in correct order.

• The German or Finnish lists do not contain any vocabularies that are not available in those languages. They are mostly in alphabetical order but not totally

  • The last one in German is in the wrong place
  • 'TL Kloonaustesti (long name) in Finnish is in wrong place, it should be after Tilastollisen tunnusluvun tyyppi and not after Samankaltaisuustyyppi.

So there are two issues: vocabularies in the wrong language list and vocabularies not in alphabetical order.

Please let me know if you need further clarification.

We made the decision to put ‘All languages’ at the bottom of the list because we noticed that the all languages search did not in many cases find the terms or definitions. It found them only when the search terms used were in the exactly same format as in the text (so if search term in singular, did not find plural or other forms). Language analyzers do not seem to work well across languages.

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

Update:

• I have fixed the issue with alphabetical ordering of vocabularies when a particular language is selected. It uses an ICU analysis plugin with collation support for different languages.

  • However, ordering when “All languages” is selected is still open.

    • This is actually a little bit tricky since each language has a separate title field in elasticsearch and the ordering is performed over a specific field, which is by default EN for “All languages”. If a vocabulary does not have a title defined for EN, it can’t be properly sorted. Maybe a universal field for sorting the titles can be added to the index. I’ll look at it.

• I can’t reproduce vocabularies in the wrong language list

  • tried to create a Slovak SL vocabulary and it was not put in other than Slovak vocabulary list

‌

[cessda-bitbucket-importer]

Original comment by Taina Jääskeläinen.

---

I think I have found one reason for alphabetical ordering not being right in the English list:

• In the production version, there is the vocabulary KategorienschemaWahlstudien. It has an English and German title.
• In the English list, the vocabulary is in the right place alphabetically according to its English title Elective studies category schema.
• However, the system is displaying the German version of it in the English list, therefore it has the German title Kategorienschema Wahlstudien so seems to be in a wrong place alphabetically.
• I think this is because the SL in this vocabulary is German and not English, the system is not properly noting that it should display the English version in the English list.

‌

Currently, the Editor in staging has these vocabularies on page 2 in the English list. None of them have an English version.

‌

‌

Would there be any defaults that would lead to them being included in the English list?

I think all the problems with vocabularies appearing in a language list when there is no version in that language are in the English list.

‌

Therefore, these two English list problem would need solving:

1. Ensure that in the English list, even if the SL is something else, the EN version is always displayed (if available).
2. If there is no English version available, the vocabulary should not be in the English list at all.

Then we would see what potential sorting issue remain after this.

‌

In ‘All languages’ list I see three options:

1. Sorting is done by SL title. This requires that some kind of sorting across languages can be achieved as SLs are not always English.
2. We use the short name for sorting as it is the same for all languages, instead of the language-specific long name. The long name displayed is the SL one. This will look a bit funny to users as the long name comes first in view but maybe they would eventually realize the sorting is by short name.
3. We drop the ‘All languages’ selection for now from view and I’ll make an issue for some later consideration.

‌

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

• All languages

  • I’ve added a new hidden field titleAll in vocabularies for correct sorting when “All languages” is selected. This field is filled with a SL title (e.g., titleAll=titleFi if sourceLanguage is Finnish) and when “All languages” is selected, it is used to sort the results using language neutral DUCET collation.

• Search results are sorted by selected language as well as displayed in that particular language in the result list

• Only versions with a selected language are displayed in the result list

This fix requires to reindex the indexes:

CVS → Admin → Maintenance →

• Indexing Vocabulary Publish → Index Published CVs
• Indexing Vocabulary Editor → Indexing

‌

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

will be ready to test in dev and staging after the new version is deployed

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

there are some problems after the deployment; i.e., "Internal server error" on "All languages”

@Joshocan Q: When a deployment (dev+staging) is made and there was an update of the elasticsearch's container command in elasticsearch.yml, is the elasticsearch (ES) also redeployed? If not, we need to redeploy all the ES shards in order to make them install collation plugin for correct alphabetical sorting, which is used by this fix. An update: It looks like there’s a different elasticsearch.yml used in master deployment than it is in the src/main/docker, because there’s an installation of repository-gcs at the docker image build, while this thing is missing in the elasticsearch.yml of master branch.

[cessda-bitbucket-importer]

Original comment by Joshua Tetteh Ocansey (GitHub: Joshocan).

---

@Stifo Issue #447 ( Analysis-icu plugin for ES installed). Please test on dev or staging and confirm if the issue is resolved.

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

Made a comment in #447, additional step is required before testing.

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

@‌TainaFSD you can now test this issue in both dev and staging

[cessda-bitbucket-importer]

Original comment by Stefan Dlugolinsky (GitHub: Stifo).

---

icu analysis plugin was installed in elasticsearch for both dev and staging in #447

[cessda-bitbucket-importer]

Original comment by Taina Jääskeläinen.

---

Tested in staging both in Chrome and Firefox. The results are good:

• English list: Problems resolved The system now shows only vocabularies that have English content. Alphabetical ordering is correct by English title (long name).
• All languages list: Problems resolved. Alphabetical ordering is correct. It is ordered by SL title, regardless of SL language.

‌