update elasticsearch version

[zaziemo]

to get rid of Hashie Mash warning and keep up with new development

[tyranja]

https://github.com/rubymonsters/speakerinnen_liste/pull/806

[zaziemo]

• [x] update gem elasticsearch_rails #806
• [x] https://www.elastic.co/products/upgrade_guide
• [x] update gem 'elasticsearch-model' (https://github.com/elastic/elasticsearch-rails/tree/master/elasticsearch-model)
• [x] https://www.elastic.co/guide/en/elasticsearch/reference/5.0/setup-upgrade.html
• [x] full cluster restart

[zaziemo]

Some links: https://logz.io/blog/upgrade-elasticsearch-5/

"Download the latest version package Install Java 8 Stop es instance Install the new package Put the new configuration Remove logging.yml (if you have) Remove unused site plugin Monitor the log Release the beast" https://www.linkedin.com/pulse/upgrading-elasticsearch-2x-5x-gian-giovani

Breaking changes: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking-changes-5.0.html

[zaziemo]

Maybe this migration helper plugin can help as well: https://github.com/elastic/elasticsearch-migration/blob/master/README.asciidoc

[zaziemo]

https://mendab1e.github.io/dev/2019/03/09/how_not_to_integrate_elasticsearch_testing_with_rspec.html

[zaziemo]

https://gist.github.com/zaziemo/1c8f6ed25b17b4200f1f53068e79f905

[zaziemo]

• [x] really understand what has changed from version 2 to 5 (and further on), e.g.https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-suggesters-completion.html
• [x] understand how these changes are covered in elasticsearch rails gem
• [x] make spec pass: ./spec/system/searching_for_profiles_spec.rb:48
• [ ] make spec pass: ./spec/controllers/admin/category_controller_spec.rb:124 --> skipped temporarily (has nothing to do with elasticsearch)
• [ ] make spec pass: rspec ./spec/controllers/profiles_controller_spec.rb:38
• [ ] make spec pass: ./spec/models/concerns/searchable_spec.rb:85
• [x] make autosuggestions work again
• [x] check if the same (amount of) search results are found

[zaziemo]

Already did some work on this branch: https://github.com/rubymonsters/speakerinnen_liste/compare/mah-upgrade-elasticsearch

[netagonen]

Steps towards upgrade elastic search locally:

1. Have production data on local machine
2. Re-Index elasticsearch with new data
3. Make sure website on localhost still works
4. Upgrade elasticsearch server and API in the docker container - look at @zaziemo 's branch first: (links which might be useful for this step that @zaziemo referred to: https://www.elastic.co/upgrade_guide https://logz.io/blog/upgrade-elasticsearch-5/ https://www.linkedin.com/pulse/upgrading-elasticsearch-2x-5x-gian-giovani/ https://github.com/elastic/elasticsearch-migration/tree/2.x
5. Upgrade elasticsearsch rail gems - look at @zaziemo 's branch first
6. Test search on website (i.e number of results, autosuggestions) ->
7. Look at changes at elasticsearch API and gems API (links that might be usefull: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking-changes-5.0.html https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-suggesters-completion.html)
8. Fix search specs (look at @zaziemo list above). (link that might be useful: https://mendab1e.github.io/dev/2019/03/09/how_not_to_integrate_elasticsearch_testing_with_rspec.html)
9. Upgrade elasticsearch server in production env (probably follow this guide: https://www.elastic.co/upgrade_guide)

[netagonen]

More links, might be helpful https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_suggester.html https://www.elastic.co/guide/en/elasticsearch/reference/5.0/search-suggesters-completion.html https://www.elastic.co/blog/strings-are-dead-long-live-strings https://asquera.de/blog/2018-01-24/elasticsearch-speakerinnen/ http://elasticsearch-cheatsheet.jolicode.com/ https://medium.com/@mourjo_sen/a-detailed-comparison-between-autocompletion-strategies-in-elasticsearch-66cb9e9c62c4 https://hackernoon.com/elasticsearch-using-completion-suggester-to-build-autocomplete-e9c120cf6d87 http://www.kadrmasconcepts.com/blog/2015/04/25/typeahead-js-elasticsearch-and-rails/

[langjoh]

@netagonen Hi Neta, about step 4 (Upgrade elasticsearch server and API in the docker container): I think all you have to do is to specify a later version of the elasticsearch image in the docker-compose.yml. At the moment it is image: elasticsearch:2.4.5 . Here you can find all available image versions: go to https://hub.docker.com/_/elasticsearch and then search under 'tags' for your preferred version.

[netagonen]

hi @zaziemo , see also PR #1091 . There is still more work to do. here is all what is done and what is left to do: First of all it was very helpful to read this post in order to understand the searchable.rb file

Upgrade:

1. go to branch neta/upgrade-elastis-6.7 and pull
2. run make setup make sure bundle install is done successfully
3. run make dev, in the console run rake elasticsearch:import:all
4. run make up or docker-compose up to see also logs
5. go to local host:3000
   • website should works
   • search should work

Problems we encountered and how we handled them:

• [x] aggregations are not working: made sure aggregations are indexed in searchable.rb correctly, then made sure they are called correctly in the profiles_controller.rb.
• [x] auto-complete is not working: syntax for that changed in the new elasticsearch version. we managed to make autocomplete work. it is not working exactly like production but that is necessarily a bad thing:
• there are no duplicates
• there are sometimes more options that seem relevant than on production
• there are sometimes options that doesn't show but show on production
• the order is different.
  We need to decide if we are fine with all the differences
• [x] we get different number of search results for the same key word on local and on production: one direction could be to compare scores between local and production.
• we got responses for a query in irb console (for production I used horuku CLI) $ profiles = Profile.is_published.includes(:taggings, :translations).search("SEARCH_TERM", "", "", "").records $ profiles.response.results.as_json or profiles.response.response
  more methods in the documentation I uploaded some search results from local and production to nextcloud so we could compare.
• we compared the scoring between the two environment for the same profile and for the same search term were very different, i.e: "_score":10.668072 and "_score":0.37613404.
• We found that there were breaking changes in scoring, such as they removed some multiplier that made scores smaller and vary around 1.0. So on production, where version 2.4.5 is running, this line make less results apear query_hash[:min_score] = 0.08 if Rails.env.production? and on local, with the new scoring system (and after removing the environment condition), min_score did not have any influence since all scores are above 0.08 anyways, so there where much more results.
• The question is: do we need the min_score threashold for some reason? why do we want to filter out those results? are we sure they are not relevant? we can test this on the latest data we have. For now this line is removed. But if we want it, then we need to set the min_score on prodution higher. 3.00 might be a good one, we need to test.
• [x] check out weight of bio field. problems:
  1. when a speaker has a de_bio and en_bio she get higher score than a speaker with bio only in one language, even though the search term has the same frequency. The solution to this is to remove the tie_breaker, because it make the score of someone with bio in both languages a higher score then for someone with bio in one language. see more here
  2. the score for a match of a search term in the bio field is relative, so the longer the text, the score is lower. English texts tend to be shorter than German ones, and then the score of the bio field in English could be higher. -> w e decided to leave it as is, since the weight of this field id anyway low.
• [x] Fix search spec: Fixed test in searchable_spec.rb:98.
• [x] Make elastic run on non linux machine and add wait_for_elastic script: when pushing to github, elastic did not start in travis, it also didn't start on Alon's machine, so we added this: environment: - discovery.type=single-node to solve the problem (more info here) and also added the script to check if elastic starts before tests are running.
• [x] in the docker-compose.yml we commented out volumes: - elastic_data:/usr/share/elasticsearch/data2 so version 6 could work. fixed by deleting the old volume on local machine with docker volume rm
• [x] suggest: in the query_hash we call for a suggest: option, so we would have suggestions for 0 matches (searchable.rb:57) but we don't use it in the controller. It was decided to delete it for now. we should deal with a case of 0 matches in another scope.

Questions and tasks left:

• [ ] min_score: do we need min_score on production? if so, how much? I think it would be good to manually test without the min_score and see if the last profiles are relevant or not.
• [x] autosuggestions: please test that works according to what we want.
• [x] Find a solution for 0 matches, so when @profiles after a search is empty.
• [x] cities: should it be keyword or unmod? unmod stands for umodified, so the cities are not analyzed when indexed. I am not sure this is still the case. would be good to check if it's better to change to keyword or if it does not matter.
• [ ] introduce HashWrapper on Hashie::Mash to avoid printing these errors? You are setting a key that conflicts with a built-in method Hashie::Mash#key defined in Hash. This can cause unexpected behavior when accessing the key as a property. You can still access the key via the #[] method.
• [x] we tried to upgrade already to version 7.9.1 of elasticsearch but did not succeed. we can try to do it at the end would be nice to have.

[zaziemo]

@netagonen @alonpeer concerning min_score

I think we need one - or probably some more tweaking with the tie_breaker or field weighs. Can you explain how the profiles in the search results are ordered? Is it after the scores?

I made tests with the following search terms: 1) for „Social Business“, being in :en

Localhost: 1.108 speakers examples for non matching profiles (for my understanding): http://localhost:3000/en/profiles/nina-mohimi http://localhost:3000/en/profiles/arlene-buehler http://localhost:3000/en/profiles/alexandra-grassler

Production: 388 speakers (but there are also some profiles I would not consider matching to the search terms)

2) „programmieren frauen“, being in :en

localhost: 541 examples for non matching profiles (for my understanding):: http://localhost:3000/en/profiles/anja-blodow http://localhost:3000/en/profiles/regula-simon http://localhost:3000/en/profiles/annika-peters

not really matching but cool to list it: http://localhost:3000/en/profiles/janina-tiedemann --> which is an argument for being not so strict with the matching terms

production: 159

[zaziemo]

@netagonen @alonpeer Concerning autosuggestions: Here I don't get the same autosuggestions as on production which is good, because currently the suggester seems to be buggy if not broken :(

I would say it works as expected.

The suggested terms seem to be sorted by a score. Do you know what this score is about?

[zaziemo]

Solution for 0 Matches --> I opened a new issue (#1098). I think this is out of the scope for this one here.

[zaziemo]

@netagonen concerning cities: I think it is good to have the cities not analyzed. I remember faintly that we had problems with people having a city name as their real name. Besides as the cities serve as filters they are already covered.

[zaziemo]

@netagonen concerning the HashWrapper. Do I get it right that we have to add it somehow? If we have to, we should do it. This was one of the reasons we wanted to update elasticsearch in the first place (we were annoyed by these warnings.)

[alonpeer]

Hi @zaziemo. I just checked, and the Hashie::Mash#key warnings appear on my localhost when I'm on the master branch, but no longer appear on the neta/upgrade-elastic-6.7 branch. 🎉

[zaziemo]

The country filtering does not work anymore.

To reproduce: 1) search for anything 2) click on one country in the most left column You get this view

--> I found the problem and will fix it. The country parameter has to be downcased http://localhost:3000/de/topics?filter_countries=FR&search=frauen+programmieren