Feature: Enhance SOLR integration and add a Schema API

syphax-bouazzouni commented 8 months ago

Prerequisites

53
SOLR >= 8.11 and started in mode cloud

Goals

Make it easier to configure SOLR programmatically.
Have the possibility to index any model on demand

Context

SOLR is the indexing tool, that we use for our search features, it works by defining a collection (a table in the databases world), and for each collection, a schema defines the properties to index by giving its type, a list, or not, ... and also some dynamic or special fields to handle fuzzy search, or other.

The requirement was that we were required to define the collection and the schema, in XML configuration files at the start, and then after we could not change it in the code. This meant we were limited in the action that we could do, and it was hard to add new search features to our system. as these files were static, and you had to update the schema and create the collection configuration files each time you wanted to add something into the index.

This PR, integration the SOLR Schema API, in this project, gives us the option to create/delete a collection and update a collection schema dynamically, the following actions were implemented (see the full list in the SOLR::Admin, SOLR::Schema and SOLR::SchemaGenerator modules in the code):

SOLR Administration API

fetch_all_collections 

create_collection(name = @collection_name, num_shards = 1, replication_factor = 1)

delete_collection(collection_name = @collection_name)

collection_exists?(collection_name)

SOLR Schema API

add_field(name, type, indexed: true, stored: true, multi_valued: false)

add_dynamic_field(name, type, indexed: true, stored: true, multi_valued: false)

add_copy_field(source, dest)

add_field_type(type_definition)

delete_field(name)

update_schema(schema_json)

fetch_schema

In Addition to the implementation of the SOLR Schema and admin APIs, we added a dsl to the Goo model, to enable index for any model, either in a schemaless mode or in a custom schema model

Schemaless mode: This will generate a collection and schema using the settings/metadata of the model properties, it is uses SOLR dynamic fields to make this possible, below an example of how to do it.

class TermSearch < Goo::Base::Resource
model :term_search, name_with: :prefLabel
attribute :prefLabel, enforce: [:existence], fuzzy_search: true # fuzzy search will permit to have autocomplete for this field 
attribute :synonym, enforce: [:list]
attribute :definition
attribute :submissionAcronym, enforce: [:existence]
attribute :submissionId, enforce: [:existence, :integer]
attribute :private, enforce: [:boolean], default: false, index: false # this field not be indexed 
attribute :semanticType
attribute :cui

enable_indexing(:my_collection) # will generate the collection called `my_collection` and index all the attributes of the model (except `:private` here)
end

Custom Schema mode: This we generate a collection and a schema directly defined in the code and decelartion of the model, below an example

class TermSearch < Goo::Base::Resource
model :term_search, name_with: :id
attribute :prefLabel, enforce: [:existence]
attribute :synonym, enforce: [:list] # array of strings
attribute :definition  # array of strings
attribute :submissionAcronym, enforce: [:existence]
attribute :submissionId, enforce: [:existence, :integer]
attribute :semanticType
attribute :cui

enable_indexing(:term_search) do
  schema_generator.add_field(:prefLabel, 'text_general', indexed: true, stored: true, multi_valued: false)
  schema_generator.add_field(:synonym, 'text_general', indexed: true, stored: true, multi_valued: true)
  schema_generator.add_field(:definition, 'string', indexed: true, stored: true, multi_valued: true)
  schema_generator.add_field(:submissionAcronym, 'string', indexed: true, stored: true, multi_valued: false)
  schema_generator.add_field(:submissionId, 'pint', indexed: true, stored: true, multi_valued: false)
  schema_generator.add_field(:cui, 'text_general', indexed: true, stored: true, multi_valued: true)
  schema_generator.add_field(:semanticType, 'text_general', indexed: true, stored: true, multi_valued: true)

  # Copy fields for fuzzy and autocomplete search
  schema_generator.add_copy_field('prefLabel', '_text_')
  schema_generator.add_copy_field('prefLabel', 'prefLabel_Exact')
  schema_generator.add_copy_field('prefLabel', 'prefLabel_Suggest')
  schema_generator.add_copy_field('prefLabel', 'prefLabel_SuggestEdge')
  schema_generator.add_copy_field('prefLabel', 'prefLabel_SuggestNgram')

  schema_generator.add_copy_field('synonym', '_text_')
  schema_generator.add_copy_field('synonym', 'synonym_Exact')
  schema_generator.add_copy_field('synonym', 'synonym_Suggest')
  schema_generator.add_copy_field('synonym', 'synonym_SuggestEdge')
  schema_generator.add_copy_field('synonym', 'synonym_SuggestNgram')

  schema_generator.add_copy_field('notation', '_text_')
end
end

Changes

add an abstraction to SOLR integration and add Schema API(https://github.com/ontoportal-lirmm/goo/commit/b16ffbd48e41915b4051f0dc796eba69531d7c37)
add SOLR Schema API tests(https://github.com/ontoportal-lirmm/goo/commit/03899436464e3be29e1e3bd0e39703592828216e)
update SOLR backend configuration and init(https://github.com/ontoportal-lirmm/goo/commit/f3815c47dde33d6fda778d6403fb6528e71e6218)
use the new Solr connector in the model search interface(https://github.com/ontoportal-lirmm/goo/commit/9ca2e1c8f192471ab1d3f66ebe6f9f6fc7bf7057)
update search test to cover the new automatic indexing and unindexing (https://github.com/ontoportal-lirmm/goo/commit/459c4ff9491bf7c785dcd7d9f8959f22be379c44)

codecov[bot] commented 8 months ago

Codecov Report

Attention: Patch coverage is 81.01604% with 71 lines in your changes are missing coverage. Please review.

Project coverage is 86.36%. Comparing base (1be1c83) to head (84aa3db).

Files	Patch %	Lines
lib/goo/search/solr/solr_schema.rb	72.34%	26 Missing :warning:
lib/goo/search/search.rb	82.79%	16 Missing :warning:
lib/goo/search/solr/solr_admin.rb	69.76%	13 Missing :warning:
lib/goo/search/solr/solr_query.rb	87.50%	6 Missing :warning:
lib/goo/search/solr/solr_schema_generator.rb	85.36%	6 Missing :warning:
lib/goo.rb	92.30%	2 Missing :warning:
lib/goo/search/solr/solr_connector.rb	91.66%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## development #54 +/- ## =============================================== + Coverage 85.93% 86.36% +0.43% =============================================== Files 41 46 +5 Lines 2702 3022 +320 =============================================== + Hits 2322 2610 +288 - Misses 380 412 +32 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mdorf commented 8 months ago

@syphax-bouazzouni, this looks great! Really useful feature for us as well! It would be great to test it against the latest Solr 9.5.0 to make sure the code is compatible. I am wrapping up my work on a few search enhancements for the RADx project, which we plan to deploy to production shortly. The next major task is upgrading Solr to the latest version, which we may very well coincide with merging this pull request. Are you planning to make any other significant changes to this feature (you mentioned that this is a "first iteration)? Thank you!

syphax-bouazzouni commented 8 months ago

9.5.0

Hello, @mdorf, I tested it locally with Solr 9.5.0 and all the tests are green. I removed the "first iteration" term, as I implemented everything I had in mind for now: administrate SOLR in code, be backward compatible with the existent, and index models automatically at save/delete.

mdorf commented 6 months ago

@syphax-bouazzouni, I am working on merging this functionality into our develop branch. It's a bit tricky given that the pull request is not against our own repo. Probably will need to do a lot of manual merging. Were you planning on submitting this pull request against the ncbo repo? If so, should I wait for that or just proceed with my manual merging? Thank you!

jonquet commented 6 months ago

Hello, the proposition was to move and create PRs directly on the OntoPortal repo now that everyone is positioned under it. I think @syphax-bouazzouni has scheduled some time to create PR related to our work soon.

syphax-bouazzouni commented 6 months ago

@syphax-bouazzouni, I am working on merging this functionality into our develop branch. It's a bit tricky given that the pull request is not against our own repo. Probably will need to do a lot of manual merging. Were you planning on submitting this pull request against the ncbo repo? If so, should I wait for that or just proceed with my manual merging? Thank you!

Hello @mdorf, I would suggest waiting at least 1 month before merging this, as it is only tested in our development environment, and will be released in our next release, see https://github.com/ontoportal-lirmm/ontologies_api/pull/73

Once deployed to our production environment and tested, I will do a PR.

Is it good with you?

mdorf commented 6 months ago

@syphax-bouazzouni, @jonquet, no problem. In general, we are very interested in this feature to facilitate the functionality sought by the RADx project, in which our Solr index would be packaged in a way to be accessible by the third-party API. See https://github.com/bmir-radx/radx-project/issues/49. However, based on my conversation with @alexskr, this does not have the immediate urgency. A month is definitely reasonable for us to wait to be able to merge this feature in its more stable and tested iteration.

ontoportal-lirmm / goo