Covers a variety of improvements to free text search functionality.
Closes #267
Changes to functionality
Expanded the number of fields that are indexed and searchable across various entities
Metadata from a search comes directly from Lucene, rather than returning a list of ids that then require a subsequent DB call.
In addition to performing authorisation on the main returned entity (Investigation/Dataset/Datafile), we also use the public steps/tables to safely return nested information (such as a DatafileParameter without additional time spent authorising)
Support for faceting of results, sorting, searching after, flexible batches of search results
Support for other search engines, specifically Opensearch/Elasticsearch
Note that a branch without the Opensearch code but all the other features exists
Changes to code
[ ] Various entities have had getDoc either added or expanded, alongside the addition of getDocumentFields which enforces that the search component does not return metadata that isn't allowed by the ICAT public steps/tables
[ ] Modified existing search functions in ICATRest and added new endpoints search/documents and facet/documents
[ ] Changes to EntityBeanManager to support new search calls
[ ] Small additions to GateKeeper to allow the publicly allowed search fields to be marked as stale and updated as public steps/tables are updated
[ ] Changes to PropertyHandler to support different SearchEngines as config options as well as other new config parameters
[ ] LuceneManager replaced with generic SearchManager which interacts with the search engine via an instance of SearchApi
[ ] Creation of SearchApi, and abstract class for interfacing with search engine implementations
[ ] LuceneApi modified extensively, and now extends SearchApi
[ ] Creation of OpensearchApi which extends SearchApi and supports either Opensearch or Elasticsearch clusters by building generic JSON. Also has some static functionality refactored into supporting classes:
[ ] OpensearchQueryBuilder
[ ] OpensearchScriptBuilder
[ ] Creation of FacetDimension and FacetLabel to represent facets of results, and replacing LuceneSearchResult with SearchResult which has facet support
[ ] Various small changes to rename variables from luceneAbc to searchAbc to reflect what is or isn't specific to the icat.lucene search engine
@patrick-austin I have finished reviewing this PR so feel free to start addressing and pushing changes. Let me know when you are done and I will review your changes.
Covers a variety of improvements to free text search functionality. Closes #267
Changes to functionality
Investigation
/Dataset
/Datafile
), we also use the public steps/tables to safely return nested information (such as aDatafileParameter
without additional time spent authorising)Changes to code
getDoc
either added or expanded, alongside the addition ofgetDocumentFields
which enforces that the search component does not return metadata that isn't allowed by the ICAT public steps/tablesICATRest
and added new endpointssearch/documents
andfacet/documents
EntityBeanManager
to support new search callsGateKeeper
to allow the publicly allowed search fields to be marked as stale and updated as public steps/tables are updatedPropertyHandler
to support differentSearchEngine
s as config options as well as other new config parametersLuceneManager
replaced with genericSearchManager
which interacts with the search engine via an instance ofSearchApi
SearchApi
, and abstract class for interfacing with search engine implementationsLuceneApi
modified extensively, and now extendsSearchApi
OpensearchApi
which extendsSearchApi
and supports either Opensearch or Elasticsearch clusters by building generic JSON. Also has some static functionality refactored into supporting classes:OpensearchQueryBuilder
OpensearchScriptBuilder
FacetDimension
andFacetLabel
to represent facets of results, and replacingLuceneSearchResult
withSearchResult
which has facet supportluceneAbc
tosearchAbc
to reflect what is or isn't specific to theicat.lucene
search engineInterdependencies on other components