[MVP] More accurate search using semantic search engine

[xinli-cai]

As a user, we would like to have a search engine that is returning accurate and relevant search results.

[xinli-cai]

After ressearch, a migration from Keyword search to Semantic search can improve the search accuracy on geo.ca. Semantic search uses AI and Natural Language Processing (NLP) to understand the intent and context of the search query, and it will provide more accurate and relevant search results compared to the current search engine (i.e.,keyword search).

To build a prototype semantic search engine using pretrained models, the following steps are required

• [x] Build a lambda function to collect and process the text from metadata records
• [x] Embed the text data to vector (embedding)
• [x] Build a Vector Database using the pretrained semantic search models
• [x] Create the search index in Amazon OpenSearch and load embedding to the index
• [x] Deploy the search function with AWS services
• [x] Deploy semantic search engine with AWS OpenSearch
• [x] Continue improve the semantic search engine (e..g, finetune models, document index)

[xinli-cai]

Updates:

• A prototype of semantic search using Amazon OpenSearch with pretrained models was deployed in the DEV environment and demoed in March 2024. Links to the demonstration is in Demo2-Semantic-search-with-pretain-model.
• Semantic Search API is deployed in the DEV envrionment. An example search query for 'floods event in Canada': https://search-recherche.geocore-dev.api.geo.ca/search-opensearch?method=SemanticSearch&searchString=floods%20event%20in%20Canada.
• Three semantic search models are finetuned and evaluated, and all the evaluated models are deployed as a SageMaker model inference in Dev environment
• OpenSearch domain and vector index is deployed in DEV environment.