Migrate Azure AI Search Indexing from Blob Storage to PostgreSQL Database

[k-allagbe]

Transition Azure AI Search indexing from the current blob storage to directly index documents stored in our PostgreSQL database. This change aims to eliminate duplicate storage and streamline our search indexing process.

Tasks

• [ ] Establish a secure connection between Azure Cognitive Search and the PostgreSQL database.
• [ ] Create a data source in Azure Cognitive Search for the PostgreSQL database.
• [ ] Develop an indexer in Azure Cognitive Search to fetch data directly from PostgreSQL.
• [ ] Map PostgreSQL database fields to the Azure Cognitive Search index schema.
• [ ] Test the new setup thoroughly to ensure data integrity and search functionality.
• [ ] Remove or repurpose the existing blob storage setup as necessary.
• [ ] Update documentation to reflect the new indexing process.

Acceptance Criteria

• Azure AI Search indexes documents directly from the PostgreSQL database without relying on blob storage.
• The search functionality is fully operational with the new PostgreSQL data source.
• The system avoids duplicate storage of documents and operates efficiently.
• Data integrity and search accuracy are maintained post-migration.
• The new setup is well-documented.

[rngadam]

Can you provide the link to the documentation that explains how to customize datastore?

[k-allagbe]

Can you provide the link to the documentation that explains how to customize datastore?

https://learn.microsoft.com/en-us/rest/api/searchservice/preview-api/create-or-update-data-source

There is of course also the azure interface option.

[ibrahim-kabir]

@rngadam, is this issue still of interest since the EDAP wants to create a database for all the agencies ?

[rngadam]

@rngadam, is this issue still of interest since the EDAP wants to create a database for all the agencies ?

Yes, that part of their workplan is for the end of the year; we can't wait until they are ready. They will probably align with our choice of database so it will still be a Postgresql instance somewhere.