Replace Elasticsearch with Vespa

[docuracy]

Pros and Cons of moving Elasticsearch into Docker Network

Moving Elasticsearch into the existing Docker network can offer significant benefits in terms of consistency, portability, and scalability, but it also introduces some complexity and potential performance considerations.

Pros:

• Isolation: Environment and resource isolation.
• Portability: Consistent deployment across different environments.
• Scalability: Easier to scale and manage with orchestration tools.
• Version Control: Ensures consistent versions and simplified upgrades.
• Maintenance: Simplified setup and automation of tasks.

Cons:

• Performance Overhead: Potential slight performance hit.
• Networking: Potential network overhead.

Assessment:

The possibility of a slight performance hit is far outweighed by the advantages of maintenance, portability, and scalability. It would be good to optimise the resources so that the index health might be green instead of yellow.

• [ ] Kibana (for index management) should be set up in a separate container on the same network.
• [ ] For persistence of data, be sure to use a bind mount of data stored in filesystem.
• [ ] Ensure that production container is accessible on an external port for access from development networks (configured in settings based on value of ENV).
• [ ] Add service healthcheck to existing Celery Beat task (see #331).
• [ ] Set up incremental snapshot backup to GCS as outlined in #320.
• [ ] It would be desirable to have a management command for rebuilding indexes afresh from the database.

[docuracy]

"Vespa excels at hybrid search, where you can combine traditional keyword-based search with vector-based similarity search. This is especially useful when you want to rank results based on both textual relevance and semantic similarity."

Vespa might be the best option for all indexing requirements, not least because it has built-in vector searching (mBERT/phonetic) and an easily-deployed Docker image.

Compile a list of current and planned index/search requirements.