Content Search Powered by Meilisearch

[qasimgulzar]

Abstract

The Open edX platform is enhancing its search functionality by implementing Meilisearch, a fast and memory-efficient search engine. This project aims to improve search performance, reduce resource usage, and provide a more developer-friendly experience.

Detailed Product Proposal

https://openedx.atlassian.net/wiki/spaces/OEPM/pages/4411031558/Proposal+Content+Search+Powered+by+Meilisearch

Context & Background (in brief, if a Product Proposal is linked above)

The Open edX platform currently implements many different forms of search. For example, users can search for course content, library content, forum posts, and more. Most of the search functionality in the core platform is powered by the Elasticsearch search engine (though other functionality developed by 2U, such as in edx-enterprise, is powered by Algolia).

Most uses of Elasticsearch in Open edX use edx-search which provides a partial abstraction over Elasticsearch. The edx-search library formerly used django-haystack as an abstraction layer across search engines, but "that was ripped out after the package was abandoned upstream and it became an obstacle to upgrades and efficiently utilizing Elasticsearch (the abstraction layer imposed significant limits)" (thanks to Jeremy Bowman for this context). Due to these changes, the current edx-search API is a mix of abstractions and direct usage of the Elasticsearch API, which makes it confusing and difficult to work with. In addition, each usage of edx-search has been implemented fairly differently. See State of edx-search for details (thanks to Andy Shultz).

Scope & Approach (in brief, if a Product Proposal is linked above)

The initiative to implement Meilisearch within the Open edX platform aims to enhance the search functionality by providing a more efficient, scalable, and user-friendly search experience. This includes the development of a new generalized search API capable of supporting multiple search engines, the integration of Meilisearch as a primary search engine, and the transition from the existing Elasticsearch implementation.

Value & Impact (in brief, if a Product Proposal is linked above)

Measurable Improvements to the Open edX Platform

Enhanced Search Performance:

Faster search response times will lead to a more efficient and satisfying user experience for course authors, instructors, and learners. Reduced memory usage will make the platform more cost-effective and capable of handling larger datasets without significant resource strain.

Improved User Experience:

A more intuitive and responsive search functionality will make it easier for users to find relevant content quickly, enhancing their overall experience on the platform. Direct search requests from users to Meilisearch, bypassing the need to route through Django, will reduce latency and improve real-time search performance.

Scalability and Flexibility:

The new generalized search API will provide the flexibility to integrate with multiple search engines, ensuring the platform remains adaptable to future technological advancements. Strong multi-tenancy support will enable the platform to scale more effectively across multiple small Open edX sites, improving resource allocation and performance.

Operational Efficiency:

Offloading search processing to Meilisearch will reduce the load on the LMS and other IDAs, improving the overall system performance and stability. Simplified maintenance and implementation processes will free up developer resources, allowing for more focus on enhancing other aspects of the platform.

Milestones and/or Epics

1. integrate openedx-search-api into edx-platform
2. Implemented LMS/CMS content classes to migrate index from elasticsearch to meilisearch
3. Update learning MFE to implement existing courseware search feature with openedx-search-api

Named Release

Sumac

Timeline (in brief, if a Product Proposal is linked above)

We are trying to make it part of Sumac release of openedx.

Proposed By

Edly

Additional Info

Under this PR we have implemented a POC

[github-actions[bot]]

Thanks for your submission, @openedx/openedx-product-managers will review shortly.

[qasimgulzar]

@jmakowski1123 is there any update regarding this proposal?

[jmakowski1123]

@jmakowski1123 is there any update regarding this proposal?

This thread contains the latest in terms of enabling via Tutor, and it would be helpful to channel any developer feedback that comes from that into the central forum thread. I'm also cc-ing @ormsbee who is going to be shepherding the implementation decisions for Sumac and Teak.

[ormsbee]

@qasimgulzar: I think that moving forward with openedx-search-api makes sense, though that would also involve bringing it into the openedx GitHub org. I haven't looked over the code since the first PRs you linked in discussions. As for Meilisearch as the backend–it's going to be the backend we'll use for at least the Sumac launch, but I want to schedule some time to evaluate TypeSense as well, to see if it could better address @blarghmatey's concerns w.r.t. High Availability, as he explained here.

[qasimgulzar]

@ormsbee I would be happy to assist you. do we have any actionable so far?

[qasimgulzar]

I can see the ticket is closed, but I believe there are still some pending actionable like moving openedx-search-api to openedx organisation as well as opening integration PR once repo will be move to openedx org.

[ormsbee]

Assigning this to myself for more followup.

[regisb]

I'm taking over this project, here's the corresponding PR: https://github.com/openedx/edx-platform/pull/35650