Lightweight Elasticsearch

[ahoogol]

Our team is developing and supporting a desktop application that can be installed on a wide range of systems from weak to strong and works offline (without internet or any network) in these systems. The application requires a search engine with all the features of Lucene with an independent language API such as elasticsearch. The search engine should be installed offline with the application in the end-user's system. Due to the fact that our team has experience in using elasticsearch, the best option to choose is elasticsearch, but elasticsearch is so heavy that it raises many challenges for publishing, installation and use on clients, from its large size for publishing, to the large resources it will occupy in the user's system. Our suggestion to the OpenSearch development team is that the OpenSearch be published and installed as a core with minimal features and minimum size using minimal system resources, especially RAM. That is, without features such as scalability and resilience (clusters, nodes, and shards) and security, with the smallest size of the bundled JVM (located within the jdk directory of the Elasticsearch home directory), with the ability for user to add the rest of the features to the core if needed. This will extend the use of OpenSearch to all applications across all operating systems, from desktop to mobile.

[forNoor]

This is a great suggestion OpenSearch can be a search engine for all desktop applications. I think that this suggestion should apply in OpenSearch architecture

[forNoor]

I looked at the latest version of Elastic. Its size in Windows is about 500 MB. The modules folder has a volume of 175 MB And a JDK folder with a volume of 260 This size is really large

[danesh110]

Although ElasticSearch can be used in some clients, it is actually a server-based search engine and is designed, optimized and tested for servers and not recommended to be used on the client-side. I think a client-based search engine is an essential need for many apps. If it was possible to have OpenSearch on the client-side with the same API as the server-based version, it would be very nice and useful. In this way, the apps can easily switch between a client-side and a server-side search engine without any need to change the code.

[aparo]

@ahoogol What are you suggesting cannot be done with ES/OS code. If you check on source code, you'll find that by design every action is a cluster/node one. It's not possible remove cluster/node code from ES/OS codebase. It's by-design distributed. My suggestion is to use a simpler implementation at top of Lucene to achieve what you need.

[ahoogol]

@aparo Thanks for your reply. You are right, at first glance, it is better to use another implementation at top of Lucene. But which implementation can easily bring these important advantages:

1. From a desktop application point of view, switching between ElasticSearch instances in client and server is as simple as changing a http url but in Lucene this is not possible to use it as server.
2. We can use Elasticsearch with powerful monitoring tool like Kibana.
3. Elasticsearch offers us, Powerful Query DSL a JSON interface for reading and writing queries on top of Lucene. Thanks to Elasticsearch, you can write complex queries without knowing Lucene syntax.
4. ElasticSearch is Schemaless (Schema-Free) that Fields(name,value pairs) for schema do not have to be defined before. When you index data, elasticsearch can create schema automatically at runtime, like magic.
5. ElasticSearch supports very comprehensive data aggregations some of which are not even supported by Lucene directly.

[dblock]

@ahoogol I see a parallel with the discussion between, say, SQLite and MySQL. Would you want the engine to be embedded into a desktop/client application (SQLite), or are you thinking it could be a pre-requisite to be installed (MySQL)?

[ahoogol]

@dblock If I understand what you said, it would be better if OpenSearch can be embeded in a client application.

[tomrade]

you could look at something like https://lunrjs.com/

[ahoogol]

@tomrade, your recommendation is a library and has javascript API. I want a service with independent language like JSON API. Also it has not many necessary features like aggregation.

[devardee]

I think couchDb or pouchDb should handle you're use case