Support client requests via NATS messaging system

[AbstractiveNord]

Is your feature request related to a problem? Please describe. In case of use ManticoreSearch, especially in distributed mode, there is good alternative for classic HTTP and SQL interfaces for querying, NATS messaging system. NATS acts like a network layer of proxy servers clusters, support PUB/SUB and Request/Reply operations. It's good to have an option for use ManticoreSearch through NATS Request/Reply system. Being similar to HTTP, NATS Requests also provides modern cloud native networking options, like auto-discovery new instances of services, instant auto observability, loadbalancing, in-place documentation, etc.

Describe the solution you'd like Support for NATS Core (NATS also have streaming persistent engine, Jetstream, currently it's out of scope) as first class client API for ManticoreSearch.

Additional context While NATS system itself is incredibly flexible, especially with Jetstream persistent engine, implementing first class API support for NATS allows not only clients to operate with ManticoreSearch, but also allow to extend ManticoreSearch with modern cool features, such as:

1. Support for ingesting data from NATS Jetstream streams, which is highly similar for Apache Kafka topics,
2. Dynamic configuration for ManticoreSearch nodes with option to directly aim any node or group of them,
3. Asynchronous replication between synchronous clusters of ManticoreSearch.

NATS have youtube playlist with live demos of it's capabilities, which may be used. Also, I can assist with some examples and code snippets, if requested.

[nickchomey]

This would be amazing. I discovered NATS a couple weeks ago and immediately decided to make it the core (or, in their parlance, "central nervous system") of my distributed application. It's hard to believe how simple and powerful NATS is.

The last remaining aspect that I wasn't sure of was how I could use it with Manticore - I was thinking I'd need to set up a separate async server just to run the NATS and Manticore clients, and surely write custom code to parse it all.

And with a strong consistency/active-active architecture, it's a great match for Manticore.

[AbstractiveNord]

This would be amazing. I discovered NATS a couple weeks ago and immediately decided to make it the core (or, in their parlance, "central nervous system") of my distributed application. It's hard to believe how simple and powerful NATS is.

The last remaining aspect that I wasn't sure of was how I could use it with Manticore - I was thinking I'd need to set up a separate async server just to run the NATS and Manticore clients, and surely write custom code to parse it all.

And with a strong consistency/active-active architecture, it's a great match for Manticore.

I am also highly like NATS project. Asynchronous replication may be implemented via publishing parts of binlog into Jetstream stream, which may be consumed by asynchronous replicas.

[tomatolog]

@nickchomey could you provide insight of what application do you integrate with nats?

[AbstractiveNord]

Mentioned YT playlists:

1. CLI Tutorials
2. NATS Basics

[tomatolog]

youtube playlists one for couple hours and another one for an hour - no way

[AbstractiveNord]

youtube playlists one for couple hours and another one for an hour - no way

Oh, well, it's just a compilation. Welcome to ask anything directly there.

[AbstractiveNord]

I was thinking I'd need to set up a separate async server just to run the NATS and Manticore clients, and surely write custom code to parse it all.

Issue is all about making ManticoreSearch instances to have first class support for NATS Request/Reply. Custom application, in fact, bridge between NATS and ManticoreSearch, will be forced to be separate application, needs to be written, tested, and deployed, it's not baken inside ManticoreSearch, it will have latency penalty and performance may not be same as SQL/HTTP client APIs, written in C++. If ManticoreSearch team will support NATS as same as SQL and HTTP API, directly inside application, it will work like a charm.

[nickchomey]

I intend to use NATS to tie together manticore, a Python-based machine learning server, a php/go webserver (frankenphp), a wordpress application, browser-based pwa, cloudflare workers, and more.

It can be used for anything really. The nats protocol is really just an alternative to http, but nats itself is useful for making a dead-simple distributed, scalable event-driven system. Each service just publishes and subscribes to whatever subjects/streams/kv buckets that are relevant to it, without knowing anything about the other services (or even nodes of the same service type) in the system.

As for manticore specifically, it would be useful to be able to trigger things upon ingestion, particular searches, etc. I'd imagine that something with Buddy would be useful as well.

[AbstractiveNord]

As for manticore specifically, it would be useful to be able to trigger things upon ingestion, particular searches, etc. I'd imagine that something with Buddy would be useful as well.

What do you mean by triggers upon ingestion? I think it's enough complicated. For now, it's good to just have low level support for NATS in ManticoreSearch.

[nickchomey]

Yes, I agree. They asked me for ideas/use cases. I was brainstorming.

At the very least, it would be nice if manticore could subscribe to NATS subjects that are published to/by other parts of the system, rather than via http.

In fact, using NATS queue groups, NATS could automatically load balance requests to/between different Manticore nodes.