search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.62k stars 1.76k forks source link

Add support for unix sockets #5531

Open mikygee opened 1 year ago

mikygee commented 1 year ago

Hello, I would like to know if opensearch supports unix sockets. I might want to use unix sockets along with syslog-ng. Thank you

dblock commented 1 year ago

~Do you mean using elasticsearch-http() destination of syslog-ng? I'll move this to opensearch-clients repo, and you might just want to give it a try and report back if something doesn't work? I found https://www.syslog-ng.com/community/b/blog/posts/elasticsearch-8-and-syslog-ng that talked about doing it with ES8, but I don't see why it wouldn't also work with OpenSearch.~

Are you looking for https://www.syslog-ng.com/community/b/blog/posts/opensearch-and-syslog-ng or something more specific? Reopen if that doesn't work.

mikygee commented 1 year ago

Euhhh, where on that page do they talk about unix sockets ?

dtaivpp commented 1 year ago

@mikygee do you mean Unix sockets for ingesting data into OpenSearch? I am not sure I understand the ask ask either.

mikygee commented 1 year ago

Hello, Yes exactly. You can either communicate with an application using tcp/ip or unix sockets or pipes... Here I'm asking about unix sockets https://serverfault.com/questions/124517/what-is-the-difference-between-unix-sockets-and-tcp-ip-sockets Thank you

dblock commented 1 year ago

I understand now. I will move this issue back to OpenSearch.

So the short answer is no, not today. OpenSearch doesn't support unix sockets (inter-process), and unless someone can convince us otherwise, or supply an implementation, does not plan to.

OpenSearch inherited from Elasticsearch, and doesn't support unix sockets because that's really geared towards a single-node deployment (same box) as far as I understand. I found some earlier discussions starting with https://github.com/elastic/elasticsearch/issues/35283 and https://github.com/elastic/kibana/issues/1652.

@mikygee What is your scenario? Curious why you're asking for this?

mikygee commented 1 year ago

Hello, I read all these links thank you.

Yes I have a single node with syslogng and opensearch on the same machine. Also as the other guys I try to avoid opening tcp sockets as much as possible. Also you might want to read data without the need of redundancy and make it through unix sockets.

I'm not sure I understand why nobody wants to implement it (except it requires somes work) but what people at Elasticsearch say would also apply to mysql for instance. Unix sockets would be usefull on mysql just in case mysql is deployed on a single node, same for postgresql and many other applications. They all offer unix sockets even it doesn't work in a multinode environnement.

Here I was asking if opensearch supports unix sockets, I now have the answer. Maybe it would be nice to add this information in the documentation.

Then, I agree that implement this feature requires some work and it might not be the priority of the opensearch projet. However i think this features is more than a "nice to have" feature.

dblock commented 1 year ago

Thanks @mikygee.

I reopened this issue as a feature request. Maybe someone (you?) will be motivated to implement it. It's probably not a small feat, but I could be wrong.

Generally we don't document what doesn't exist, but if you think this is useful, please make a PR into the documentation into https://github.com/opensearch-project/documentation-website.