search

babenkoivan / elastic-scout-driver-plus

Extension for Elastic Scout Driver
MIT License
267 stars 52 forks source link

AWS opensearch with IAM credentials? #139

Closed onlinesid closed 2 years ago

onlinesid commented 2 years ago

Is there a way to make this work with AWS opensearch with IAM credentials?

babenkoivan commented 2 years ago

Hi there 👋🏻

This package uses elastic-client as a dependency, which is basically a wrapper for elasticsearch-php. Both packages are flexible and allow you to modify the client, the transport, etc. As far as I know, there is no out-of-the-box solution for AWS, but I'm sure you can find some Guzzle handlers or similar published on github that can be reused.

Please, share your findings if you manage to crack it 🙂

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

onlinesid commented 1 year ago

Sorry I got side tracked for a while, I didn't find a solution but managed to find a workaround for the time being.

chrillep commented 1 year ago

@babenkoivan @onlinesid i believe u need https://github.com/opensearch-project/opensearch-php

so elastic-client would need to be able to use both, not sure of the size of that refactor 😅

and testing ...

babenkoivan commented 1 year ago

@chrillep I will look into it, but this looks like too much hassle. There should be an easier way to connect to OpenSearch. elasticsearch-php allows you to configure your own HTTP client, maybe it would be enough to extend the Guzzle client and sign requests before sending them to AWS. I'll try to prototype a solution when I have time.

chrillep commented 1 year ago

@babenkoivan ❤️. just FYI your packages

"babenkoivan/elastic-migrations": "^2.0.1", "babenkoivan/elastic-scout-driver": "^2.0.0", "babenkoivan/elastic-scout-driver-plus": "^3.5.1",

work for opensearch 2.6 . After that i believe elasticsearch-php does a version check on requests with headers. And since opensearch has a lower major but same functionality as elastic it fails and says to low version something something 😀.

Docker opensearch current v2.6 elasticsearch current 8.7

babenkoivan commented 1 year ago

@chrillep I have an update on this topic. As I don't have much experience with OpenSearch, I thought that it just runs Elasticsearch under the hood, but it is a little bit more complicated than that. It turned out that OpenSearch is based on an old Elasticssearch fork and now being developed in a separate way. Even though I was able to sign requests with IAM credentials, I faced multiple issues when sending requests to OpenSearch via Elasticsearch client.

As both OpenSearch and Elasticsearch can be considered as two separate products now, the difference in implementation and functionality will be only increasing with time. Maintaining both implemetations (i.e. two different clients and adapters) might be extremely difficult and, unfortunately, I don't have that amount of time.

I did some prototyping, estimated the amount of work and decided not to go this way. Sorry, but I'll focus on Elasticsearch for now.

If you have time and desire, you can fork elastic- libraries and adjust the code to use opensearch-php instead of elasticsearch-php client.

chrillep commented 1 year ago

@chrillep I have an update on this topic. As I don't have much experience with OpenSearch, I thought that it just runs Elasticsearch under the hood, but it is a little bit more complicated than that. It turned out that OpenSearch is based on an old Elasticssearch fork and now being developed in a separate way. Even though I was able to sign requests with IAM credentials, I faced multiple issues when sending requests to OpenSearch via Elasticsearch client.

As both OpenSearch and Elasticsearch can be considered as two separate products now, the difference in implementation and functionality will be only increasing with time. Maintaining both implemetations (i.e. two different clients and adapters) might be extremely difficult and, unfortunately, I don't have that amount of time.

I did some prototyping, estimated the amount of work and decided not to go this way. Sorry, but I'll focus on Elasticsearch for now.

If you have time and desire, you can fork elastic- libraries and adjust the code to use opensearch-php instead of elasticsearch-php client.

@babenkoivan yes i totally understand! Thanks for looking into it! :)

We will keep using "babenkoivan/elastic-migrations": "^2.0.1", "babenkoivan/elastic-scout-driver": "^2.0.0", "babenkoivan/elastic-scout-driver-plus": "^3.5.1",

until opensearch breaks it :)