search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.59k stars 1.55k forks source link

Support AWS OpenSearch Serverless for `elasticsearch` sink #16252

Open AvihaiSam opened 1 year ago

AvihaiSam commented 1 year ago

AWS OpenSearch Serverless Service endpoint is aoss instead of es.

Please add a config to distinguish ES from AOSS, and fix sign_request call.

https://github.com/vectordotdev/vector/blob/c7a4d7b46a4b351f936b4ac675db3b12ece59734/src/sinks/elasticsearch/common.rs#L258

I'm not sure if there are future changes needs to be done afterwards.

davidhuie-dd commented 1 year ago

I agree that we should support this.

zamazan4ik commented 1 year ago

@AvihaiSam Could you please test the fix from the branch - https://github.com/vectordotdev/vector/pull/16590 ? I have no ability to test it on my own right now. You need to build Vector from the provided PR and test with your setup with AWS OpenSearch Serverless.

Thanks in advance!

AvihaiSam commented 1 year ago

Hey @zamazan4ik, I have actually compiled a different version a week ago and tried it. I have noticed several things:

  1. the health API is not available
  2. the version API is not available (at this point, my guess is that all the /_cluster APIs are not available)
  3. my sigv4-auth signature was wrong, using the assume-role auth-option (maybe my version made it fail, i'm no rust expert...) For the last thing - I have opened a support ticket to AWS and they are looking into it. Since this issue makes the cluster impossible to work with we have decided to delete the cluster due do its high costs. So - unfortunately, I can no longer test it.

Hope to hear from AWS and will let you know if/when I will have access to a new cluster...

zamazan4ik commented 1 year ago

I have actually compiled a different version a week ago and tried it.

What version did you compile? Does it have the same fixes for AOSS as from my branch?

AvihaiSam commented 1 year ago

I tried my own implementation here

handlerbot commented 5 months ago

@AvihaiSam 👋🏻 I have interest in getting vector working with OpenSearch Serverless; any objection to me taking what you've got in your tree and seeing what the current gaps are, and me getting it committed with you listed as primary author if I can get to success? Or do you have current plans or thoughts?

AvihaiSam commented 5 months ago

@handlerbot sure. I left it there hoping someone will find the time to work on that.

vitalvas commented 3 weeks ago

I'm so interested in this support. Do you have any progress?