search

stagemonitor / stagemonitor-mailinglist

GitHub issues abused as a mailing list
3 stars 0 forks source link

Support for AWS managed elasticsearch #36

Open quintonm opened 7 years ago

quintonm commented 7 years ago

Does StageMonitor support using AWS managed ElasticSearch? The AWS version does not work with the normal java client. Only the REST API is exposed. I didn't really see anything in the docs to indicate if it would or would not work.

felixbarny commented 7 years ago

As stagemonitor only uses Elasticsearch's HTTP API I don't see a problem although I've never tried the AWS ES version. What is the version number of underlying ES server?

quintonm commented 7 years ago

2.3

felixbarny commented 7 years ago

Stagemonitor is compatible with 2.x as listed in the wiki: https://github.com/stagemonitor/stagemonitor/wiki/Installation#requirements

quintonm commented 7 years ago

Cool. I will try it out and update it if works as expected.

ryanrupp commented 7 years ago

When the AWS ES service originally was released I looked into it and the problem was the security mechanisms were pretty limited (at that time at least) in terms of making API requests to Elasticsearch. Seemed the access options were either whitelisting by IP or you have to sign the requests. If signing the requests is still a requirement Stagemonitor would have to provide some hookpoint do to this.

For reference, here's the issue where Zipkin added support for AWS Elasticsearch where they use an interceptor to do the signing as an example of a project doing this.

felixbarny commented 7 years ago

Have you already tried this out? Is signing the request still required or are there other options now?

quintonm commented 7 years ago

I haven't gotten to it yet. However, it will have some problems with the AWS managed elasticsearch.

1) The only way to implement security are ip whitelisting and request signing. Some people get around this by adding a proxy in front of elasticsearch to perform basic auth. This allows for the proxy to perform the request signing. It also give people a way to secure Kibana with basic auth. However, I think most would prefer to have support for request signing.

2) It is not possible to change the value of threadpool.bulk.queue_size. That means that you can't have a large number of JVMs all sending metrics directly to elasticsearch. Some options around this might be to add support for shipping metrics to logstash, writing to a logfile to be picked up by filebeats, kafka, etc.

On Mon, Dec 12, 2016 at 1:12 AM, Felix notifications@github.com wrote:

Have you already tried this out? Is signing the request still required or are there other options now?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/stagemonitor/stagemonitor-mailinglist/issues/36#issuecomment-266358123, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8gtaYV_g3SJnlm9qK9xxLf1LywlIQsks5rHPPdgaJpZM4LFmaE .