search

open-guides / og-aws

📙 Amazon Web Services — a practical guide
Creative Commons Attribution 4.0 International
35.74k stars 3.88k forks source link

Cover Elasticsearch #64

Open jlevy opened 8 years ago

jlevy commented 8 years ago

AWS' own Elasticsearch service is growing in popularity. Discuss basics, when to manage yourself vs with AWS' managed service, etc.

jlevy commented 8 years ago

@nitingithub want to take this one?

zoellner commented 8 years ago

This would probably need some comment on how AWS is lacking behind quite a bit regarding the available ES versions and the implications to get a system running securely in a VPC

jlevy commented 8 years ago

@zoellner yeah, what I've heard too. That's exactly the sort of tip we should put in!

jlevy commented 8 years ago

PR welcome, or if you'd like to put that tip here (any links/details to add?) we can incorporate it.

dcosson commented 8 years ago

In addition to version lagging (which right now is not actually too bad, they bumped up to 2.3 I believe in July), they have scripts disabled. And there are some relatively straightforward uses that require scripts in elasticsearch, e.g. sorting an aggregation by query score is one I just ran into.

kimschmidtsbrain commented 8 years ago

I can help here. I think Elasticsearch Service is one of the most powerful yet unknown service...as long as I ramp up my GitHub game lol ;-)

jlevy commented 8 years ago

dorrin [8:22 AM]
I'd like to hear stories from people who switched from managed ElasticSearch to building and maintaining their own cluster

rjhintz [10:01 AM]
@dorrin Regarding: Elasticsearch. This was a recent topic on Reddit/aws. It seemed to draw some informed opinion (as opposed to "normal" Reddit.) https://www.reddit.com/r/aws/comments/55dpn6/elasticsearch_on_aws_managed_or_roll_your_own/

DorrinPk commented 8 years ago

Price can also be a factor. ElasticSearch service's instances are more expensive than regular EC2 instances.

untergeek commented 8 years ago

Piping in from the field here. AWS ES does not allow use of the /_cluster/state/metadata endpoint. This causes problems with supporting tools, like Elasticsearch Curator.

See https://github.com/elastic/curator/issues/717

DorrinPk commented 7 years ago

Troubleshooting the ES service is also a problem. You can't see the instances behind the ES cluster so if you can't fix an issue by adding more memory or storage then you should contact support.

DorrinPk commented 7 years ago

AWS has a list of ES service limits as well : http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html Mainly no scripting, plugins, heap size limit, max instance count.

jlevy commented 7 years ago

Another major security gotcha to list: http://www.zdnet.com/article/first-came-mass-mongodb-ransacking-now-copycat-ransoms-hit-elasticsearch/