Setup Elasticsearch in Terraform

jermnelson commented 5 years ago

Setup an AWS Elasticsearch Service for Sinopia Server.

ndushay commented 5 years ago

I may be misunderstanding this but it seems to me this involves a number of non-trivial steps (i.e. issues of their own with significant work)

decide what flavor of notifications we want

from the Q&A email exchange John had w/ coburn at the very end of october when first researching trellis:

John

Where does Kafka fit in? Is it required by either trellis or trellis-ext-db, or is it just one of the options for a message consumer?

acoburn

Kafka is not required. Both trellis and trellis-ext-db have a mechanism for publishing notifications when resources are created, updated or deleted. Notifications can be turned on or off by configuration, and the two publication mechanisms that come bundled with the app are kafka and jms. For both of those you'd need to have an external broker up and running (trellis doesn't contain an embedded broker of any sort). I also have a notification service that uses AMQP, for those who prefer RabbitMQ or QPid -- that, however, isn't part of the app because no one has indicated an interest in using AMQP, but that could be added as an option. In fact, it's trivial* to add additional protocols. (edited)

John

which i guess answers a part of one of your earlier questions: probably not AMQP, because there's no reason i know of to do any extra work to use RabbitMQ or QPid, since we're generally happy as a shop to use JMS (and kafka if need be? do we actually use kafka in any prod apps right now?)

chose consumer of notifications

Kafka? a JMS consumer? Managed by AWS? ActiveMQ? SNS? SQS?

a broker that can take the notification from Trellis and turn it into the right info to send to ElasticSearch, all in one step?

consumer of notifications, in AWS

managed service such as AmazonMQ? SNS? SQS? etc.

ElasticSearch in AWS

we want BFF to be able to issue GET and POST and PUT requests (and maybe DELETE) to search
Trellis notifications must get to notification consumer
notification consumer must have a way to POST/PUT into ElasticSearch ... and presumably we will need to have some sort of software to take the notification and put it into the index the way we want

turn on notification service in trellis

somehow related to trellis/etc/config.yml ?

notifications:
    enabled: ${TRELLIS_NOTIFICATIONS_ENABLED:-false}
    type: ${TRELLIS_NOTIFICATIONS_TYPE:-JMS}
    topicName: ${TRELLIS_NOTIFICATIONS_TOPIC_NAME:-trellis}
    connectionString: ${TRELLIS_NOTIFICATIONS_CONNECTION_STRING:-tcp://localhost:61616}

assuming this makes trellis a producer of notifications ...

turn on notification consumer in AWS

post/put /delete to trellis and ensure elastic search is changed appropriately.

BFF in AWS can query elastic search, and can use results to get what it wants from trellis (e.g. a resource template)

ndushay commented 5 years ago

@rsmith11 has set up a dev elastic search service on AWS using point and click

rsmith11 commented 5 years ago

I have setup a basic Elasicsearch setup in Terraform. The endpoint is currently here and only accessible from within the AWS VPC:

https://vpc-sinopia-es-development-vzugklhwtqspgqusnelnmlga6m.us-west-2.es.amazonaws.com/

I tested access by sshing into a bastion host in AWS and running:

curl -X GET 'https://vpc-sinopia-es-development-vzugklhwtqspgqusnelnmlga6m.us-west-2.es.amazonaws.com'
{
  "name" : "9BoaqYI",
  "cluster_name" : "418214828013:sinopia-es-development",
  "cluster_uuid" : "mKDUM9LEStmx-WGjaNGIQg",
  "version" : {
    "number" : "6.4.2",
    "build_flavor" : "oss",
    "build_type" : "zip",
    "build_hash" : "04711c2",
    "build_date" : "2018-10-16T09:16:35.059415Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

mjgiarlo commented 5 years ago

@jermnelson what are the acceptance criteria for this issue? Can we close it now that @rsmith11 has AWS ES spun up in Terraform?

jermnelson commented 5 years ago

I think this has been resolved sufficiently for closing this ticket. Closing now...

LD4P / sinopia_server