Initial search indexer implementation

[franckc]

Checklist:

• [X] Code contains relevant tests for the problem you are solving
• [X] Code has been formatted with autopep8 --in-place --recursive --a --a .
• [X] Ensure all new and existing tests pass
• [X] Update any relevant READMEs and docs
• [X] Submit to the develop branch instead of master

Description:

This diff implements a primitive SearchIndexer module that indexes events in ElasticSearch. This is just a starting point - code needs to be harden and there is additional data we should discuss indexing as well.

In terms of search engine, it assumes we'll use Heroku's "Bonsai ElasticSearch" add-on but it would be simple to switch to a different ElasticSearch hosted solution.

Next step will be to implement a Search API that the DAPP can hit.

[franckc]

I don't have experience with Heroku, but it does look like Bonsai ES is fairly well supported as an add-on. Here is how to configure it: https://devcenter.heroku.com/articles/bonsai#installing-the-add-on

Bonsai does not provide any added-functionality on top of ES. It just runs it as a hosted service. That means we would not be locked in. We could easily use a different ES hosted solution on Heroku (like the official ES add-on). If we ever decide to move our bridge-server infra to AWS we could use their hosted solution https://aws.amazon.com/elasticsearch-service/ or alternatively we could even run ES ourself on AWS instances. Another thought is that in terms of development environment, we can easily package ES as part of the bridge-server docker container we provide to developers since ES runs without any config changes as a single process on a single node.

[joshfraser]

As discussed on the call today, we'll want to get this added to the docker file before merging. Want to minimize the pain for developer setup.

Looks like we have a few small merge conflicts to fix as well.

[franckc]

Sounds good - I'll follow up regarding adding ElasticSearch to the development docker container. Will also rebase and update this PR to get rid of the merge conflicts.

[ambertch]

According to https://docs.bonsai.io/docs/supported-elasticsearch-versions the version of elasticsearch depends on region.

I'm reaching out to Bonsai support to inquire how to know which region it will be in when added via Heroku.

[franckc]

@ambertch: Thank you ! It would indeed be ideal if the version of ElasticSearch that we package in the Origin's development docker container matches with the version that will be used in production. Regarding what version will be running in production, I would assume Bonsai will be colocated in the same region as Origin's bridge-server Heroku deployment.

@joshfraser: Do you know in which region the bridge-server is deployed to ? I would think the Heroku admin console/UI would show that information somewhere ? Or we could simply go-ahead and install the Bonsai add-on, then determine what version got installed... :) I'd recommend we start with the cheapest plan called "Staging" that costs $10/month - seems easy to upgrade later on if we need to.

Sorry I can't help since I don't have access to Origin's Heroku console...

[ambertch]

According to https://devcenter.heroku.com/articles/regions#default-regions the region the app is hosted in (these look to to correspond to AWS availability zones) can be set via CLI, although "Unless a different region is specified, apps in the Common Runtime will be created in the us region"

From Bonsai: Virginia | 2.4.0 / 5.3.2 / 5.4.3 / 6.0.0 Oregon | 2.4.0 / 5.4.3

[ambertch]

Added Elasticsearch 2.4.0 as a temporary default (it is installed in all Bonsai regions) https://github.com/OriginProtocol/origin-js/commit/c7c3d0f119e80c283b229e6a45aa19cc8844a7e0

[ambertch]

Ok! Got clarification from Bonsai:

• The Bonsai cluster is in the same region as the Heroku app
• "The region and version are also displayed on the Bonsai dashboard which you can get to by running heroku addons:open bonsai in your terminal." -"Iif you don't specify a version when you provision a cluster (you can specify using the --version=x.y flag) it will default to the highest version available in that region which is currently either 5 or 6 depending on the region."
• "The table with the different versions shows the all the different versions available in those regions, so you can provision on 2 if you specified 2 during provisioning."

According to the docs, the minimum of the highest versions across regions (and also across the north american regions, which is the default for a heroku app) is 5.4.3

The docker file has been updated to install 5.4.3

[franckc]

Thanks for the review ! Yep totally makes sense re using parseurl -- will switch to that.

Oops the git branch on my fork is totally busted that's why it has this weird git commit history. I'll consolidate the two search PRs I have out into a single new clean PR that I'll send out for review.

[franckc]

Closing this PR - implemented as part of #81