setup gcp hosted search

[micahstubbs]

setup gcp (google cloud platform) hosted search

💭 reasoning

1) gcp with dev credits is likely lower-cost than our existing elastic cloud search index instance. 2) support for the old version of Elasticsearch we currently run (2.4.x something) is going away on 8/28, so we have to re-write our search queries in our blockbuilder search frontend code soon anyway 3) since we have to re-write search queries anyway, might as well migrate to a lower cost provider while we are at it

📑 tasks

• [ ] write a nodejs client for App Engine Searchable Document Indexes
• [ ] figure out a mapping from gists contents to one or more AppEngine Documents
• [ ] write a script to import gists into AppEngine Documents
• [ ] write some command line search queries to test the mapping / schema
• [ ] write new queries for each search action possible today from the blockbuilder.org/search UI
• [ ] make a branch of the blockbuilder.org/search UI with new queries
• [ ] a/b test GCP AppEngine Search with Elasticsearch

[micahstubbs]

@enjalot curious, is this the App Engine Searchable Document Indexes service that you had in mind? https://cloud.google.com/appengine/docs/standard/java/search/?hl=en_GB&_ga=2.194014253.-1243297456.1534856335

[micahstubbs]

there's also this Cloud Search product, but it seems targeted at big enterprises rather than developers 🤔 https://cloud.google.com/products/search/

[micahstubbs]

so there is also Google Cloud Datastore, but that looks like a document store rather than a search engine https://db-engines.com/en/system/Elasticsearch%3BGoogle+Cloud+Datastore

[micahstubbs]

this list looks helpful

https://db-engines.com/en/ranking/search+engine

ah, but this Google Search Appliance https://enterprise.google.com/search/products/gsa.html is just the earlier version of the enterprise-y Google Cloud Search we found earlier. hhhmmm.

[micahstubbs]

ok, it looks like this is the one App Engine Searchable Document Indexes https://cloud.google.com/appengine/docs/standard/python/search/

[micahstubbs]

here are some code examples for using the search API from a Python client https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/search/snippets/snippets.py

[micahstubbs]

I did some research.

I want a Node.js client for the AppEngine Searchable Document Indexes, which today only has clients in Python 2.7, Java, and Golang. I'm thinking if I can find the source code for one of the existing SDKs, perhaps I can translate it into JavaScript 💡

let's search for the google.appengine.api python package mentioned on this line from google.appengine.api import search in one of the official SDK example code snippets https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/search/snippets/snippets.py

prior work

• like-minded people at optimizely posted a similiar mirror of the SDK back in 2015 in a repo called python-appengine. this repo appears to have no search directory, and so no support for the particular service I would like to use. 🤔 https://github.com/optimizely/python-appengine

• there is this promisingly named appengine-sdk repo in the official GoogleCloudPlatform github account. sadly, this repo contains only a LICENSE file added back in 2016 😢 https://github.com/GoogleCloudPlatform/appengine-sdk

• there is another promising repo called google-cloud-python in the official GoogleCloudPlatform github account, which does appear to be actively maintained. strangely, this repo appears to have no search directory, and so no support for the particular service I would like to use. 🤔 https://github.com/GoogleCloudPlatform/google-cloud-python

[micahstubbs]

ok, posted in the gcp community slack to see if they know about this search module

https://googlecloud-community.slack.com/archives/C0FEPH2SW/p1535011115000100

[micahstubbs]

@enjalot the answer to this question suggests that what's on offer may not really match what we're looking for (provided nothing has changed in the year or two since this answer)

https://stackoverflow.com/questions/40797735/does-google-cloud-offer-server-less-search-api

[micahstubbs]

got some answers on the gcp slack 😄

think you can find what you're looking for by downloading the original App Engine SDK for Python from here: https://storage.googleapis.com/appengine-sdks/featured/google_appengine_1.9.74.zip (which can be found at the bottom of https://cloud.google.com/appengine/docs/standard/python/download)

Once you download that, the search API is in google_appengine/google/appengine/api/search

https://googlecloud-community.slack.com/archives/C0FEPH2SW/p1535040913000100

[micahstubbs]

here's the source code for the Python client, that we could perhaps use as a template for writing a nodejs client

https://github.com/micahstubbs/google_appengine/tree/master/google/appengine/api/search