Improve integration of News Search API

[philbudne]

Deploy the News Search API via the story-indexer docker-compose template for better integration (ie' shared config with importer)?

This could "just" use an image baked separately, or:

1. Reference a separate repo (as a github subtree?) in a new build clause in the story-indexer docker-compose file template
2. Move the news search API code to the story-indexer repo and: a. still build a separate news search API image using a separate Dockerfile b. integrate the news search api into a single story-indexer image (rename from indexer-worker to indexer-common or story-indexer)

[thepsalmist]

Since this is a short-term solution since eventually ES would run on bare metal and the major challenge here is accessing Es running from the swarm, I would go for

Reference a separate repo (as a github subtree?) in a new build clause in the story-indexer docker-compose file template

To maintain the independence of the new-search API.

[rahulbot]

General agreement today is to keep the code separate, but integrate into deployment process, perhaps via git submodule. This is needed to make sure that in the short term the news-search-api and ES are on the same network and can see each other.

[philbudne]

@rahulbot asked (on slack)

Does news-search-api needs a tag with the version of story-indexer that it is deployed with?

One reason (in my mind) is that it leaves an easily viewable artifact in the new-search-api (NSA) repo. Submodule references are stored in the referencing repo, and only as a SHA hash, so the only way to view correspondence is to "check out" the tagged version of the super-module (either on the command line, then use git submodule status, or select the tag on github).

I also want to tag all Docker images in a way that makes it obvious what sources they were built from, and trivial to retrieve those sources (and if needed to branch and patch them).

As a practical matter, I might just as soon treat the NSA as a "foreign" project, and simply have the story-indexer docker-compose file launch an explicit version of an externally/pre-built image. This would mean the NSA project does its own release-engineering (staging and production branches). This would push the complication of making the story-indexer deploy process make sure it has a clean checkout of the NSA sources onto the NSA project's build process.

I understand that the search API/UI is likely to be viewed, and perhaps modified by a different population than the indexer proper, but it's SO little code (three Python source files?) that it feels (to me) like a lot of make-work to maintain it as its own project.

[philbudne]

My current thinking is to have story-indexer treat news-search-api (NSA) as an external project, and to make semantically versioned releases (only incrementing the major version to 2 if a new top level version of the API becomes visible (ie; /v2/ in the URL), and for the story-indexer docker-compose file expect that an image tagged with the NSA version/release number appear in a well-known location (be that a private or public image repo).

(Some of) our other JSON APIs make a server software version number available in all responses (or in response to an API version number endpoint?), so if we go this route, I would vote to make NSA server version visible in responses, which would allow client software (ie; the news search Django app) to detect when:

1. new top level API versions are available (major version incremented)
2. new calls available, or new optional arguments available for old calls (minor version incremented)
3. bug fixes available, and workarounds no longer needed (patch version incremented)

[rahulbot]

This sounds reasonable. That app is fairly well containerized already and won't take much to start to get into shape. I'll make issues for first steps on in that news-search-api repo. Can revise and revisit if others disagree.

[philbudne]

The story-indexer docker-compose.yml.j2 template now deploys news-search-{api,ui} containers, the image registry, name and tag are configured in deploy.sh (with place-holder values):

NEWS_SEARCH_IMAGE_NAME=colsearch       # XXX news-search-api???
NEWS_SEARCH_IMAGE_REGISTRY=localhost:5000/ # XXX replace with real registry!
NEWS_SEARCH_IMAGE_TAG=latest    # XXX replace with version????

[rahulbot]

Closing - versioning and such can be handled on that repo on bugs I created a bit ago to capture those ideas.