Document similarity worflow

[saggu]

Pipeline should work as follows:

Process each incoming document: create sentence vectors indices

Store the indices so that it can be re created if the process dies

For each query: compute vector, find k nearest matches irrespective of any threshold and return the ranked result which is a list of document ids with similarity scores

Fetch the documents from ES and return to DIG UI

If the user chooses a facet, add filter to the list of documents for a query, re rank the results and return to DIG UI. So, if originally we had k documents, adding a facet will always return <= k documents. The facets act as a filter

[saggu]

Updated Pipeline

[saggu]

The followings tasks are done:

1. Vectorize each sentence using tensorflow
2. Index the vectors in FAISS index, and store the link in hbase
3. Able to query a string and return k similar docs back