Consider using pinecone as database backend

[dglazkov]

https://docs.pinecone.io/docs/indexes

[dglazkov]

This looks fairly straightforward. Pseudo-code:

pinecone.init(
    api_key=os.getenv("PINECONE_API_KEY"),
    environment="us-west1-gcp")

index_name = 'glazkov-polymath'
dimensions = 1536

index = pinecone.create_index(name=index_name, dimension=dimensions, metric="cosine")

index.upsert(
    vectors=vectors,
    namespace=index_name, values=True,    
    include_metadata=True)

results = index.query([query_embeddings], top_k=5, include_metadata=True, include_Values=True, 
    namespace=index_name)

[dglazkov]

Probably need to factor "storage" out of Library to get there.

[dglazkov]

• [x] Implement convert.out pinecone

[dglazkov]

• [x] Prototype querying pinecone backend

[dglazkov]

Working prototype code for host.server:

        library = polymath.Library()
        library.omit = 'embedding'
        pinecone.init(
            api_key=PINECONE_API_KEY,
            environment=PINECONE_ENVIRONMENT)
        index = pinecone.Index('polymath')
        embedding = vector_from_base64(query_embedding).tolist()
        result = index.query(
            namespace='wdl',
            top_k=10,
            include_metadata=True,
            vector=embedding
        )
        for item in result['matches']:
            bit = polymath.Bit(data={
                'id': item['id'],
                'text': item['metadata']['text'],
                'token_count': item['metadata']['token_count'],
                'access_tag': item['metadata'].get('access_tag'),
                'info': {
                    'url': item['metadata']['url'],
                    'image_url': item['metadata'].get('image_url'),
                    'title': item['metadata'].get('title'),
                    'description': item['metadata'].get('description'),
                }
            })
            library.insert_bit(bit)
        return jsonify(library.serializable())

Average query type is around 0.3s, from initialization to result.

[dglazkov]

This works! Yay!