acquire train data from lmdb

[jemmyshin]

Sometimes we need to train the PCA model when we already created an indexer. (for example, there is a memory issue after we have indexed thousands or even millions of data, and we need PCA to fix it.)

We need to fetch train data from lmdb, but this is tricky when we move to jcloud since we need to fetch data from the server instead of local machine.

One way to solve this is to add a new endpoint in client called /fetch:

data = client.post('/fetch', params={'batch_size': 1024})

for training we can use partial_train():

annlite.partial_train(data)

[JoanFM]

lets make sure we are not overcomplicating the indexer and its usage.

[JoanFM]

If this LMDB is not going to work on jcloud, please do not proceed with this idea. If there is a need for less memory, for now I would just expect to have proper contiguration from beginning.

[JoanFM]

what will be the difference between this training data and the indexed data?