clouway / cuse

clouWay universal search engine
1 stars 7 forks source link

Issue #2 added bulk register for indexes #34

Closed VasilMitovClouway closed 7 years ago

VasilMitovClouway commented 7 years ago

It is now possible to pass in a list of documents and it will be registered in chunks of 200 (GAE limitation).

mgenov commented 7 years ago

Few notes:

Safe Indexing taken from original docs at: https://cloud.google.com/appengine/docs/standard/java/search/

public static void indexADocument(String indexName, Document document)
    throws InterruptedException {
  IndexSpec indexSpec = IndexSpec.newBuilder().setName(indexName).build();
  Index index = SearchServiceFactory.getSearchService().getIndex(indexSpec);

  final int maxRetry = 3;
  int attempts = 0;
  int delay = 2;
  while (true) {
    try {
      index.put(document);
    } catch (PutException e) {
      if (StatusCode.TRANSIENT_ERROR.equals(e.getOperationResult().getCode())
          && ++attempts < maxRetry) { // retrying
        Thread.sleep(delay * 1000);
        delay *= 2; // easy exponential backoff
        continue;
      } else {
        throw e; // otherwise throw
      }
    }
    break;
  }
}

Another note:

You can pass up to 200 documents at a time to the put() method. 
Batching puts is more efficient than adding documents one at a time.