Currently, we retrieve 500 druids from Solr of the least recently updated items, build the items, update Solr, and wait for Solr to commit. The reason for the wait for the commit is so that we the next 500 druids are retrieved, they don't include the just updated items.
In the new approach, we will retrieve a large number of druids of the least recently updated items, split them in batches of 500, build the items in each batch and post to Solr. Only the last batch will wait for Solr to commit.
Currently, we retrieve 500 druids from Solr of the least recently updated items, build the items, update Solr, and wait for Solr to commit. The reason for the wait for the commit is so that we the next 500 druids are retrieved, they don't include the just updated items.
In the new approach, we will retrieve a large number of druids of the least recently updated items, split them in batches of 500, build the items in each batch and post to Solr. Only the last batch will wait for Solr to commit.
This eliminates the wait after every batch.