Fix indexing issues

[WadeBarnes]

On occasion our indexes get out of sync with our credential records.

There are a a couple classes involved in the process, the TxnAwareSearchIndex on which the CredentialIndex is built and the SolrQueue class. The TxnAwareSearchIndex class queues up credentials with the SolrQueue class for indexing once the transaction for the related credential is complete. The SolrQueue's indexing methods get fired off periodically to process the queued items. The indexing process uses features of the search index to get the back-ends for the database and indexing service (Solr).

Somewhere in this process credentials are being saved to the database, but are not being indexed.

[WadeBarnes]

Investigation:

With the help of some additional logging and tools created to investigate the issue, it appears we have identified a potential issue ...

While inside the SolrQueue's indexing process, if the indexing service or database service are unavailable an uncaught exception is thrown. This has 2 affects; 1) The current item is not indexed, nor is it placed back on the queue for later processing, 2) The exception bubbles up through the class's timing mechanism and exists, which stops the indexing process from ever checking the queue again. Credentials continue to be added to the queue, but they are never indexed.

[WadeBarnes]

The fix;

• Wrap the indexing process(es) in exception handling and re-queue items for later processing when there are exceptions.

[WadeBarnes]

Addressed the identified issue, and also addressed an issue where the indexing process could fail silently when there is an error registering the index(es) with Solr.

[WadeBarnes]

One thing the SolrQueue class does not address, something we're planning on dealing with when migrating to bcgov/indy-catalyst, is the persistence of queued items to protect against indexes being lost due to pod restarts.

[WadeBarnes]

Updates deployed and all environments have had their indexes synced.