The core plugin plgCronSolr is meant to provide full and queue-based Solr reindexing of Hub components via Hubzero cron. The plugin's runFullIndex() function for full reindexing of all indexed Solr content was buggy and did not reindex Hub content as expected. This PR repairs that functionality.
plugin's runFullIndex() function should call indexSearchResults() for each indexed component, and should return a boolean when all indexing is complete
plugin should only reindex those components that were previously indexed
language files and xml manifest were cleaned up and corrected; references to other plugins and unused options were removed
an isset() check was done for $recordsIndexed['offset'] in components/com_search/admin/controllers/searchable.php prior to accessing the variable.
Note
Previous PRs have corrected 500 errors thrown by this plugin (See #1586). The present PR builds on this earlier work by fixing the reindexing functionality. An additional function is roughed out in this plugin to enable queue-based indexing of components. This will be implemented in future work, as it is also non-functional at present (see [HZ-342]).
Motivation
The plgCronSolr plugin's runFullIndex() function could not be run reliably via Hubzero cron as designed. It only intermittently indexed components; it attempted to index components not configured for indexing; and it did not return a boolean, as required by Hubzero cron.
From the Google Doc called “Search improvements for hubzero platform”:
Running the full index task originally picked the non-indexed component with the lowest unique id from jos_solr_search_searchcomponents, enabled it regardless of its state, and (re)indexed it.
Testing
This work was tested by scheduling the Solr Full Index option via Hubzero cron for various configurations of enabled and disabled components. During testing, monitoring various logs and database tables to ensure that the reindexing occurred as specified.
Testing is performed using the admin back-end to set up Solr indexing and Hubzero cron full-indexing; using terminals to monitor logs; and querying database tables.
For Solr administration, access the admin interface, Components:Search, then Searchable Components. Indexing is enabled on a given component (e.g. Groups or Resources) by clicking its "Active" icon. The Records column for a component indicates the number of records currently indexed in Solr.
Logs:
Solr log: /var/log/hubzero-solr/solr.log (may be found in /srv/hubzero-solr/logs/, in which case symlink needed)
jos_cron_jobs contains cron jobs and their status and schedules
jos_solr_search_searchcomponents should show indexed components, index item counts
Other
No hotfix is needed. Review will be assigned to Nick and Dave.
Out of scope
Other findings were noted on undertaking this work. These should be scoped separately:
searchable components that are not presently indexed sometimes show a nonzero record count in the admin Solr searchable components interface. This should be fixed.
note that regardless, these items are not indexed, but the total record count should be 0. Likely this is in: components/com_search/admin/controllers/searchable.php
On indexing the Blog component, then clicking on the record count: admin interface displays 500 error: "undefined index: access_level". This is likely to be a model class problem.
Similarly, in Citations, no records display on indexing. Then, on clicking through on the record count, admin interface throws "Undefined index: owner_type". This is likely to be a model class problem.
It would be helpful to display indexing timestamp (indexed) and actual record count (indexed_records) in the jos_solr_search_searchcomponents table
Logging and error handling is wanted. Presently, one accomplishes this by scrutinizing the solr.log and/or the hub-cmscron.log while indexing.
Summary
The core plugin
plgCronSolris meant to provide full and queue-based Solr reindexing of Hub components via Hubzero cron. The plugin'srunFullIndex()function for full reindexing of all indexed Solr content was buggy and did not reindex Hub content as expected. This PR repairs that functionality.This work is described in Jira Card #HZ-341.
Fix details
The following issues are addressed:
runFullIndex()function should callindexSearchResults()for each indexed component, and should return a boolean when all indexing is complete$recordsIndexed['offset']in components/com_search/admin/controllers/searchable.php prior to accessing the variable.Note
Previous PRs have corrected 500 errors thrown by this plugin (See #1586). The present PR builds on this earlier work by fixing the reindexing functionality. An additional function is roughed out in this plugin to enable queue-based indexing of components. This will be implemented in future work, as it is also non-functional at present (see [HZ-342]).
Motivation
The
plgCronSolrplugin'srunFullIndex()function could not be run reliably via Hubzero cron as designed. It only intermittently indexed components; it attempted to index components not configured for indexing; and it did not return a boolean, as required by Hubzero cron.From the Google Doc called “Search improvements for hubzero platform”:
Testing
This work was tested by scheduling the Solr Full Index option via Hubzero cron for various configurations of enabled and disabled components. During testing, monitoring various logs and database tables to ensure that the reindexing occurred as specified.
Testing is performed using the admin back-end to set up Solr indexing and Hubzero cron full-indexing; using terminals to monitor logs; and querying database tables.
For Solr administration, access the admin interface, Components:Search, then Searchable Components. Indexing is enabled on a given component (e.g. Groups or Resources) by clicking its "Active" icon. The Records column for a component indicates the number of records currently indexed in Solr.
Logs:
/var/log/hubzero-solr/solr.log(may be found in/srv/hubzero-solr/logs/, in which case symlink needed)/var/log/hubzero-cms/hub-cmscron.logDatabase tables:
jos_cron_jobscontains cron jobs and their status and schedulesjos_solr_search_searchcomponentsshould show indexed components, index item countsOther
No hotfix is needed. Review will be assigned to Nick and Dave.
Out of scope
Other findings were noted on undertaking this work. These should be scoped separately:
components/com_search/admin/controllers/searchable.phpindexed) and actual record count (indexed_records) in thejos_solr_search_searchcomponentstable