Use cursor to navigate query result in sitemap generation

clarin-eric / VLO

Virtual Language Observatory

GNU General Public License v3.0

14 stars 6 forks source link

Use cursor to navigate query result in sitemap generation #245

Closed twagoo closed 4 years ago

twagoo commented 5 years ago

Scope: SOLRService class of the vlo-sitemap subproject. Currently all record ids are requested at once. This is heavy on resource uses (heap space in Java + a strain on Solr), so better to use cursor based navigation and fetch them in batches. For an example, see the AvailabilityStatusUpdater class in the vlo-import project.

twagoo commented 5 years ago

Also it would be good to use the SolrJ client and not a 'raw' http request

twagoo commented 4 years ago

See 9c504b2e007512a036a631f0d6704d1785174fc6