GSA / ckanext-geodatagov

data.gov extension
Other
35 stars 39 forks source link

speed up relink by improving solr indexing #260

Closed FuhuXia closed 1 year ago

FuhuXia commented 1 year ago

Looks like we need to turn manual relink process into a nightly job since we see a few thousand packages that need to be relinked in every a few days. This commit change solr indexing from package by package to batch processing, so it speeds up the process to make it ready to turn into a GH cron action.

jbrown-xentity commented 1 year ago

Hopefully this doesn't result in more errors, but since the single processing has been fine hopefully the batch should be as well... 🤞