search

AtlasOfLivingAustralia / bie-index

Taxonomic search services
https://bie-ws.ala.org.au/ws
Other
1 stars 17 forks source link

Non-taxonomy pages to be regularly indexed via externally scheduled/triggered service #300

Closed nickdos closed 3 years ago

nickdos commented 4 years ago

The index contains non-taxomony sources such as Wordpress pages, Knowledge Base pages, spatial layers, collectory pages, etc. that change regularly and need to be accurately reflected in the SOLR index.

The problem is the index only gets updated when we have a new taxonomy source, which is a handful of times a year.

Suggested solution is to allow some sources to be indexed into the live "core" and be triggered via Jenkins on a daily or weekly basis. An API key is probably needed to protect the remote trigger end-point.

Edit: For WP and KB pages, the ideal solution is to use some sort of push notification to trigger a reindex of the changes pages.

A pub/sub or MQ system (e.g. JMS, Kafka, Google pub-sub) would be ideal (could handle outages, etc) but probably overkill if we want to get this feature "out".

charvolant commented 3 years ago

See https://github.com/AtlasOfLivingAustralia/bie-index/pull/305

Existing API allows triggering imports by web service calls or admin UI. This is probably all that is needed to allow external systems to trigger an update.

charvolant commented 3 years ago

Fixed in https://github.com/AtlasOfLivingAustralia/bie-index/releases/tag/v1.4.12