search

AtlasOfLivingAustralia / biocache-service

Occurrence & mapping webservices
https://biocache-ws.ala.org.au/ws/
Other
9 stars 26 forks source link

Pipelines resiliance #884

Open adam-collins opened 4 months ago

adam-collins commented 4 months ago

Placeholder task for gbif/piplines livingatlas changes

  1. Retrofit calls to collectory, namematching, sensitive should back off and retry upon failure. This is prevent failure caused by server restarts and servers under load. e.g. status codes 404, 502, 504.

  2. Instances of aggregation and getOnly exceptions are not logged clearly. Starting with IndexRecordToSolrPipeline

    • Log dataResourceUid for index record duplicates
    • Log if jackknife needs rebuilding
    • Log if clustering needs rebuilding
    • Log if outliers need rebuilding
    • Log if annotations needs rebuilding
    • Log if sampling needs rebuilding