wellcomecollection / concepts-pipeline

Some sort of ETL pipeline for concepts in the Wellcome Collection catalogue
MIT License
0 stars 0 forks source link

Aggregator warns when attempting to fetch a redirected work #114

Closed paul-butcher closed 1 year ago

paul-butcher commented 1 year ago

When running using the WorkIdSource, the aggregator emits a warning if the workid resolves to a resource returning 302 Found. e.g:

15:44:39.127 [main-akka.actor.default-dispatcher-17] WARN w.c.aggregator.sources.WorkIdSource - Could not fetch eg4a5xkp: request returned 302 Found 
15:44:39.318 [main-akka.actor.default-dispatcher-17] WARN w.c.aggregator.sources.WorkIdSource - Could not fetch fhhzkwxd: request returned 302 Found  
15:44:39.458 [main-akka.actor.default-dispatcher-7] WARN w.c.aggregator.sources.WorkIdSource - Could not fetch squm78qk: request returned 302 Found 

I suspect that this is not a problem, if a changed work results in a redirect, and the change is pertinent (i.e. the work with the id contains Concepts that make their way onto the dominant work in the merge), then I expect the work to which it redirects will also be updated. If there are no changes in the redirection target, then there is nothing to do.

If that is the case, then this warning is still useful, as it will help us diagnose why something hasn't happened, but seeing this warning in the logs is a bit concerning

There are two options here: