search

MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

OAI endpoint error handling #432

Open antmoth opened 5 years ago

antmoth commented 5 years ago

We should kindly inform the user that there is a problem if an attempted OAI harvest finds: (1) a down endpoint (2) no records

antmoth commented 5 years ago

image ^ no records scenario maybe taken care of

handling "endpoint down or unreachable" might require a completely different code intervention. I knew there was a reason I thought those should be separate issues :D

antmoth commented 5 years ago

I'm not entirely sure that marking the job as invalid when it just has no records is the right thing to do... Maybe I should add a 'warning' state, 'errored' or something.

antmoth commented 5 years ago

The problem with 'down endpoint' is that when I try to, e.g., put in an invalid endpoint URL, the job seems to never stop trying, and to run forever? @ghukill

antmoth commented 4 years ago

Could run a health test on the endpoint/ping it before the Spark task runs. Error language needs to be friendly