search

MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

BG task spark polling needs improvement #192

Closed ghukill closed 5 years ago

ghukill commented 6 years ago

Using the library polling to poll Spark when tasks are complete from bg tasks.

Currently, waiting for available from response headers, and polling infinitely. This seems to work so far, given even an error in a Spark job returns available. But there are likely edge cases like Spark goes down or is reset, which would return something different. Need to figure these out and handle.

ghukill commented 5 years ago

Not an issue with Celery anymore (or at least changed).