SWODLR-SDS Issues - Githubissues

joshgarde commented 7 months ago

Issues which have been observed on the SWODLR HySDS instance:

Jobs failing upon retry due to noclobber setting
Odd API behavior where jobs report as completed during intermediate step and change to failed later
Job timeouts being too low
ElasticSearch failures lead to unpredictable/difficult to diagnose API issues

Current plan for corrective action:

SDS: handle noclobber issue
SWODLR: Investigated and found to be false issue
SWODLR: Timeouts to be lowered to 10 hours instead of increased (TODO)
SDS: implement automated monitoring of ElasticSearch on instance

joshgarde commented 6 months ago

HySDS ticket launched: https://hysds-core.atlassian.net/browse/HC-516
SWOT aware of noclobber behavior and ticket launched
SWODLR fixes deployed
- Issues with exception handling fixed in latest commons PR
- https://github.com/podaac/swodlr-common-py/pull/4

viviant100 commented 6 months ago

3/25/24: The update of "SDS: handle noclobber issue" was developed in v5.1 and will be deployed on the week of 4/8/24. v5.1 is currently in SDR and should be wrapped up on the week of 4/1/14.

podaac / swodlr

SWODLR-SDS Issues #105