epag
commented
3 weeks ago Original Redmine Comment Author Name: James (James) Original Date: 2023-11-08T14:34:58Z
Hank wrote:
Is there any reasonable way for us to adjust the WRES code to handle this situation? If necessary, I can start tracking the amount of time between the different machine reboots, in case its just a matter of increasing some sort of retry-cycle to wait longer for Postgres to come back.
Within the core application, I don't think so. Ultimately, database connections can be lost for a variety of reasons and, once retries have been exhausted, the context doesn't matter.
Likewise, if a postgres server returns an error code that could appear in a variety of contexts, some of which might be recoverable from a service-level or higher perspective and others not, then I don't think there's anything we can do in the core app, we need to look higher up.
Of course, we can always propagate error codes and the service layer has complete control over what it retries.
In theory, we could implement a rule above the service level (but respected by the service) that says "any evaluation that fails during this datetime interval should be retried by default". It would be a bit of a tank-at-knife-fight approach, but I don't see much joy in inspecting error codes from postgres etc. because they likely don't clarify machine reboots as distinct from other server problems.
Author Name: Hank (Hank) Original Redmine Issue: 122395, https://vlab.noaa.gov/redmine/issues/122395 Original Date: 2023-11-08
The problem: When a Wednesday reboot sequence occurs, the order of the machines going down and coming up is unpredictable. If the Postgres server for an evaluation goes down before the application servers, then on-going evaluations will fail. If the Postgres server comes back after the application servers, then on-going evaluations that were picked back up will fail. In both cases, its because the Postgres server communication is cutoff.
The WRES should be able to pick back up on-going evaluations if the application servers are brought down before Postgres is brought down, and Postgres is brought back up before the application servers are brought back up. I asked for that sequence to be put in place in ITSG-1250:
https://jira.nws.noaa.gov/servicedesk/customer/portal/1/ITSG-1250
However, ITSG reported,
Is there any reasonable way for us to adjust the WRES code to handle this situation? If necessary, I can start tracking the amount of time between the different machine reboots, in case its just a matter of increasing some sort of retry-cycle to wait longer for Postgres to come back.
This ticket can be resolved once an approach is identified and implemented. This ticket can be rejected if there is really nothing we can do.
There are problem old tickets related to this request; I'll find them and relate them later. Thanks,
Hank