CDLUC3 / mrt-doc

Documentation and Information regarding the Merritt repository
8 stars 4 forks source link

Merritt UI: trigger an alert if a 500 error is logged to opensearch. Determine if a restart of a service is needed #2063

Open terrywbrady opened 1 month ago

terrywbrady commented 1 month ago

On 10/14, there were 3700 500 errors in the log for UI02. The errors indicated that the automatic retries were exhausted.

Unfortunately, the retry logic seemed to re-use bad database connections. It seems that the database connections were corrupted.