acozine
commented
1 month ago Additional issues: allsearch-api-staging1 rails health, recovered ansible-tower1 unreachable, down, NTP error, recovered ansible-tower2 unreachable, recovered catalog-indexer-staging1 orangelight-sneakers deactivated, recovered catalog-staging1 orangelight-sneakers deactivated, recovered catalog-staging2 orangelight-sneakers deactivated, recovered for sneakers, the current threshold is warn at 30 seconds and critical at 1 minute - do we want to change that? cdh-test-prosody1 unreachable, recovered dpul-staging4 dpul-sneakers deactivated, recovered lib-dssproc03 down, recovered lib-dssproc04 unreachable, down lib-entserv DotNet Memory Management, recovered lib-illsql unreachable, recovered lib-papercut DotNet Memory, recovered lib-prdstest unreachable, recovered lib-shipping down, recovered lib-staff927b down again, recovered lib-tickdata unreachable, recovered mysql-db-staging1 / alias video_reserves_staging_db - state Unknown orcid-staging2 unreachable, recovered pdc-discovery-staging2 unreachable tigerdata-staging2 rails health status, recovered
Over the weekend we saw errors on the following systems:
allsearch-api-staging1/2 rails health status critical, recovered ansible-exec-node2 went down cdh-derrida-crawl1 unreachable cdh-test-prosody1 missing monitoring data dpul-collections-staging2 down figgy-db-staging1 down lib-communicate missing monitoring data lib-geoserv1 down, recovered lib-gisportal down lib-postgres-staging1 no monitoring data lib-prdstest unreachable lib-relay01 - unreachable lib-relay02 - unreachable lib-staff997 down lib-stafftst down lib-staff927b unreachable lib-staff 999 down lib-tsserver missing monitoring data lib-vmserv002a-dev down (overloaded?) lib-vmserv01vc also down lrp-prod1 unreachable pdc-describe-prod3 down pdc-discovery-staging1 down pdc-discovery home page prds-staging1 down tigerdata-staging1 missing monitoring data our ec2 check also errored -
p-10-0-1-122.ec2.internalmultiple times