department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 205 forks source link

Out of space issue on GIBCT. Please see this sentry http://... #60085

Open platform-support-slack-integration[bot] opened 1 year ago

platform-support-slack-integration[bot] commented 1 year ago

Description:

Noah Stern (VFEP, GCIO) brought an email to our attention that linked the Sentry Error below. We noticed that the server id, gibct-data-service-75f485797-5tp9w, looked a lot like an EKS pod id, despite this service supposedly living outside of EKS. As it turns out, we have pods running in EKS for this service, but they haven't synced in years. Dev/Staging instances are "healthy", but Production seems to have issues.

We are assuming that Noah was correct in that the actual instances serving these requests live in AWS/EC2, and that the ones we see in ArgoCD were never properly setup. Therefore, we're seeing errors pop up every now and then that aren't actually affecting users.

We're requesting someone confirm our theories, and perhaps scale the number of instances from 3 to 0 (or remove altogether) so that we don't get notified of these errors anymore.

-----Original Ticket-----

Submitted By: noah.stern Topic: Environments Team: Backend Support

Out of space issue on GIBCT.

Please see this sentry http://sentry.vfs.va.gov/organizations/vsp/issues/207861/?project=5

platform-support-slack-integration[bot] commented 1 year ago

Slack Thread Link: https://dsva.slack.com/archives/CBU0KDSB1/p1686351489143999