Noah Stern (VFEP, GCIO) brought an email to our attention that linked the Sentry Error below. We noticed that the server id, gibct-data-service-75f485797-5tp9w, looked a lot like an EKS pod id, despite this service supposedly living outside of EKS. As it turns out, we have pods running in EKS for this service, but they haven't synced in years. Dev/Staging instances are "healthy", but Production seems to have issues.
We are assuming that Noah was correct in that the actual instances serving these requests live in AWS/EC2, and that the ones we see in ArgoCD were never properly setup. Therefore, we're seeing errors pop up every now and then that aren't actually affecting users.
We're requesting someone confirm our theories, and perhaps scale the number of instances from 3 to 0 (or remove altogether) so that we don't get notified of these errors anymore.
-----Original Ticket-----
Submitted By: noah.stern
Topic: Environments
Team: Backend Support
Description:
Noah Stern (VFEP, GCIO) brought an email to our attention that linked the Sentry Error below. We noticed that the server id,
gibct-data-service-75f485797-5tp9w
, looked a lot like an EKS pod id, despite this service supposedly living outside of EKS. As it turns out, we have pods running in EKS for this service, but they haven't synced in years. Dev/Staging instances are "healthy", but Production seems to have issues.We are assuming that Noah was correct in that the actual instances serving these requests live in AWS/EC2, and that the ones we see in ArgoCD were never properly setup. Therefore, we're seeing errors pop up every now and then that aren't actually affecting users.
We're requesting someone confirm our theories, and perhaps scale the number of instances from 3 to 0 (or remove altogether) so that we don't get notified of these errors anymore.
-----Original Ticket-----
Submitted By: noah.stern Topic: Environments Team: Backend Support
Out of space issue on GIBCT.
Please see this sentry http://sentry.vfs.va.gov/organizations/vsp/issues/207861/?project=5