ccostino
commented
1 month ago We think we have a couple of solid leads to investigate here that might be at play:
- Throttling happening with S3 that isn't readily apparent (meaning it's presenting as a different issue and AWS is limiting traffic for us at times, hence the inability to retrieve a file from S3 even though it's there
- The fact that cloud.gov platform will restart apps ad hoc because of platform-level updates - these are completely transparent to us, but when an instance of an app restarts, that will wipe out the contents of any app memory, hence the expiring dict losing track of things
- Other possible issues with the expiring dict not working quite right
At the moment, it sounds like the best path forward is exploring why we aren't able to reach S3 properly as the fallback here, in addition to seeing or confirming why the expiring dict is failing like it is, which we surmise is likely because of app instance reboots clearing out memory. If we can iron out the S3 connection, we can create a fallback mechanism to safely rebuild the expiring dict cache by reaching out to S3 and pulling the data it needs upon app initialization (so at the time the app starts back up) - this'll have to be safe and account for the app running in cloud.gov vs. locally.
Additionally, we should engage with our ISSO/ISSM and start having another conversation about longer term ramifications and solutions that involve the handling of PII to try and make this all easier for ourselves. This will take some time though, and the potential path forward above with fixing the S3 and expiring dict handling seems like a very good lead to us.
em-herrick
Who discovered this?
Frank from WA
What happened?
A bug happened! CSV not displaying all phone number data.
Environment
production
What pages did this appear on?
7-day report download
Detail the steps for someone to reproduce
unclear
What browsers are you seeing the problem on?
No response
Relevant log output
No response