O+M 2024-10-21 - Githubissues

hkdctol commented 1 week ago

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

[ ] Check Production State/Actions

Note: Catalog Auto Tasks You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.
[ ] Check auto generated O&M tickets from no status column
[ ] Check Harvesting Emails
[ ] New Relic Alerts Triaged
[ ] Triage DMARC Report from Google

Weekly Checklist

[ ] DB-Solr Sync
[ ] Audit Log (more info on AU-3 and AU-6 Log auditing)
[ ] Tracking Update
- NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/GSA/data.gov/issues/4345])
[ ] Check Catalog Solr
[ ] Catalog Dupe Check
[ ] Check user management requests
[ ] Check Snyk Dashboard and cleanup

Monthly Checklist

[ ] Invicti Scan

ad-hoc checklist

[ ] audit/review applications on cloud foundry and determine what can be stopped and/or deleted.

Reference

Watch for user email requests
Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
Monitor and improve Data.gov O&M Dashboard
Update and revise Data.gov O&M Tasks

FuhuXia commented 6 days ago

@hkdctol The weekly harvest source /harvest/fdic-data-json has been failing for the past 10 weeks, the source url https://www.fdic.gov/data.json is giving 404 error. We might need to notify FDIC poc.

FuhuXia commented 4 days ago

Solr leader was restarted last night causing harvesting issues. Manually ran db-solr-sync and tracking-update to ensure all datasets are properly indexed.

Solr leader restart has no impact on public users. It only affects catalog-admin users and some backend activity.

Looking at the past three month logs, solr leader restarted 8 times. Comparing the timing to catalog-admin activity, it seems to be semi-closely related to the spike of API requests to /harvest/object/*, which is mostly originated from geoplatform.com automated script. If we can coordinate with geoplatform.com team and give them a time windows when Solr leader restart has minimal impact on our system, that would be ideal. I am thinking it is 6pm ET. Their script can last up to 5 hours, so it should be finished by 11pm ET, that is the window that we dont expect any activity from admin users or backend harvesting.

btylerburton commented 4 days ago

This is good research here, Fuhu. I like seeing the clear correlation. I don't like that we need to coordinate this; the system should be able to throttle itself.

GSA / data.gov