O+M 2023-12-01

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• [ ] Check Production State/Actions

  Note: Catalog Auto Tasks You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• [ ] DB-Solr Sync
• [ ] Check Harvesting Emails
• [ ] New Relic Alerts Triaged
• [ ] Triage DMARC Report from Google

Weekly Checklist

• [ ] Audit Log (more info on AU-3 and AU-6 Log auditing)
• [ ] Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/GSA/data.gov/issues/4345])
• [ ] Check Catalog Solr
• [ ] Catalog Dupe Check

Monthly Checklist

• [ ] Invicti Scan

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[Jin-Sun-tts]

Monday 11/27

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• [x] DB-Solr Sync:
  0 packages need to be removed from Solr 4123 packages need to be updated/added to Solr 252 packages without harvest_object need to be mannually deleted Finished 988s

• [x] Tracking Update: 2023-11-27T07:38:10.79+0000 [APP/TASK/ckangeodatagovtracking-update-7001939114-1/0] OUT 2023-11-27 07:38:10,790 INFO [ckanext.geodatagov] 5879 package indexes to be rebuilt starting from 2023-11-24 00:00:00

Check Harvesting Emails

• [x] Catalog:

Other

• [x] New Relic Alerts

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

[Jin-Sun-tts]

Will check with Bret for the datagov-brokerpack-solr and datagov-ssb. just ignore them for now.

resource.data.gov: change saas version check and fix the expiration date for existing exceptions.

datagov-harvesting-logic: - still in development phase. ignore

inventory-app : reported from requirements-dev.txt, exclude this file from snyk scan.

[Jin-Sun-tts]

Wednesday 11/29

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• [x] DB-Solr Sync:
  0 packages need to be removed from Solr 2483 packages need to be updated/added to Solr 252 packages without harvest_object need to be mannually deleted Finished 713s

Check Harvesting Emails

• [x] Catalog

Other

• [x] New Relic Alerts

[Jin-Sun-tts]

Thursday 11/30

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• [x] DB-Solr Sync:
  2 packages need to be removed from Solr 1794 packages need to be updated/added to Solr 252 packages without harvest_object need to be mannually deleted Finished 566s

Check Harvesting Emails

• [x] Catalog

Other

• [x] New Relic Alerts

• [x] Snyk vulnerability check : inventory-app: Bumps cryptography from 41.0.5 to 41.0.6. PR: https://github.com/GSA/inventory-app/pull/675

  resource.data.gov: add two exceptions from https://app.snyk.io/org/data.gov/project/22700983-7a0f-49aa-b664-c3f0abc51588 PR: https://github.com/GSA/resources.data.gov/pull/642

[Jin-Sun-tts]

Friday 12/01

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• [x] DB-Solr Sync:
  0 packages need to be removed from Solr 1608 packages need to be updated/added to Solr 252 packages without harvest_object need to be mannually deleted Finished 835s

Check Harvesting Emails

• [x] Catalog:

Other

• [x] New Relic Alerts

Checked catalog, inventory production, works fine. Also checked Solr leader and followers, all work as normal.

Weekly Checklist

• [x] Audit Log (more info on AU-3 and AU-6 Log auditing)
• [x] Tracking Update - Done on Monday 09/25
• [x] Check Catalog Solr
• [x] Catalog Dupe Check

Run de-dup for above organizations.

• [x] Snyk vulnerability check : PR https://github.com/GSA/inventory-app/pull/676

[nickumia]

@Jin-Sun-tts You should close your issue 😛