Resigning script OOMs pods

sastels commented 1 year ago

Describe the bug

the resign_database.py script OOMs the api and celery pods on staging when trying to resign 70K notificaitons.

Bug Severity

See examples in the documentation

SEV-3 in prod

To Reproduce

run the resigning script when there's 70K notifications

Expected behavior

No crashing

Impact

Can't rotate keys without waiting a week for the notifications to be redacted
This could be or become a problem whe we resign api keys (currently 1724 api keys in prod)

Additional context

https://app.zenhub.com/workspaces/notify-planning-core-6411dfb7c95fb80014e0cab0/issues/gh/cds-snc/notification-planning-core/102

sastels commented 1 year ago

PR https://github.com/cds-snc/notification-api/pull/1868

Works locally, needs a good test in staging

jimleroyer commented 1 year ago

@sastels fixed the script to do it in batches and found that 50K was working in staging. To be extra safe, we can size the batch through a command line parameter when executing in production.

jimleroyer commented 1 year ago

@jimleroyer to test with @sastels

ben851 commented 1 year ago

@jimleroyer and @sastels to take a look at this this morning.

ben851 commented 1 year ago

Merged and tested to staging
Going to production on Wednesday

cds-snc / notification-planning-core