Review alarms - Githubissues

cds-snc / platform-forms-client

NextJS application that serves the public-facing website for Forms

https://forms-staging.cdssandbox.xyz/

MIT License

35 stars 13 forks source link

Review alarms #2260

Closed srtalbot closed 1 year ago

srtalbot commented 1 year ago

Acceptance criteria:

[x] Establish severity methodology (S0 - Critical, S1 - High, S2 - Moderate, S3 - Low)
[x] Identify which alarms fall into which categories
[x] Review reliability lambda errors to ensure operations leads understands their meaning.
[x] Threat modeling exercise
[x] Identify new alarms
- [x] Add privilege escalation alarms
[x] Implement new alarms in the relevant channels

Added on July 24th

[x] See how we can add the user email address to ownership and privilege escalation logs (Access granted to clinhfrmp0002yl6629nlz6ld should be Access granted to someone@something.com) - https://github.com/cds-snc/platform-forms-client/pull/2459
[x] Add new privileges logs to the alarm sheet
[x] Add severity tag to alarms so that it appears on Slack when we get the error/warning. This will help decide whether the alarm should be dealt with right away or if we can wait - https://github.com/cds-snc/forms-terraform/pull/451

srtalbot commented 1 year ago

Started a document with the alarm severity definitions - open for comments.

craigzour commented 1 year ago

@srtalbot We have the new channels on Slack where alarms (warnings and errors) are being sent to. The channels are forms-staging-events (currently active) and forms-production-events (still pending production release). Reviewed and added alarms have been referenced in this document . I put an initial severity level but they will probably have to be reassessed.

I will let you decide when you want to close the ticket.

cc @bryan-robitaille

craigzour commented 1 year ago

Added new acceptance criteria to the initial ticket description.

craigzour commented 1 year ago

@srtalbot

Adding severity level (1 and 2) to alarms/warnings we send on Slack is not as straightforward as I initially imagined it. Sometimes what we have individually referenced as error log in the spreadsheet could be coming in pair on Slack. We could end up having two logs with two different severity levels.

If we would really want to have that implemented we will need to rework a little bit the way we mix logs together.

srtalbot commented 1 year ago

Ah, good to know Clement. Thanks for looking into that. How much effort is it to separate the logs? (Approximately in time).

I think it'll be very helpful for the people on call to know what the severity is, so if it's a day or two of work we should do it.

craigzour commented 1 year ago

Ah, good to know Clement. Thanks for looking into that. How much effort is it to separate the logs? (Approximately in time).

I think it'll be very helpful for the people on call to know what the severity is, so if it's a day or two of work we should do it.

It should not take very long. I will look into it today or tomorrow.

craigzour commented 1 year ago

@srtalbot I did prepare two pull requests (see Github issue description) with the additional changes we have talked about. I think we should wait for @bryan-robitaille 's review before merging in staging.

craigzour commented 1 year ago

@srtalbot Going to close this as my work on the new acceptance criteria has been completed. Code was merged to develop and will need to be pushed to production.