cds-snc / notification-planning-core

Project planning for GC Notify Core Team

0 stars 0 forks source link

Add common IP/Firewall use case scenarios for Athena queries #204

Open jimleroyer opened 9 months ago

jimleroyer commented 9 months ago

Description

As an ops leads, I want to have handy reusable Athena queries So that I can look into the Firewall data, And I can investigate / resolve support cases quickly.

WHY are we building?

To improve investigation speed and ease of use around ingress traffic of GCNotify.

WHAT are we building?

Adding and saving pre-existing Athena queries with proper names, that can cover a list of use case that developers will come up with. Sharing back these queries to developers to get feedback on.

VALUE created by our solution

Faster resolution of Firewall related investigation.

Acceptance Criteria

[x] List of use case scenarios is collected following a developers' brainstorm.
[x] Queries are added to the Athena list of saved collection.
[x] Share back with developers during a code review meeting.

QA Steps

[ ] Tested by developers during the code review meeting or async afterward, getting stamp of approval.

sastels commented 4 months ago

Reorging Athena and adding queries here: https://github.com/cds-snc/notification-terraform/pull/1183

sastels commented 4 months ago

merged PR that reorged things a bit. Still need to have a meeting to go over what additional queries / use cases we want to have covered.

sastels commented 4 months ago

Going to move this card to blocked until we have the discussion (hopefully the March 5 Dev Review)

ben851 commented 4 months ago

Hopefully will talk about this today at dev review!

ben851 commented 4 months ago

Spoke at the dev review - Steve will implement some suggestions.

sastels commented 4 months ago

Suggestions I noted from meeting:

Queries to add: more fuzzing attack queries, query to see stuff that isn’t being blocked, ie grouped by ip and / or url
Also add a bit to retro template to ask if any queries we should make more permanent

Looking at the list of past queries, ones that stand out:

WAF: Direct IPs hits by hosts, countries, actions and URIs
WAF: monitor blocked requests
WAF: Fuzzing attack

sastels commented 3 months ago

https://github.com/cds-snc/notification-terraform/pull/1192

jimleroyer commented 3 months ago

Merged into staging. Ready to be released. @ben851 to check if these are available in staging.

ben851 commented 3 months ago

I checked the queries and they work - two notes:

Even though you set the query to the notify athena database, it defaulted to "default" to me... not a big deal
The logs are slow to update - the latest in staging is from yesterday. Not sure if that's by design or there's something else I need to do first to update them?

sastels commented 3 months ago

Steve and Ben will investigate!

ben851 commented 3 months ago

I think Ben is just dumb - I took a filter off the test query and the test data came back as expected.