search

cds-snc / url-shortener

An API written in Python that shortens URLs
MIT License
4 stars 1 forks source link

feat: add Athena queries for WAF access logs #334

Closed patheard closed 1 year ago

patheard commented 1 year ago

Summary

Add the Athena Terraform module which will allow us to easily inspect the WAF ACL access logs.

Related

github-actions[bot] commented 1 year ago

Staging: alarms

✅   Terraform Init: success ✅   Terraform Validate: success ✅   Terraform Format: success ✅   Terraform Plan: success ✅   Conftest: success 

Plan: 7 to add, 0 to change, 0 to destroy
Show summary | CHANGE | NAME | |--------|----------------------------------------------------------------------------| | add | `module.athena_access_logs.aws_athena_database.logs` | | | `module.athena_access_logs.aws_athena_named_query.waf_all_requests[0]` | | | `module.athena_access_logs.aws_athena_named_query.waf_blocked_requests[0]` | | | `module.athena_access_logs.aws_athena_named_query.waf_create_table[0]` | | | `module.athena_access_logs.aws_athena_workgroup.logs` | | | `module.athena_bucket.aws_s3_bucket.this` | | | `module.athena_bucket.aws_s3_bucket_public_access_block.this` |
Show plan ```terraform Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # module.athena_access_logs.aws_athena_database.logs will be created + resource "aws_athena_database" "logs" { + bucket = (known after apply) + force_destroy = true + id = (known after apply) + name = "access_logs" + encryption_configuration { + encryption_option = "SSE_S3" } } # module.athena_access_logs.aws_athena_named_query.waf_all_requests[0] will be created + resource "aws_athena_named_query" "waf_all_requests" { + database = "access_logs" + id = (known after apply) + name = "WAF: all requests" + query = <<-EOT SELECT date_format(from_unixtime(timestamp / 1000e0), '%Y-%m-%d %T') as time, httprequest.httpmethod, httprequest.uri, action, httprequest FROM "access_logs"."waf_logs" WHERE httprequest.uri <> '/healthcheck' ORDER BY time DESC; EOT + workgroup = "logs" } # module.athena_access_logs.aws_athena_named_query.waf_blocked_requests[0] will be created + resource "aws_athena_named_query" "waf_blocked_requests" { + database = "access_logs" + id = (known after apply) + name = "WAF: blocked requests" + query = <<-EOT WITH data AS ( SELECT date_format(from_unixtime(timestamp / 1000e0), '%Y-%m-%d') as time, header.value as host, action, terminatingruleid, httprequest.clientip, httprequest.country, httprequest.httpmethod, httprequest.uri FROM "access_logs"."waf_logs" CROSS JOIN UNNEST(httprequest.headers) AS t(header) WHERE lower(header.name) = 'host' AND action = 'BLOCK' ) SELECT min(time) as start_date, max(time) as end_date, count(*) as num, clientip, country, array_join(array_agg(distinct terminatingruleid), ', ') as terminatingruleid, host, array_join(array_agg(distinct uri), ', ') as uris FROM data GROUP BY clientip, country, host ORDER BY end_date DESC EOT + workgroup = "logs" } # module.athena_access_logs.aws_athena_named_query.waf_create_table[0] will be created + resource "aws_athena_named_query" "waf_create_table" { + database = "access_logs" + id = (known after apply) + name = "WAF: create table" + query = <<-EOT CREATE EXTERNAL TABLE `access_logs.waf_logs`( `timestamp` bigint, `formatversion` int, `webaclid` string, `terminatingruleid` string, `terminatingruletype` string, `action` string, `terminatingrulematchdetails` array< struct< conditiontype:string, location:string, matcheddata:array > >, `httpsourcename` string, `httpsourceid` string, `rulegrouplist` array< struct< rulegroupid:string, terminatingrule:struct< ruleid:string, action:string, rulematchdetails:string >, nonterminatingmatchingrules:array, excludedrules:string > >, `ratebasedrulelist` array< struct< ratebasedruleid:string, limitkey:string, maxrateallowed:int > >, `nonterminatingmatchingrules` array< struct< ruleid:string, action:string > >, `requestheadersinserted` string, `responsecodesent` string, `httprequest` struct< clientip:string, country:string, headers:array< struct< name:string, value:string > >, uri:string, args:string, httpversion:string, httpmethod:string, requestid:string >, `labels` array< struct< name:string > >, `captcharesponse` struct< responsecode:string, solvetimestamp:string, failureReason:string > ) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://cbs-satellite-843973686572/waf_acl_logs/AWSLogs/843973686572/' EOT + workgroup = "logs" } # module.athena_access_logs.aws_athena_workgroup.logs will be created + resource "aws_athena_workgroup" "logs" { + arn = (known after apply) + force_destroy = true + id = (known after apply) + name = "logs" + state = "ENABLED" + tags = { + "CostCentre" = "url-shortener-staging" + "Terraform" = "true" } + tags_all = { + "CostCentre" = "url-shortener-staging" + "Terraform" = "true" } + configuration { + enforce_workgroup_configuration = true + publish_cloudwatch_metrics_enabled = true + requester_pays_enabled = false + result_configuration { + output_location = (known after apply) + encryption_configuration { + encryption_option = "SSE_S3" } } } } # module.athena_bucket.aws_s3_bucket.this will be created + resource "aws_s3_bucket" "this" { + acceleration_status = (known after apply) + acl = "private" + arn = (known after apply) + bucket = "url-shortener-staging-athena-bucket" + bucket_domain_name = (known after apply) + bucket_regional_domain_name = (known after apply) + force_destroy = false + hosted_zone_id = (known after apply) + id = (known after apply) + object_lock_enabled = (known after apply) + policy = (known after apply) + region = (known after apply) + request_payer = (known after apply) + tags = { + "CostCentre" = "url-shortener-staging" + "Critical" = "false" + "Terraform" = "true" } + tags_all = { + "CostCentre" = "url-shortener-staging" + "Critical" = "false" + "Terraform" = "true" } + website_domain = (known after apply) + website_endpoint = (known after apply) + cors_rule { + allowed_headers = (known after apply) + allowed_methods = (known after apply) + allowed_origins = (known after apply) + expose_headers = (known after apply) + max_age_seconds = (known after apply) } + grant { + id = (known after apply) + permissions = (known after apply) + type = (known after apply) + uri = (known after apply) } + lifecycle_rule { + enabled = true + id = "expire-objects-after-7-days" + expiration { + days = 7 + expired_object_delete_marker = false } } + logging { + target_bucket = (known after apply) + target_prefix = (known after apply) } + object_lock_configuration { + object_lock_enabled = (known after apply) + rule { + default_retention { + days = (known after apply) + mode = (known after apply) + years = (known after apply) } } } + replication_configuration { + role = (known after apply) + rules { + delete_marker_replication_status = (known after apply) + id = (known after apply) + prefix = (known after apply) + priority = (known after apply) + status = (known after apply) + destination { + account_id = (known after apply) + bucket = (known after apply) + replica_kms_key_id = (known after apply) + storage_class = (known after apply) + access_control_translation { + owner = (known after apply) } + metrics { + minutes = (known after apply) + status = (known after apply) } + replication_time { + minutes = (known after apply) + status = (known after apply) } } + filter { + prefix = (known after apply) + tags = (known after apply) } + source_selection_criteria { + sse_kms_encrypted_objects { + enabled = (known after apply) } } } } + server_side_encryption_configuration { + rule { + bucket_key_enabled = false + apply_server_side_encryption_by_default { + sse_algorithm = "AES256" } } } + versioning { + enabled = (known after apply) + mfa_delete = (known after apply) } + website { + error_document = (known after apply) + index_document = (known after apply) + redirect_all_requests_to = (known after apply) + routing_rules = (known after apply) } } # module.athena_bucket.aws_s3_bucket_public_access_block.this will be created + resource "aws_s3_bucket_public_access_block" "this" { + block_public_acls = true + block_public_policy = true + bucket = (known after apply) + id = (known after apply) + ignore_public_acls = true + restrict_public_buckets = true } Plan: 7 to add, 0 to change, 0 to destroy. Warning: Argument is deprecated with module.athena_bucket.aws_s3_bucket.this, on .terraform/modules/athena_bucket/S3/main.tf line 8, in resource "aws_s3_bucket" "this": 8: resource "aws_s3_bucket" "this" { Use the aws_s3_bucket_server_side_encryption_configuration resource instead (and 3 more similar warnings elsewhere) ───────────────────────────────────────────────────────────────────────────── Saved the plan to: plan.tfplan To perform exactly these actions, run the following command to apply: terraform apply "plan.tfplan" ```
Show Conftest results ```sh WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.cloudfront_ddos"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.route53_ddos"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.url_shoretener_api_suspicious"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.url_shoretener_api_warning"] WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.url_shortener_api_error"] WARN - plan.json - main - Missing Common Tags: ["module.cloudwatch_alarms_slack.aws_cloudwatch_log_group.notify_slack_lambda"] WARN - plan.json - main - Missing Common Tags: ["module.cloudwatch_alarms_slack.aws_iam_policy.notify_slack_lambda"] WARN - plan.json - main - Missing Common Tags: ["module.cloudwatch_alarms_slack.aws_iam_role.notify_slack_lambda"] WARN - plan.json - main - Missing Common Tags: ["module.cloudwatch_alarms_slack.aws_lambda_function.notify_slack"] WARN - plan.json - main - Cloudwatch log metric pattern is invalid: ["aws_cloudwatch_log_metric_filter.url_shortener_api_error"] WARN - plan.json - main - Cloudwatch log metric pattern is invalid: ["aws_cloudwatch_log_metric_filter.url_shortener_api_warning"] 27 tests, 16 passed, 11 warnings, 0 failures, 0 exceptions ```