cds-snc / forms-terraform

Infrastructure as Code for the GC Forms environment
MIT License
16 stars 7 forks source link

Notify Slack undefined log messages #839

Closed patheard closed 2 months ago

patheard commented 2 months ago

Summary

Zitadel is sending log messages to the notify-slack Lambda function that parse as JSON but to not contain the expected fields. The result is a Slack post that says undefined:

image.png

These should be fixed so the error message is properly sent to Slack.

patheard commented 2 months ago

Here's an example of one of the posts: https://gcdigital.slack.com/archives/C05G75MNP4Z/p1726673820091379

The CloudWatch event that triggered this is as follows:

{
    "messageType": "DATA_MESSAGE",
    "owner": "687401027353",
    "logGroup": "/aws/ecs/idp/zitadel",
    "logStream": "task/zitadel/afdf166b7ea5480fb9a481da3437df2d",
    "subscriptionFilters": [
        "error_detection_in_idp_logs"
    ],
    "logEvents": [
        {
            "id": "38506112824212770437109140030114674228494778556846637066",
            "timestamp": 1726673816566,
            "message": "time=\"2024-09-18T15:36:56Z\" level=error msg=\"statement execution failed\" caller=\"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:626\" error=\"sql: transaction has already been committed or rolled back\" projection=projections.users13",
            "extractedFields": {
                "w1": "time=\"2024-09-18T15:36:56Z\" level=error msg=\"statement execution failed\" caller=\"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:626\" error=\"sql: transaction has already been committed or rolled back\" projection=projections.users13"
            }
        }
    ]
}

The problem is that the following message results in the number 3 when it's parsed by JSON.parse:

time=\"2024-09-18T15:36:56Z\" level=error msg=\"statement execution failed\" caller=\"/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:626\" error=\"sql: transaction has already been committed or rolled back\" projection=projections.users13

Fix will be to check if the parsing result is actually an object.

patheard commented 2 months ago

notify-slack Lambda function has been updated in Staging and tested.