TI Internal Error Monitoring

[sfradkin]

Story

As an Intermediary engineer, so that I can notify CA about any errors that occur, I need a way to identify error that occur during intermediary processing within the TI service.

Pre-conditions

• [ ] Automated alerting is not mandatory for this story.

Acceptance Criteria

• [ ] If processing of a message fails in TI, that failure can be identified via a query.

Tasks

Research

• [ ] Research work needed to complete the story
• [ ] Foundational: High-level research that will support this and future efforts

Engineering

• [x] Make sure log capture and monitoring is set up the prod environment
• [ ] Add a query to the azure monitor (in all used environments) to view failures
  • [ ] Add terraform to create query in azure
• [x] Clean up log output: remove test log messages at start (e.g. "Error", "Warning", etc)
• [ ] Submit test failure scenarios and make sure the error messages are helpful

Definition of Done

• [ ] Documentation tasks completed
  • [ ] Documentation and diagrams created or updated
  • [ ] Implementation guide (/ig folder)
  • [ ] ADRs (/adr folder)
  • [ ] Main README.md
  • [ ] Other READMEs in the repo
  • [ ] If applicable, update the ReportStream Setup section in README.md
  • [ ] Threat model updated
  • [ ] API documentation updated
• [ ] Code quality tasks completed
  • [ ] Code refactored for clarity and no design/technical debt
  • [ ] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
  • [ ] Code is reviewed or developed by pair; 1 approval is needed but consider requiring an outside-the-pair reviewer
  • [ ] Code quality checks passed
• [ ] Security & Privacy tasks completed
  • [ ] Security & privacy gates passed
• [ ] Testing tasks completed
  • [ ] Load tests passed
  • [ ] Unit test coverage of our code >= 90%
• [ ] Build & Deploy tasks completed
  • [ ] Build process updated
  • [ ] API(s) are versioned
  • [ ] Feature toggles created and/or deleted. Document the feature toggle
  • [ ] Source code is merged to the main branch

Research Questions

• Optional: Any initial questions for research

Decisions

• Optional: Any decisions we've made while working on this story

Notes

• Optional: Any reference material or thoughts we may need for later reference

[luis-pabon-tf]

Tied to #500, might be blocked at the moment. Will update after discussion...

[luis-pabon-tf]

• Research what is needed to get our system hooked to Splunk
• Reach out again to Azure support as seen in #500

[jcrichlake]

Splunk integration shouldn't be required for this card. The Splunk integration is to satisfy a separate CDC log aggregation requirement. For this card we probably just need to add some error based queries to our logs.tf file in TI. There are 2 example queries already in that file, you just have to tinker in the Azure portal to figure out what would be a helpful query or a helpful set of queries for the above mentioned scenario

[basiliskus]

We can use this log query to get the errors in the azure logs:

AppServiceConsoleLogs
| project JsonResult = parse_json(ResultDescription)
| evaluate bag_unpack(JsonResult)
| where level == 'ERROR'

[basiliskus]

People in the team that were tasked with adding the query are still waiting for access to the azure resources needed. I'll look into this ticket today to check if I can add it

[basiliskus]

I added the terraform with the query and tested in the dev environment. Tomorrow will deploy to rest of the environments