search

ModaOperandi / agora

Engineering standards
3 stars 1 forks source link

Proper logging data flow #61

Closed rlmartin closed 3 years ago

rlmartin commented 3 years ago

Current state: some things push directly to New Relic Logs (via a sidecar), some other things (fewer things) push to CloudWatch Logs then get forwarded to NR.

How are we liking New Relic Logs?

Some people would additionally like to get logs into Snowflake for querying.

We should also clean up the log format so it is more easily parseable.

ruxy commented 3 years ago

Q: Is there a way to send from NR to Snowflake?

jackfarzan commented 3 years ago

Docs for EKS platform's NR practices https://modaoperandi.atlassian.net/wiki/spaces/DEVOPS/pages/171999337/New+Relic

ruxy commented 3 years ago

Next steps: try sending logs to snowflake for a few days to see what the cost runs up to

bjmiller commented 3 years ago

We're thinking that we should just keep logs in glacier for now.

bjmiller commented 3 years ago

Also, we need a policy for how long we keep logs.

jackfarzan commented 3 years ago

To start, we should log anything that touches user info (login, payment, order) as well as CDNs. We can put these logs in deep archive in AWS, and restore when needed. We'll store logs going back to a year. Logs to archive:

ruxy commented 3 years ago

This now exists! Last step moving to SDLC.