Monitoring and logging - Githubissues

department-of-veterans-affairs / vets-website

Frontend for VA.gov

Creative Commons Zero v1.0 Universal

238 stars 123 forks source link

Monitoring and logging #3405

Closed akainic closed 7 years ago

akainic commented 7 years ago

[ ] Alerting when validation failure occurs (maybe not PagerDuty, look more into CloudWatch)

akainic commented 7 years ago

@markolson @lihanli do we have any updates on this?

lihanli commented 7 years ago

devops added sentry to vets-api, alerts still need to be setup though

akainic commented 7 years ago

Things we need to check off here:

[x] Create and implement log management plan
[x] Define log collection and analysis architecture
[x] Test log collection and analysis architecture
[x] Define production versus debugging logging standards
[x] Document method to enable debugging log level on production system
[x] Test enabling different logging levels
[x] Verify correct log messages produced at different logging levels

lihanli commented 7 years ago

devops is adding splunk which will be our log collection and analysis architecture https://github.com/department-of-veterans-affairs/devops/issues/617

the rails logger allows different logging levels. we are outputting all log levels in the production log to have access to all possible information.

logger.debug "Person attributes hash: #{@person.attributes.inspect}"
logger.info "Processing the request..."
logger.fatal "Terminating application, raised unrecoverable error!!!"

Document method to enable debugging log level on production system

Rails.application.configure do
  config.log_level = :debug
end

akainic commented 7 years ago

Follow up with devops and make sure we can access the logs and are getting alerts

akainic commented 7 years ago

@knkski wanted to loop you into this ticket and make sure that we can access the logs for edu-benefits and that we're getting alerts for issues. Do we need to get access to anything in particular for this?

lihanli commented 7 years ago

we have access to the logs through cloudwatch. me and mark both have access and tested it out already. i still need to set up sentry access and test out alerts though

lihanli commented 7 years ago

i'm registered on sentry and subscribed to alerts now

knkski commented 7 years ago

If anybody else needs CloudWatch access, just file a ticket over in the devops repo. As far as alerting, we are sending alerts currently whenever the server gives a 500 error. Will the validation failure alerts that you'd like above be included in that?

akainic commented 7 years ago

Got it, thanks @lihanli and @knkski for that information! I think we need some kind of alerting when a validation failure occurs, but I'm not sure what the right tool is for this. @lihanli do you have thoughts on this?

lihanli commented 7 years ago

i made a pr to add the validation errors to sentry which can send out alerts when they happen https://github.com/department-of-veterans-affairs/vets-api/pull/404

akainic commented 7 years ago

Awesome, thanks @lihanli!

akainic commented 7 years ago

As all of these items are checked off now, going to close this ticket.