Determine why lms.views.lti_launches:lti_launches is slow between lms and h services

[hmstepanek]

The request to h.views.api.users:update takes up to 8 seconds to respond, since it does not take this long in h, this must be network time. There is a potential fix in-progress that hopefully will fix this https://github.com/hypothesis/playbook/issues/196. This will need to be verified after the switch. detailed trace in New Relic

More than 3% of all lti launch requests per week experience a launch time > .5s.

[seanh]

Some things we want to do to get to the bottom of this:

• [x] Install New Relic's load balancer integration: https://github.com/hypothesis/playbook/issues/192
• [x] Also install other New Relic Infrastructure integrations, such as EC2 and nginx. These should give us better insight into what's happening
• [ ] If the h application servers themselves are a bottleneck, we may give lms its own separate h app servers or configure nginx to let lms requests through immediately

[dmfine]

New Relic AWS integrations are loaded, we need to make dashboards to make them easy to use. I'll install agents for nginx next.

[hmstepanek]

Unfortunately, I wasn't able to track down a specific lms request in the AWS load balancer logs we captured this week, despite capturing a slow lms request in New Relic. Looking at the logs, it does look like there is a delay in the nginx-gunicorn layer for many requests (some of which I would expect while others I would not) which leads me to believe the occasional slowness of lms requests may be due to nginx rate limiting. Because we really shouldn't be rate limiting internal lms requests made directly to h anyway, I'm going to go ahead and file a ticket to white list lms requests in nginx and close this for now. I'd also like to look into more informative monitoring tools for nginx.

[hmstepanek]

ticket for white listing lms requests in nginx: https://github.com/hypothesis/h/issues/5649 @dmfine @lyzadanger Let's get this added to the queue for next sprint.