Incident Log Sun Dec 04 2016 Edition

[arunasank]

Sun Dec 04 2016

Daily Cloudability cost check!

• Service: cloudability
• Paged:
  • Aruna Sankaranarayanan at 2016-12-04T14:00:22Z
  • Emily McAfee at 2016-12-04T14:00:22Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • cloudability at 2016-12-04T18:00:22Z
• Notes: N/A

• Duration: 240 minutes

  ---

[arunasank]

Sun Dec 04 2016

Daily Cloudability cost check!

• Service: cloudability
• Paged:
  • Aruna Sankaranarayanan at 2016-12-04T14:00:22Z
  • Emily McAfee at 2016-12-04T14:00:22Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • cloudability at 2016-12-04T18:00:22Z
• Notes: N/A

• Duration: 240 minutes

  ---

[arunasank]

Sun Dec 04 2016

Daily Cloudability cost check!

• Service: cloudability
• Paged:
  • Aruna Sankaranarayanan at 2016-12-04T14:00:22Z
  • Emily McAfee at 2016-12-04T14:00:22Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • cloudability at 2016-12-04T18:00:22Z
• Notes: N/A

• Duration: 240 minutes

  ---

ALARM: "api-gl-production-HTTPCodeELB5XXAlarm-3T14QJJXJCZR" in EU - Ireland

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-04T14:38:36Z
• Acknowledged:
  • Ryan Clark at 2016-12-04T14:49:34Z
  • Xiaowei Wang at 2016-12-04T14:49:28Z
• Escalated:
  • pagerduty escalated to Xiaowei Wang at 2016-12-04T14:48:36Z
  • pagerduty escalated to Ryan Clark at 2016-12-04T14:48:36Z
• Resolved:
  • Xiaowei Wang at 2016-12-04T15:01:21Z
• Notes: N/A

• Duration: 22 minutes

  ---

ALARM: "api-gl-production-HTTPCodeELB5XXAlarm-3T14QJJXJCZR" in EU - Ireland

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-04T15:26:37Z
• Acknowledged:
  • Jeff Miccolis at 2016-12-04T15:30:05Z
• Escalated: N/A
• Resolved:
  • Jeff Miccolis at 2016-12-04T15:31:42Z
• Notes: N/A

• Duration: 5 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T16:31:17Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T17:11:13Z
• Notes: N/A

• Duration: 39 minutes

  ---

Search Alert: 0 results found for api-osrm-routed processing finished

• Service: api-osrm-routed
• Paged:
  • Patrick Niklaus at 2016-12-04T17:02:23Z
• Acknowledged:
• Escalated: N/A
• Resolved:
• Notes: N/A

• Duration: 0 minutes

  ---

ALARM: "api-gl-production-HTTPCodeELB5XXAlarm-3T14QJJXJCZR" in EU - Ireland

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-04T17:19:36Z
• Acknowledged:
  • Jeff Miccolis at 2016-12-04T17:20:12Z
• Escalated: N/A
• Resolved:
  • Jeff Miccolis at 2016-12-04T17:36:04Z
• Notes: N/A

• Duration: 16 minutes

  ---

ALARM: "analytics-indexer-production-CPUAlarm-1U66R8N32TTWR" in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-04T19:02:58Z
• Acknowledged:
  • Laurier Rochon at 2016-12-05T00:07:16Z
  • Laurier Rochon at 2016-12-04T23:06:31Z
  • Laurier Rochon at 2016-12-04T23:06:29Z
  • Laurier Rochon at 2016-12-04T19:05:36Z
• Escalated: N/A
• Resolved:
  • Laurier Rochon at 2016-12-05T04:09:02Z
• Notes: N/A

• Duration: 546 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T20:48:40Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T20:49:34Z
• Notes: N/A

• Duration: 0 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T20:50:57Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T20:51:23Z
• Notes: N/A

• Duration: 0 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T20:50:59Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T20:51:23Z
• Notes: N/A

• Duration: 0 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T22:31:25Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • sparky-staging at 2016-12-04T22:41:25Z
• Notes: N/A

• Duration: 10 minutes

  ---

[arunasank]

Sun Dec 04 2016

Daily Cloudability cost check!

• Service: cloudability
• Paged:
  • Aruna Sankaranarayanan at 2016-12-04T14:00:22Z
  • Emily McAfee at 2016-12-04T14:00:22Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • cloudability at 2016-12-04T18:00:22Z
• Notes: N/A

• Duration: 240 minutes

  ---

ALARM: "api-gl-production-HTTPCodeELB5XXAlarm-3T14QJJXJCZR" in EU - Ireland

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-04T14:38:36Z
• Acknowledged:
  • Ryan Clark at 2016-12-04T14:49:34Z
  • Xiaowei Wang at 2016-12-04T14:49:28Z
• Escalated:
  • pagerduty escalated to Xiaowei Wang at 2016-12-04T14:48:36Z
  • pagerduty escalated to Ryan Clark at 2016-12-04T14:48:36Z
• Resolved:
  • Xiaowei Wang at 2016-12-04T15:01:21Z
• Notes: N/A

• Duration: 22 minutes

  ---

ALARM: "api-gl-production-HTTPCodeELB5XXAlarm-3T14QJJXJCZR" in EU - Ireland

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-04T15:26:37Z
• Acknowledged:
  • Jeff Miccolis at 2016-12-04T15:30:05Z
• Escalated: N/A
• Resolved:
  • Jeff Miccolis at 2016-12-04T15:31:42Z
• Notes: N/A

• Duration: 5 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T16:31:17Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T17:11:13Z
• Notes: N/A

• Duration: 39 minutes

  ---

Search Alert: 0 results found for api-osrm-routed processing finished

• Service: api-osrm-routed
• Paged:
  • Patrick Niklaus at 2016-12-04T17:02:23Z
• Acknowledged:
• Escalated: N/A
• Resolved:
• Notes: N/A

• Duration: 0 minutes

  ---

ALARM: "api-gl-production-HTTPCodeELB5XXAlarm-3T14QJJXJCZR" in EU - Ireland

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-04T17:19:36Z
• Acknowledged:
  • Jeff Miccolis at 2016-12-04T17:20:12Z
• Escalated: N/A
• Resolved:
  • Jeff Miccolis at 2016-12-04T17:36:04Z
• Notes: N/A

• Duration: 16 minutes

  ---

ALARM: "analytics-indexer-production-CPUAlarm-1U66R8N32TTWR" in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-04T19:02:58Z
• Acknowledged:
  • Laurier Rochon at 2016-12-05T00:07:16Z
  • Laurier Rochon at 2016-12-04T23:06:31Z
  • Laurier Rochon at 2016-12-04T23:06:29Z
  • Laurier Rochon at 2016-12-04T19:05:36Z
• Escalated: N/A
• Resolved:
  • Laurier Rochon at 2016-12-05T04:09:02Z
• Notes: N/A

• Duration: 546 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T20:48:40Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T20:49:34Z
• Notes: N/A

• Duration: 0 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T20:50:57Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T20:51:23Z
• Notes: N/A

• Duration: 0 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T20:50:59Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-04T20:51:23Z
• Notes: N/A

• Duration: 0 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-04T22:31:25Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • sparky-staging at 2016-12-04T22:41:25Z
• Notes: N/A

• Duration: 10 minutes

  ---

[arunasank]

Mon Dec 05 2016

ALARM: "api-core-production-AlarmHTTPCodeELB5XX-RIEXATOEZCCZ" in US East - N. Virginia

• Service: api-core
• Paged:
  • Jake Pruitt at 2016-12-05T21:16:55Z
  • Ian Ward at 2016-12-05T21:16:55Z
• Acknowledged:
  • Jake Pruitt at 2016-12-05T21:17:11Z
• Escalated: N/A
• Resolved:
  • Jake Pruitt at 2016-12-05T21:47:55Z
• Notes: Jake Pruitt: Seems to be part of the high-traffic white noise we're getting on api-geocoder and many other places.

• Duration: 31 minutes

  ---

ALARM: "api-events-production-AlarmHTTPCodeBackend5XXDuration-9Y55I519UNE4" in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-05T21:18:58Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Mick Thompson at 2016-12-05T21:25:25Z
• Notes: N/A

• Duration: 6 minutes

  ---

ALARM: "analytics-indexer-production-CPUAlarm-1U66R8N32TTWR" in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-05T22:01:00Z
• Acknowledged:
  • Laurier Rochon at 2016-12-05T22:03:35Z
• Escalated: N/A
• Resolved:
  • Laurier Rochon at 2016-12-06T01:30:59Z
• Notes: N/A

• Duration: 209 minutes

  ---

ALARM: "api-maps-staging-LatencyAlarm-1EG0FSNQYSFMW" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-06T02:06:36Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • sparky-staging at 2016-12-06T02:16:36Z
• Notes: N/A

• Duration: 10 minutes

  ---

ALARM: "api-maps-staging-AlarmHTTPCodeBackend5XXDuration-1GF7Q3ZXO4247" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-06T02:13:37Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • sparky-staging at 2016-12-06T02:23:37Z
• Notes: N/A

• Duration: 10 minutes

  ---

ALARM: "HTTPCode_Backend_5XX-api-directions-car-staging-AutoScalingGroup" in US East - N. Virginia

• Service: directions low priority
• Paged:
  • Jeff Miccolis at 2016-12-06T02:41:16Z
  • Karen Zilla Shea at 2016-12-06T02:41:16Z
• Acknowledged:
  • Patrick Niklaus at 2016-12-06T13:38:45Z
  • Jeff Miccolis at 2016-12-06T04:07:11Z
• Escalated: N/A
• Resolved:
  • Jeff Miccolis at 2016-12-06T14:12:17Z
• Notes: N/A

• Duration: 691 minutes

  ---

ALARM: "api-maps-staging-AlarmHTTPCodeBackend5XXDuration-1GF7Q3ZXO4247" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-06T02:49:34Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • sparky-staging at 2016-12-06T02:59:34Z
• Notes: N/A

• Duration: 10 minutes

  ---

ALARM: "api-maps-staging-AlarmHTTPCodeBackend5XXDuration-1GF7Q3ZXO4247" in US East - N. Virginia

• Service: sparky-staging
• Paged:
  • devnull at 2016-12-06T03:04:33Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • sparky-staging at 2016-12-06T03:14:33Z
• Notes: N/A

• Duration: 10 minutes

  ---

ALARM: "analytics-table-production-WriteThrottleAlarm-1CWU5ZQ95OQ0P" in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T03:32:22Z
• Acknowledged:
  • Laurier Rochon at 2016-12-06T04:35:55Z
  • Laurier Rochon at 2016-12-06T03:33:55Z
• Escalated: N/A
• Resolved:
• Notes: N/A

• Duration: 63 minutes

  ---

Search Alert: More than 0 results found for Out Of Memory - api-gl

• Service: api-gl
• Paged:
  • Jeff Miccolis at 2016-12-06T04:05:26Z
• Acknowledged:
  • Jeff Miccolis at 2016-12-06T04:06:32Z
• Escalated: N/A
• Resolved:
  • Jeff Miccolis at 2016-12-06T04:09:12Z
• Notes: N/A

• Duration: 3 minutes

  ---

ALARM: "knightwriter-collector-production-CollectorErrorAlarm-1MBOUPUJR..." in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T04:41:11Z
• Acknowledged:
  • Laurier Rochon at 2016-12-06T05:43:09Z
  • Laurier Rochon at 2016-12-06T04:41:55Z
• Escalated: N/A
• Resolved:
  • Laurier Rochon at 2016-12-06T06:44:01Z
• Notes: N/A

• Duration: 122 minutes

  ---

INSUFFICIENT_DATA: "knightwriter-collector-production-niko-CollectorLam..." in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T08:21:07Z
• Acknowledged:
  • Email at 2016-12-06T12:11:17Z
  • Laurier Rochon at 2016-12-06T08:23:39Z
• Escalated: N/A
• Resolved:
  • Rodolfo Wilhelmy at 2016-12-06T12:43:22Z
• Notes: Rodolfo Wilhelmy: Sorry, Not resolved yet

• Duration: 262 minutes

  ---

INSUFFICIENT_DATA: "knightwriter-collector-production-niko-FirehoseAlar..." in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T08:27:00Z
• Acknowledged:
  • Laurier Rochon at 2016-12-06T08:30:09Z
• Escalated: N/A
• Resolved:
• Notes: N/A

• Duration: 3 minutes

  ---

INSUFFICIENT_DATA: "knightwriter-collector-production-niko-FirehoseDeli..." in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T08:28:09Z
• Acknowledged:
  • Laurier Rochon at 2016-12-06T08:30:04Z
• Escalated: N/A
• Resolved:
• Notes: N/A

• Duration: 1 minutes

  ---

ALARM: "knightwriter-collector-production-niko-FirehoseAlarm-QYIFPLXYCMRK" in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T10:23:58Z
• Acknowledged:
  • Laurier Rochon at 2016-12-06T10:26:42Z
• Escalated: N/A
• Resolved:
• Notes: N/A

• Duration: 2 minutes

  ---

Search Alert: More than 0 results found for OSMCHA RabbitMQ Death

• Service: Data Team
• Paged:
  • Sanjay Bhangar at 2016-12-06T11:33:55Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Email at 2016-12-06T11:34:04Z
• Notes: N/A

• Duration: 0 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: Data Team
• Paged:
  • Sanjay Bhangar at 2016-12-06T12:25:17Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Email at 2016-12-06T12:25:45Z
• Notes: N/A

• Duration: 0 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: Data Team
• Paged:
  • Sanjay Bhangar at 2016-12-06T12:25:55Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Email at 2016-12-06T12:26:58Z
• Notes: N/A

• Duration: 1 minutes

  ---

Auto Scaling: test notification

• Service: Data Team
• Paged:
  • Sanjay Bhangar at 2016-12-06T12:28:23Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Email at 2016-12-06T12:28:38Z
• Notes: N/A

• Duration: 0 minutes

  ---

INSUFFICIENT_DATA: "knightwriter-collector-production-niko-CollectorLam..." in US East - N. Virginia

• Service: api-events
• Paged:
  • Laurier Rochon at 2016-12-06T13:01:07Z
• Acknowledged:
  • Laurier Rochon at 2016-12-06T14:01:46Z
  • Email at 2016-12-06T13:09:03Z
  • Laurier Rochon at 2016-12-06T13:03:55Z
• Escalated:
  • pagerduty escalated to Laurier Rochon at 2016-12-06T13:59:03Z
  • pagerduty escalated to Mick Thompson at 2016-12-06T13:49:03Z
• Resolved:
• Notes: N/A

• Duration: 60 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: Data Team
• Paged:
  • Sanjay Bhangar at 2016-12-06T13:30:10Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Email at 2016-12-06T13:30:35Z
• Notes: N/A

• Duration: 0 minutes

  ---

AWS Notification - Subscription Confirmation

• Service: Data Team
• Paged:
  • Sanjay Bhangar at 2016-12-06T13:38:14Z
• Acknowledged:
  • Email at 2016-12-06T13:46:11Z
• Escalated: N/A
• Resolved:
  • Email at 2016-12-06T13:49:07Z
• Notes: N/A

• Duration: 10 minutes

  ---

Daily Cloudability cost check!

• Service: cloudability
• Paged:
  • Aruna Sankaranarayanan at 2016-12-06T14:00:28Z
  • Emily McAfee at 2016-12-06T14:00:28Z
• Acknowledged:
• Escalated: N/A
• Resolved:
  • Emily McAfee at 2016-12-06T15:42:56Z
• Notes: N/A

• Duration: 102 minutes

  ---