Update - We are seeing recovery in Actions start times and are observing for any further impact.
Jul 5, 20:32 UTC
Update - We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are scaling and shifting resources to encourage recovery of the problem.
Jul 5, 19:58 UTC
Update - We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are evaluating mitigations to increase capacity to decrease latency.
Jul 5, 19:19 UTC
Update - We are seeing about 5% of Actions runs not starting within 5 minutes. We are continuing investigation.
Jul 5, 18:40 UTC
Update - We have seen recovery of Actions run delays. Keeping the incident open to monitor for full recovery.
Jul 5, 18:10 UTC
Update - Webhooks is operating normally.
Jul 5, 18:09 UTC
Update - We are seeing delays in Actions runs due to the recovery with webhook deliveries. We expect this to resolve with the recovery of webhooks.
Jul 5, 18:07 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Jul 5, 17:57 UTC
Update - We are seeing recovery as webhooks are being delivered again. We are burning down our queue of events. No events have been lost. New webhook deliveries will be delayed while this process recovers.
Jul 5, 17:55 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Jul 5, 17:42 UTC
Update - We are reverting a configuration change that is suspected to contribute to the problem with webhook deliveries.
Jul 5, 17:20 UTC
Update - Our telemetry shows that most webhooks are failing to be delivered. We are queueing all undelivered webhooks and are working to remediate the problem.
Jul 5, 17:17 UTC
Update - Webhooks is experiencing degraded availability. We are continuing to investigate.
Jul 5, 17:04 UTC
Investigating - We are investigating reports of degraded performance for Webhooks
Jul 5, 20:57 UTC
Resolved - This incident has been resolved.
Jul 5, 20:44 UTC
Update - We are seeing recovery in Actions start times and are observing for any further impact.
Jul 5, 20:32 UTC
Update - We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are scaling and shifting resources to encourage recovery of the problem.
Jul 5, 19:58 UTC
Update - We are still seeing about 5% of Actions runs taking longer than 5 minutes to start. We are evaluating mitigations to increase capacity to decrease latency.
Jul 5, 19:19 UTC
Update - We are seeing about 5% of Actions runs not starting within 5 minutes. We are continuing investigation.
Jul 5, 18:40 UTC
Update - We have seen recovery of Actions run delays. Keeping the incident open to monitor for full recovery.
Jul 5, 18:10 UTC
Update - Webhooks is operating normally.
Jul 5, 18:09 UTC
Update - We are seeing delays in Actions runs due to the recovery with webhook deliveries. We expect this to resolve with the recovery of webhooks.
Jul 5, 18:07 UTC
Update - Actions is experiencing degraded performance. We are continuing to investigate.
Jul 5, 17:57 UTC
Update - We are seeing recovery as webhooks are being delivered again. We are burning down our queue of events. No events have been lost. New webhook deliveries will be delayed while this process recovers.
Jul 5, 17:55 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Jul 5, 17:42 UTC
Update - We are reverting a configuration change that is suspected to contribute to the problem with webhook deliveries.
Jul 5, 17:20 UTC
Update - Our telemetry shows that most webhooks are failing to be delivered. We are queueing all undelivered webhooks and are working to remediate the problem.
Jul 5, 17:17 UTC
Update - Webhooks is experiencing degraded availability. We are continuing to investigate.
Jul 5, 17:04 UTC
Investigating - We are investigating reports of degraded performance for Webhooks
https://www.githubstatus.com/incidents/5yx1d67vq9hg