A recent (2/18) change to Cloud Storage on Google's end temporarily impacted pushes to our self-hosted registry. Certain layers were consistently failing to upload, corresponding to bursts of HTTP 429 errors showing up in the backend storage logs.
Because this caused a number of related issues, Google rolled back the change rather quickly. Support indicates they are not currently planning to implement a similar change in the near future, but they recommend changing request libraries to use exponential backoff in case of failure, to prevent recurrence on future storage changes:
A recent (2/18) change to Cloud Storage on Google's end temporarily impacted pushes to our self-hosted registry. Certain layers were consistently failing to upload, corresponding to bursts of HTTP 429 errors showing up in the backend storage logs.
Because this caused a number of related issues, Google rolled back the change rather quickly. Support indicates they are not currently planning to implement a similar change in the near future, but they recommend changing request libraries to use exponential backoff in case of failure, to prevent recurrence on future storage changes:
https://cloud.google.com/storage/docs/exponential-backoff
I've not examined the code in detail... perhaps this issue could be considered as a future enhancement.