Pushing to registry error: "Your Authorization Token has expired" ...

[felipejfc]

I use ECR as repo.

Not the first time this happens, normally restarting the registry token refresher solves the problem.

It happened after I restarted the deis-builder pod because it was stuck with a push that never succeeded.

The error is that:

Pushing to registry
{"errorDetail":{"message":"error parsing HTTP 403 response body: invalid character 'Y' looking for beginning of value: \"Your Authorization Token has expired. Please run 'aws ecr get-login' to fetch a new one.\""},"error":"error parsing HTTP 403 response body: invalid character 'Y' looking for beginning of value: \"Your Authorization Token has expired. Premote: n 'aws ecr get-login' to fetch a new one.\""}

Maybe when the builder restarts the registry token refresher needs to detect and update the tokens?

[kmala]

@felipejfc can you give the logs of the registry token refresher as it is supposed to refresh the token regularly. Builder just uses the token so a builder restart shouldn't be an issue here.

[felipejfc]

how regularly? I don't have the logs now because I've deleted the faulty registry pod... If it happens again I'll grab the logs... but I think it is easily reproducible killing the builder, letting deis schedule another and then trying to push and let it try to push an image to ECR

regards

[kmala]

for ecr it is every 11 hours https://github.com/deis/registry-token-refresher/blob/master/pkg/credentials/ecr.go#L84 as the default value is 12hr(http://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_GetAuthorizationToken.html) but it can be configured to a different value by changing the https://github.com/deis/charts/blob/master/workflow-v2.5.0/tpl/generate_params.toml#L168

[felipejfc]

@kmala sometimes im getting this in log:

2016/09/12 16:44:18 updating secret for app some-app W0912 16:44:18.197328 1 request.go:627] Throttling request took 194.59479ms, request: PUT:https://100.64.0.1:443/api/v1/namespaces/some-app/secrets/private-registry-ecr

I do not get the log very well, I don't know if after throttling it completes the update or not... maybe this is the problem?

[kmala]

@felipejfc do you have many apps? because we try to update the secrets in parallel and this might be the issue

[felipejfc]

@kmala It happened on an environment with 13 apps

Also I think this issue should've been opened in registry repo, sorry for that

[felipejfc]

@kmala, what time unit should I use here https://github.com/deis/charts/blob/master/workflow-v2.5.0/tpl/generate_params.toml#L168 ? I think I'll shorten the time until 2.6 release...

[kmala]

@felipejfc its should be in minutes....you can edit the deployment kubectl edit deployment deis-registry-token-refresher --namesapce-deis and change the value.

[kmala]

i couldn't reproduce this, even if the request throttles the secret gets updated.If you face this issue again can you get the logs of the registry-token-refresher.

[felipejfc]

@kmala maybe your tokens from previous generation were still valid when it throttled?

[kmala]

i had checked that new tokens are getting generated and secrets updated with them.But i will fix that throttle issue so that log is neat.

[kmala]

@felipejfc have you faced the issue again?

[felipejfc]

@kmala no, not until now... maybe it has to do with deis upgrades? I think it happened right after an upgrade...

[bacongobbler]

Let's close this ticket until we can get a solid reproduction case. Please feel free to re-open if you can reproduce this issue. Thanks!