Getting 503 when using S3 as backend, with no hints in logs about reason.

[c4milo]

docker run starts the service just fine, it returns 200 when queried in /v2/ but after a couple of seconds it only returns 503 errors, including requests to /debug/health

docker run -p 5000:5000 -ti -e LOG_LEVEL=debug -e REGISTRY_STORAGE="s3" -e REGISTRY_STORAGE_S3_ACCESSKEY="" -e REGISTRY_STORAGE_S3_SECRETKEY="" -e REGISTRY_STORAGE_S3_REGION="us-east-1" -e REGISTRY_STORAGE_S3_BUCKET="mybucket" -e REGISTRY_STORAGE_S3_ROOTDIRECTORY="/foo/bar/docker-registry"  registry:2

Logs

WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.  go.version=go1.5.2 instance.id=b4f04df2-abda-4b9d-8965-ce00762ec406 version=v2.2.1
INFO[0000] redis not configured                          go.version=go1.5.2 instance.id=b4f04df2-abda-4b9d-8965-ce00762ec406 version=v2.2.1
INFO[0000] listening on [::]:5000                        go.version=go1.5.2 instance.id=b4f04df2-abda-4b9d-8965-ce00762ec406 version=v2.2.1
INFO[0000] Starting upload purge in 57m0s                go.version=go1.5.2 instance.id=b4f04df2-abda-4b9d-8965-ce00762ec406 version=v2.2.1
INFO[0008] response completed                            go.version=go1.5.2 http.request.host=localhost:5000 http.request.id=db630281-e085-4cb0-a102-164917d17bda http.request.method=GET http.request.remoteaddr=172.17.0.1:57190 http.request.uri=/v2/ http.request.useragent=curl/7.29.0 http.response.contenttype=application/json; charset=utf-8 http.response.duration=2.62294ms http.response.status=200 http.response.written=2 instance.id=b4f04df2-abda-4b9d-8965-ce00762ec406 version=v2.2.1
172.17.0.1 - - [18/Dec/2015:21:04:47 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "curl/7.29.0"
INFO[0011] response completed                            go.version=go1.5.2 http.request.host=localhost:5000 http.request.id=bb8e7663-156d-4d9b-bdae-6b15822f31df http.request.method=GET http.request.remoteaddr=172.17.0.1:57193 http.request.uri=/v2/ http.request.useragent=curl/7.29.0 http.response.contenttype=application/json; charset=utf-8 http.response.duration=3.186717ms http.response.status=200 http.response.written=2 instance.id=b4f04df2-abda-4b9d-8965-ce00762ec406 version=v2.2.1
172.17.0.1 - - [18/Dec/2015:21:04:50 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "curl/7.29.0"
172.17.0.1 - - [18/Dec/2015:21:05:14 +0000] "GET /v2/ HTTP/1.1" 503 125 "" "curl/7.29.0"
172.17.0.1 - - [18/Dec/2015:21:05:22 +0000] "GET /debug/health HTTP/1.1" 503 125 "" "curl/7.29.0"
172.17.0.1 - - [18/Dec/2015:21:05:26 +0000] "GET /v2/debug/health HTTP/1.1" 503 125 "" "curl/7.29.0"

[djenriquez]

@c4milo did you leave -e REGISTRY_STORAGE_S3_ACCESSKEY="" -e REGISTRY_STORAGE_S3_SECRETKEY="" blank for the post, or do you really run your command that way?

If its the latter and you are using IAM roles to authenticate, just remove those two environment variables, they cause problems :fire:

[c4milo]

That did it @djenriquez, thank you. The documentation is confusing then, it says those parameters are required.

[dmp42]

cc @BrianBland

[jaychris]

I am having this same issue or at least, the same symptoms - it's been working fine for weeks and now suddenly I get 503's with no clue as to the reason.

EDIT: This is the reason (for others reference):

time="2016-01-04T15:51:53.422028138Z" level=error msg="response completed with error" err.code="MANIFEST_UNKNOWN" err.detail="s3: The difference between the request time and the current time is too large." err.message="manifest unknown" go.version=go1.5.2 http.request.host=dockerregistry.blacksky.com http.request.id=bb5fd136-99de-482c-bbbb-54c4e9810acb http.request.method=GET http.request.remoteaddr="192.168.100.62:57683" http.request.uri="/v2/bsg/mission_control/manifests/0.0.21" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/3.19.0-15-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=24.670284ms http.response.status=404 http.response.written=381 instance.id=d07825d1-442d-4d18-a28c-965e170c8508 service=registry vars.name="bsg/mission_control" vars.reference=0.0.21 version=v2.2.1

[RichardScothern]

@jaychris : I see err.detail="s3: The difference between the request time and the current time is too large." in your logs. Do you have clock skew on your server?

[RichardScothern]

Closing.