s3server as a backend for docker registry problem

[mikasaari]

Description

When using s3 as a backend for the docker registry with docker swarm enabled (latest docker engine used), the docker push kinda works but it loops the push process infinitely. After the push for a layer is done, it retries a few seconds and then pushes the layer again.

Highly possible I have misconfigured something.

Steps to reproduce the issue

I assume the docker swarm is up and running at this point. Also even in swarm mode, I will use constraint to force the container to run in one node, just to easy the debugging task.

# docker network create --driver overlay --subnet 10.0.3.0/24 testnet

# docker node update --label-add registry=true mds1

# docker service create --name s3test --network testnet --replicas 1 --constraint 'node.labels.registry==true' -e S3BACKEND=mem -e SCALITY_ACCESS_KEY_ID=abcdefghijk -e SCALITY_SECRET_ACCESS_KEY=kjihgfedcba -e ENDPOINT=s3test -e LOG_LEVEL=trace -p 8000:8000 scality/s3server

# aws configure
  AWS Access Key ID [****************hijk]: 
  AWS Secret Access Key [****************dcba]: 
  Default region name [us-east-1]: 
  Default output format [json]: 

# aws s3api --endpoint-url=http://10.50.50.25:8000 create-bucket --bucket registry
{
    "Location": "/registry"
}

# aws s3 --endpoint-url=http://10.50.50.25:8000 ls --recursive s3://
2017-10-25 17:15:28 registry

# docker secret create registry.crt registry.crt

# docker secret create registry.key registry.key

# docker service create --name registry --replicas 1 --network testnet --secret registry.crt --secret registry.key --mount type=bind,src=/mnt/registry/config/registry.yml,dst=/etc/docker/registry/config.yml --constraint 'node.labels.registry==true' -p 5000:5000 registry:2

# docker pull nginx

# docker tag nginx:latest registry:5000/nginx:latest

#  docker push registry:5000/nginx:latest
  a1a53f8d99b5: Retrying in 6 seconds 
  453fc2d51e11: Pushing [=====>                                             ]  5.978MB/53.03MB
  cec7521cdf36: Pushing [=>                                                 ]   1.61MB/55.25MB

Actual result

Docker push, pushes all the layers again and again

Registry debug from stdout (only error part added)

# docker service logs -f registry
registry.1.pse49kc3z0bx@mds1    | time="2017-10-25T14:44:54.485179003Z" level=error msg="unknown error reading request payload: unexpected EOF" environment=staging go.version=go1.7.6 http.request.host="registry:5000" http.request.id=d40253b6-9005-4758-b9ba-0c1cd9c262ed http.request.method=PATCH http.request.remoteaddr="10.255.0.2:23144" http.request.uri="/v2/nginx/blobs/uploads/a8549665-9b17-40cd-a233-f75b54c9eea8?_state=lQXcGQ1G3z8SMJm_FATk_o1LxQ1EFCosxGkDcRELE-l7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiJhODU0OTY2NS05YjE3LTQwY2QtYTIzMy1mNzViNTRjOWVlYTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMTAtMjVUMTQ6NDQ6NTQuMjUyMDEyNjU2WiJ9" http.request.useragent="docker/17.10.0-ce go/go1.8.3 git-commit/f4ffd25 kernel/3.10.0-693.5.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \\(linux\\))" instance.id=9d8759d1-d9d9-4f42-a19f-438db31c1f74 service=registry vars.name=nginx vars.uuid=a8549665-9b17-40cd-a233-f75b54c9eea8 version=v2.6.2 

s3server logs, errors only

s3test.1.v8rp0r916qud@mds1    | {"name":"S3","bucketName":"registry","objectKey":"docker/registry/v2/repositories/nginx/_uploads/84ceb032-eb38-4450-b771-14dd0d60cf85/data","bytesReceived":203,"bodyLength":203,"implName":"memorybucket","err":{"code":404,"description":"The specified key does not exist.","NoSuchKey":true},"time":1508942717985,"req_id":"caae8b47dc4aa549de46","level":"debug","message":"error from metadata","hostname":"32df6a99cdca","pid":101}

Expected result

Docker push would push the layers only once

Additional information: (Node.js version, Docker version, etc)

Docker Name : docker-ce Version : 17.10.0.ce Release : 1.el7.centos

Additional information

registry.yml

version: 0.1
log:
  level: error
  formatter: text
  fields:
    service: registry
    environment: staging
storage:
  s3:
    accesskey: abcdefghijk
    secretkey: kjihgfedcba
    region: us-east-1
    regionendpoint: http://s3test:8000
    bucket: registry
    encrypt: false
    secure: false
    v4auth: true
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
  tls:
    certificate: /run/secrets/registry.crt
    key: /run/secrets/registry.key
health: 
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

[LaurenSpiegel]

Hi @mikasaari, can you please provide more of the s3 logs? The error you are showing is irrelevant. It is just showing the error our in-memory metadata is returning when we check to see if there are acl's on the object you are putting (you would expect a 404 on a new object).

[mikasaari]

Hi. Here is zerobin link to the longer log: https://zerobin.net/?54f2b623d8d6d334#RrfH7svvh/5e3WnubKeIwgyPPM0SnfiPJ3f+YZ5DRH0=

[LaurenSpiegel]

Nothing helpful jumps out to me in the logs. It is odd that there is a NoSuchBucket error after there has already been successful requests handled. As an experiment, instead of using the mem backend, will you try file? Just don't set the "-e S3BACKEND=mem".

[mikasaari]

Hi. I have tried mounting data and metadata directories to LVM volume. All data directories are created but exactly same problems occur. I assume file would be exactly the same, but will test it tomorrow.

[mikasaari]

Hi. Removed the S3BACKEND flag started, created bucket and tried tagging ang pushing to the registry, but unfortunately still exactly the same problem. Will test something else.

[rahulreddy]

Please let us know if the issue still persists.