Docker Registry ingress going to default backend with POST

[jpds]

Is this a request for help? Yes, though weird behaviour.

What keywords did you search in NGINX Ingress controller issues before filing this one? Docker registry, ingress, found https://github.com/kubernetes/contrib/issues/2444 but suggestions don't help.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

NGINX Ingress controller version: Image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15

Kubernetes version (use kubectl version): 1.8.1

Environment:

• Cloud provider or hardware configuration: Baremetal
• OS (e.g. from /etc/os-release): Debian
• Kernel (e.g. uname -a): 4.12
• Install tools: kubespray

What happened:

I deployed nginx-ingress with the helm chart and am trying to use my docker-registry helm chart from https://github.com/kubernetes/charts/pull/2612 . When I browse to the domain which points to the IP of my ingress node, I can view the [blank] docker registry successfully and it's in the nginx-controller logs:

xxxx:yyyy:zzzz::6 - [xxxx:yyyy:zzzz::6] - - [29/Oct/2017:22:26:05 +0000] "GET / HTTP/1.1" 200 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0" 346 0.001 [default-odd-zorse-docker-registry-80] 10.233.78.89:80 0 0.001 200

And the docker-registry logs:

- - [29/Oct/2017:22:26:05 +0000] "GET / HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0"

However, whenever there's a POST request (with docker push), it goes to the default backend:

$ docker push registry.local/jpds/test-app
The push refers to a repository [registry.local/jpds/test-app]
...
error parsing HTTP 404 response body: invalid character 'd' looking for beginning of value: "default backend - 404"

...
xxxx:yyyy:zzzz::6 - [xxxx:yyyy:zzzz::6] - - [29/Oct/2017:22:27:29 +0000] "POST /v2/jpds/test-app/blobs/uploads/ HTTP/1.1" 404 21 "-" "docker/17.10.0-ce go/go1.9.1 git-commit/f4ffd2511c kernel/4.9.56-1-lts os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \x5C(linux\x5C))" 305 0.002 [upstream-default-backend] 10.233.98.117:8080 21 0.002 404

And of course, the request never makes it to the docker-registry pod. This is the configuration I'm using for my helm chart:

ingress:
  enabled: true
  hosts:
    - registry.local
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/proxy-body-size: "0"
persistence:
  enabled: true
  storageClass: 'rook-block'

[aledbf]

@jpds please post the ingress rules you are using

[jpds]

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-body-size: "0"
    kubernetes.io/ingress.class: nginx
  creationTimestamp: 2017-10-29T21:17:58Z
  generation: 1
  labels:
    app: docker-registry
    chart: docker-registry-0.2.3
    heritage: Tiller
    release: odd-zorse
  name: odd-zorse-docker-registry
  namespace: default
  resourceVersion: "2141598"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/odd-zorse-docker-registry
  uid: a3267feb-bcee-11e7-bcc8-f44d306b566e
spec:
  rules:
  - host: registry.local
    http:
      paths:
      - backend:
          serviceName: odd-zorse-docker-registry
          servicePort: 5000
        path: /

I've also tried to add these to the annotations:

    ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS
    ingress.kubernetes.io/cors-allow-origin: http://registry.local:80/
    ingress.kubernetes.io/enable-cors: "true"

[aledbf]

@jpds did you configure your local docker daemon with the insecure flag to registry.local? Otherwise docker will only use port 443 and reach the default backend (you don't have a TLS section in your ingress rule)

[jpds]

I did indeed configure a insecure repository for the domain (the docker client complains about attempting HTTPS otherwise).

-------- Original-Nachricht -------- An 30. Okt. 2017, 00:16, Manuel Alejandro de Brito Fontes schrieb:

@jpds did you configure your local docker daemon with the insecure flag to registry.local? Otherwise docker will only use port 443 and reach the default backend (you don't have a TLS section in your ingress rule)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[andymaguo]

I meet the same issue, anyone can fix it?

[kbrwn]

@aledbf can you reopen this issue? It is not fixed by your PR.

[morvencao]

@kbrwn +1

[JinsYin]

@kbrwn @morvencao You can try add this in ingress.yaml

spec:
  tls:
  - hosts:
    - registry.local

[morvencao]

@JinsYin Awesome, it works. My complete ingress yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nexus-sonatype-nexus
  labels:
    app: sonatype-nexus
    chart: sonatype-nexus-0.1.7
    release: nexus
    heritage: Tiller
  annotations:
      ingress.kubernetes.io/proxy-body-size: "0"
      kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: docker.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nexus-sonatype-nexus
              servicePort: 5509
    - host: nexus.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nexus-sonatype-nexus
              servicePort: 8081
  tls:
    - hosts:
      - docker.example.com
      secretName: docker-tls

BTW, I set up the docker registry with sonatype-nexus chart: https://github.com/kubernetes/charts/tree/master/stable/sonatype-nexus

[harper1011]

@JinsYin This works fine, but could you share some documentation about why this help?

[gintautassulskus]

when using the insecure registry, does docker work with http-only or does it require both http and https?

[lanmingle]

我刚才也碰到这些问题，我是使用 rancher 2.0 部署的，目前已经解决（添加）：

  tls:
  - hosts:
    - registry.dev-clelo.org

完整例子（编辑或者查看YAML）：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    field.cattle.io/creatorId: user-6dnfc
    field.cattle.io/ingressState: '{"bmV4dXMtaW5ncmVzcy1odHRwL2RlZmF1bHQvbmV4dXMuZGV2LWNsZWxvLm9yZy8vODA4MQ==":"deployment:default:nexus","bmV4dXMtaW5ncmVzcy1odHRwL2RlZmF1bHQvcmVnaXN0cnkuZGV2LWNsZWxvLm9yZy8vODA4Mg==":"deployment:default:nexus"}'
    field.cattle.io/publicEndpoints: '[{"addresses":["192.168.1.250"],"port":80,"protocol":"HTTP","serviceName":"default:ingress-ff975c0a85322a3c706f0a5a4a6fc6bb","ingressName":"default:nexus-ingress-http","hostname":"nexus.dev-clelo.org","allNodes":true},{"addresses":["192.168.1.250"],"port":443,"protocol":"HTTPS","serviceName":"default:ingress-092baed37d35f2a2b5f002b809506728","ingressName":"default:nexus-ingress-http","hostname":"registry.dev-clelo.org","allNodes":true}]'
    ingress.kubernetes.io/proxy-body-size: '0'
  creationTimestamp: 2018-08-04T08:02:11Z
  generation: 5
  name: nexus-ingress-http
  namespace: default
  resourceVersion: "49828"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/nexus-ingress-http
  uid: b1423c6e-97bc-11e8-a2b2-ac220b033a76
spec:
  rules:
  - host: nexus.dev-clelo.org
    http:
      paths:
      - backend:
          serviceName: ingress-ff975c0a85322a3c706f0a5a4a6fc6bb
          servicePort: 8081
  - host: registry.dev-clelo.org
    http:
      paths:
      - backend:
          serviceName: ingress-092baed37d35f2a2b5f002b809506728
          servicePort: 8082
  tls:
  - hosts:
    - registry.dev-clelo.org
status:
  loadBalancer:
    ingress:
    - ip: 192.168.1.250

希望后续的查看这些问题可以快速解决。

[CindyXing0315]

I met the same problem and solved it by adding registry.local:80 to insecure registries, and using registry.local:80 when tag images.

[TheDukeDK]

I also have this problem. But with Artifactory acting as my docker registry. I do not have the option to run it with TLS enabled. Has anyone solved this without enabling TLS?

[ncerquera]

I am having an issue with https://www.edc4it.com/blog/java/minikube-nexus-statefulset-ingress

• when I go to http://nexus.minikube/ I get 503 Service Temporarily Unavailable
• I have completed the steps before with success. That is all the way until they prompt to go to the website.

Is this a common issue?