search

BrunoVernay / VM-images-packer

Script VM images creation, focus on proxy support.
MIT License
3 stars 0 forks source link

Pushing image docker-registry.default.svc:5000/... failed #2

Open BrunoVernay opened 7 years ago

BrunoVernay commented 7 years ago

I have a clean install, systemctl show no apparent errors, but when trying to build my first app it fails:

Writing lock file
Generating optimized autoload files
Pushing image docker-registry.default.svc:5000/picture-uploader/cam-cli:latest ...
Warning: Push failed, retrying in 5s ...
...

Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: After retrying 6 times, Push image still failed

And I may have done something really wrong because 

  oc logs dc/docker-registry 
  Error from server: deploymentconfigs.apps.openshift.io "docker-registry" not found
BrunoVernay commented 7 years ago

Some extracts from the ansible.log

TASK [openshift_hosted : Create OpenShift registry] ***************************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "results": {
        "results": [
            {
                "cmd": "/bin/oc create -f /tmp/deploymentconfigIjkLmy -n default", 
                "results": {}, 
                "returncode": 0
            }
        ], 
        "returncode": 0
    }, 
    "state": "present"
}

TASK [openshift_hosted : Ensure OpenShift registry correctly rolls out (best-effort today)] ***********************************************************************************************************************
changed: [192.168.56.10] => {
    "ansible_job_id": "477320030950.33739", 
    "changed": true, 
    "cmd": [
        "oc", 
        "rollout", 
        "status", 
        "deploymentconfig", 
        "docker-registry", 
        "--namespace", 
        "default", 
        "--config", 
        "/etc/origin/master/admin.kubeconfig"
    ], 
    "delta": "0:03:41.817930", 
    "end": "2017-09-02 21:19:24.829289", 
    "failed": false, 
    "failed_when_result": false, 
    "finished": 1, 
    "rc": 0, 
    "start": "2017-09-02 21:15:43.011359"
}

STDOUT:

Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 of 1 updated replicas are available...
Waiting for latest deployment config spec to be observed by the controller loop...
replication controller "docker-registry-1" successfully rolled out

TASK [openshift_hosted : Determine the latest version of the OpenShift registry deployment] ***********************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "cmd": [
        "oc", 
        "get", 
        "deploymentconfig", 
        "docker-registry", 
        "--namespace", 
        "default", 
        "--config", 
        "/etc/origin/master/admin.kubeconfig", 
        "-o", 
        "jsonpath={ .status.latestVersion }"
    ], 
    "delta": "0:00:00.256147", 
    "end": "2017-09-02 21:19:31.771675", 
    "rc": 0, 
    "start": "2017-09-02 21:19:31.515528"
}

STDOUT:

1

TASK [openshift_hosted : Sanity-check that the OpenShift registry rolled out correctly] ***************************************************************************************************************************
changed: [192.168.56.10] => {
    "attempts": 1, 
    "changed": true, 
    "cmd": [
        "oc", 
        "get", 
        "replicationcontroller", 
        "docker-registry-1", 
        "--namespace", 
        "default", 
        "--config", 
        "/etc/origin/master/admin.kubeconfig", 
        "-o", 
        "jsonpath={ .metadata.annotations.openshift\\.io/deployment\\.phase }"
    ], 
    "delta": "0:00:00.262986", 
    "end": "2017-09-02 21:19:32.262025", 
    "failed": false, 
    "failed_when_result": false, 
    "rc": 0, 
    "start": "2017-09-02 21:19:31.999039"
}

STDOUT:

Complete

TASK [openshift_hosted : Get registry DeploymentConfig] ***********************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false, 
    "skip_reason": "Conditional result was False", 
    "skipped": true
}

TASK [openshift_hosted : Wait for registry pods] ******************************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false, 
    "skip_reason": "Conditional result was False", 
    "skipped": true
}

TASK [openshift_hosted : Determine registry fsGroup] **************************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false, 
    "skip_reason": "Conditional result was False", 
    "skipped": true
}

TASK [cockpit-ui : fetch the docker-registry route] ***************************************************************************************************************************************************************
ok: [192.168.56.10] => {
    "changed": false, 
    "results": [
        {
            "apiVersion": "v1", 
            "kind": "Route", 
            "metadata": {
                "creationTimestamp": "2017-09-02T19:15:34Z", 
                "name": "docker-registry", 
                "namespace": "default", 
                "resourceVersion": "1371", 
                "selfLink": "/oapi/v1/namespaces/default/routes/docker-registry", 
                "uid": "181d770c-9013-11e7-ad41-080027491194"
            }, 
            "spec": {
                "host": "docker-registry-default.ocp-2.192.168.56.12.nip.io", 
                "tls": {
                    "termination": "passthrough"
                }, 
                "to": {
                    "kind": "Service", 
                    "name": "docker-registry", 
                    "weight": 100
                }, 
                "wildcardPolicy": "None"
            }, 
            "status": {
                "ingress": [
                    {
                        "conditions": [
                            {
                                "lastTransitionTime": "2017-09-02T19:15:34Z", 
                                "status": "True", 
                                "type": "Admitted"
                            }
                        ], 
                        "host": "docker-registry-default.ocp-2.192.168.56.12.nip.io", 
                        "routerName": "router", 
                        "wildcardPolicy": "None"
                    }
                ]
            }
        }
    ], 
    "state": "list"
}

TASK [cockpit-ui : Create passthrough route for registry-console] *************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "results": {
        "cmd": "/bin/oc get route registry-console -o json -n default", 
        "results": [
            {
                "apiVersion": "v1", 
                "kind": "Route", 
                "metadata": {
                    "annotations": {
                        "openshift.io/host.generated": "true"
                    }, 
                    "creationTimestamp": "2017-09-02T19:19:36Z", 
                    "name": "registry-console", 
                    "namespace": "default", 
                    "resourceVersion": "1492", 
                    "selfLink": "/oapi/v1/namespaces/default/routes/registry-console", 
                    "uid": "a8607999-9013-11e7-ad41-080027491194"
                }, 
                "spec": {
                    "host": "registry-console-default.ocp-2.192.168.56.12.nip.io", 
                    "tls": {
                        "termination": "passthrough"
                    }, 
                    "to": {
                        "kind": "Service", 
                        "name": "registry-console", 
                        "weight": 100
                    }, 
                    "wildcardPolicy": "None"
                }, 
                "status": {
                    "ingress": [
                        {
                            "conditions": [
                                {
                                    "lastTransitionTime": "2017-09-02T19:19:36Z", 
                                    "status": "True", 
                                    "type": "Admitted"
                                }
                            ], 
                            "host": "registry-console-default.ocp-2.192.168.56.12.nip.io", 
                            "routerName": "router", 
                            "wildcardPolicy": "None"
                        }
                    ]
                }
            }
        ], 
        "returncode": 0
    }, 
    "state": "present"
}

....................................

TASK [cockpit-ui : Deploy registry-console] ***********************************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true, 
    "cmd": [
        "oc", 
        "new-app", 
        "--template=registry-console", 
        "-p", 
        "OPENSHIFT_OAUTH_PROVIDER_URL=https://ocp-1.192.168.56.10.nip.io:8443", 
        "-p", 
        "REGISTRY_HOST=docker-registry-default.ocp-2.192.168.56.12.nip.io", 
        "-p", 
        "COCKPIT_KUBE_URL=https://registry-console-default.ocp-2.192.168.56.12.nip.io", 
        "--config=/tmp/openshift-ansible-f8Z6wJ/admin.kubeconfig", 
        "-n", 
        "default"
    ], 
    "delta": "0:00:00.365694", 
    "end": "2017-09-02 21:19:37.538520", 
    "failed": false, 
    "failed_when_result": false, 
    "rc": 0, 
    "start": "2017-09-02 21:19:37.172826"
}

STDOUT:

--> Deploying template "openshift/registry-console" to project default

     registry-console
     ---------
     Template for deploying registry web console. Requires cluster-admin.

     * With parameters:
        * IMAGE_NAME=cockpit/kubernetes
        * IMAGE_VERSION=latest
        * OPENSHIFT_OAUTH_PROVIDER_URL=https://ocp-1.192.168.56.10.nip.io:8443
        * COCKPIT_KUBE_URL=https://registry-console-default.ocp-2.192.168.56.12.nip.io
        * OPENSHIFT_OAUTH_CLIENT_SECRET=user6RlOqTUCr2e8rKyxmvJvVlj5Tn1L0i3hTHA1YxJb0ibCHNdM0b0NSJaJo6LUs3eB # generated
        * OPENSHIFT_OAUTH_CLIENT_ID=cockpit-oauth-client
        * REGISTRY_HOST=docker-registry-default.ocp-2.192.168.56.12.nip.io

--> Creating resources ...
    deploymentconfig "registry-console" created
    service "registry-console" created
    imagestream "registry-console" created
    oauthclient "cockpit-oauth-client" created
--> Success
    Run 'oc status' to view your app.
BrunoVernay commented 7 years ago

/tmp/Ansible.Log only contains very limited info ! I should have redirected the installation output to a file !!! ansible.olg.txt

BrunoVernay commented 7 years ago

Looking at /etc/origin/master/admin.kubeconfig there is nothing about registry, nothing on port 5000.

/etc/containers/registries.d/default.yaml looks like a default file, never edited.

default-docker:
  sigstore-staging: file:///var/lib/atomic/sigstore

Also systemctl status docker-registry ... Unit docker-registry.service could not be found.

I will try to install 'Docker-Distribution' (it is the new name for docker-registry). The Ansible script features lots of docker-registry, but no 'docker-distribution' .

BrunoVernay commented 7 years ago

ansiop.log.txt New test with docker-distribution installed, enabled and started, before installing openshift: The install went fine (complete verbose log joined) Building the app still fails, same message.

> grep -r -i registry /etc/origin/
/etc/origin/master/policy.json:                "name": "system:registry",
/etc/origin/master/policy.json:                "name": "registry-admin",
/etc/origin/master/policy.json:                "name": "registry-editor",
/etc/origin/master/policy.json:                "name": "registry-viewer",

The file /etc/sysconfig/origin-master contains OPENSHIFT_DEFAULT_REGISTRY=docker-registry.default.svc:5000

The URL is OK and accessible from the master (to itself): 

curl -v http://docker-registry.default.svc:5000/
* About to connect() to docker-registry.default.svc port 5000 (#0)
*   Trying 192.168.56.10...
* Connected to docker-registry.default.svc (192.168.56.10) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docker-registry.default.svc:5000
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Sun, 03 Sep 2017 18:53:17 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host docker-registry.default.svc left intact

But on the node, the connexion is refused:

curl -v http://docker-registry.default.svc:5000/
* About to connect() to docker-registry.default.svc port 5000 (#0)
*   Trying 192.168.56.10...
* Connected to docker-registry.default.svc (192.168.56.10) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docker-registry.default.svc:5000
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Sun, 03 Sep 2017 18:53:17 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host docker-registry.default.svc left intact
BrunoVernay commented 7 years ago 
TASK [openshift_hosted : create the default registry service] *****************************************************************************************************************************************************
changed: [192.168.56.10] => {
    "changed": true,
    "results": {
        "clusterip": "172.30.208.115",
        "cmd": "/bin/oc get service docker-registry -o json -n default",
        "results": [
            {
                "apiVersion": "v1",
                "kind": "Service",
                "metadata": {
                    "creationTimestamp": "2017-09-03T17:22:36Z",
                    "name": "docker-registry",
                    "namespace": "default",
                    "resourceVersion": "1289",
                    "selfLink": "/api/v1/namespaces/default/services/docker-registry",
                    "uid": "7aacc3d2-90cc-11e7-af4a-080027491194"
                },
                "spec": {
                    "clusterIP": "172.30.208.115",
                    "ports": [
                        {
                            "name": "5000-tcp",
                            "port": 5000,
                            "protocol": "TCP",
                            "targetPort": 5000
                        }
                    ],
                    "selector": {
                        "docker-registry": "default"
                    },
                    "sessionAffinity": "ClientIP",
                    "type": "ClusterIP"
                },
                "status": {
                    "loadBalancer": {}
                }
            }
        ],
        "returncode": 0
    },
    "state": "present"
}

TASK [openshift_hosted : include] *********************************************************************************************************************************************************************************
included: /usr/share/ansible/openshift-ansible/roles/openshift_hosted/tasks/registry/secure.yml for 192.168.56.10

TASK [openshift_hosted : Set fact docker_registry_route_hostname] *************************************************************************************************************************************************
ok: [192.168.56.10] => {
    "ansible_facts": {
        "docker_registry_route_hostname": "docker-registry-default.ocp-2.192.168.56.12.nip.io"
    },
    "changed": false

Seems the registry is created! There are also certificates and routes created Then 

TASK [openshift_hosted : Sanity-check that the OpenShift registry rolled out correctly] ***************************************************************************************************************************
changed: [192.168.56.10] => {
    "attempts": 1,
    "changed": true,
    "cmd": [
        "oc",
        "get",
        "replicationcontroller",
        "docker-registry-1",
        "--namespace",
        "default",
        "--config",
        "/etc/origin/master/admin.kubeconfig",
        "-o",
        "jsonpath={ .metadata.annotations.openshift\\.io/deployment\\.phase }"
    ],
    "delta": "0:00:00.279206",
    "end": "2017-09-03 19:25:05.051730",
    "failed": false,
    "failed_when_result": false,
    "rc": 0,
    "start": "2017-09-03 19:25:04.772524"
}

STDOUT:

Complete

But it skip others 

TASK [openshift_hosted : Get registry DeploymentConfig] ***********************************************************************************************************************************************************
skipping: [192.168.56.10] => {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}