search

dwmkerr / terraform-aws-openshift

Create infrastructure with Terraform and AWS, install OpenShift. Party!
http://www.dwmkerr.com/get-up-and-running-with-openshift-on-aws
MIT License
170 stars 174 forks source link

CHECK [memory_availability] fail when run ansible playbook on OKD 3.11 #85

Open dearteno opened 5 years ago

dearteno commented 5 years ago

CHECK [memory_availability : ip-10-0-1-238.ap-southeast-1.compute.internal] **** fatal: [ip-10-0-1-238.ap-southeast-1.compute.internal]: FAILED! => {"changed": true, "checks": {"disk_availability": {}, "docker_image_availability": {"changed": true, "failed": true, "failures": [["OpenShiftCheckException", "One or more required container images are not available:\n quay.io/coreos/etcd:v3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=:] docker:///\n"]], "msg": "One or more required container images are not available:\n quay.io/coreos/etcd:v3.2.22\nChecked with: skopeo inspect [--tls-verify=false] [--creds=:] docker:///\n"}, "docker_storage": {"changed": true, "data_pct_used": 0.023671361645169142, "data_threshold": 90.0, "data_total": "34.18 GB", "data_total_bytes": 36700495544.32, "data_used": "19.92 MB", "data_used_bytes": 20887633.92, "metadata_pct_used": 0.00011372055802915023, "metadata_threshold": 90.0, "metadata_total": "88.08 MB", "metadata_total_bytes": 92358574.08, "metadata_used": "57.34 kB", "metadata_used_bytes": 58716.16, "msg": "Thinpool usage is within thresholds.", "vg_free": "48.00g", "vg_free_bytes": 51539607552.0}, "memory_availability": {}, "package_availability": {"changed": false, "invocation": {"module_args": {"packages": ["PyYAML", "bash-completion", "bind", "ceph-common", "dnsmasq", "docker", "firewalld", "flannel", "glusterfs-fuse", "httpd-tools", "iptables", "iptables-services", "iscsi-initiator-utils", "libselinux-python", "nfs-utils", "ntp", "openssl", "origin", "origin-clients", "origin-hyperkube", "origin-node", "pyparted", "python-httplib2", "yum-utils"]}}}, "package_version": {"changed": false, "invocation": {"module_args": {"package_list": [{"check_multi": false, "name": "origin", "version": "3.11"}, {"check_multi": false, "name": "origin-master", "version": "3.11"}, {"check_multi": false, "name": "origin-node", "version": "3.11"}], "package_mgr": "yum"}}}}, "msg": "One or more checks failed", "playbook_context": "install"}



  1. Hosts:    ip-10-0-1-182.ap-southeast-1.compute.internal
     Play:     OpenShift Health Checks
     Task:     Run health checks (install) - EL
     Message:  One or more checks failed
     Details:  check "docker_image_availability":
               One or more required container images are not available:
                   quay.io/coreos/etcd:v3.2.22
               Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>```
dwmkerr commented 5 years ago

Will take a look into this shortly. Any special changes you made to the setup that made it non-standard?

dearteno commented 5 years ago

no, just make infrastructure and make openshift.

dwmkerr commented 5 years ago

Have you tried rerunning make openshift? It can be a little flakey on the ansible side, but is idempotent so safe to run and rerun

On Fri, 5 Apr 2019 at 12:21 PM, Pichate Ins notifications@github.com wrote:

no, just make infrastructure and make openshift.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dwmkerr/terraform-aws-openshift/issues/85#issuecomment-480143501, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1nSATxXHb6rbEk7XbbboA8-GUl-fY0ks5vds9EgaJpZM4cc_F9 .

dearteno commented 5 years ago

i try to run many time. but still got same error

dwmkerr commented 5 years ago

I have completely cleaned and retried locally, everything works:

image

I think you are encountering the below:

https://github.com/openshift/openshift-ansible/issues/11369

I have some other things I'm looking into (see below) but for now I think it is the issue above. Will update this comment as tI debug more. There's not been any activity on that issue for a while - have you tried again recently? It might have been that the image was just unavailable for a period of time...

Potential things to try

  1. try deleting your local .terraform folder (after running terraform destroy) and then re-run terraform init && terraform get so that you get the absolute latest version of the AWS providers.
  2. Let me know which region you are running in, if not the default

Similar Issues

Searching for similar issues, I've found these:

Summarising:

  1. The first suggests the issue is due to transient failures or outages in the redhat container registries it is trying to pull images from. Highly unlikely to be your issue as it is failing consistently.
  2. The second was the wrong registry
  3. The third was proxy settings for a registry

In all cases it is a failure to download a specific image. In your case, the specific image which is failing to download is:

As indicated by:

["OpenShiftCheckException", "One or more required container images are not available:
 quay.io/coreos/etcd:v3.2.22
Checked with: skopeo inspect [--tls-verify=false] [--creds=:] docker:///\n"]]

So right now it looks like an issue with quay.io/coreos/etcd:v3.2.22

dearteno commented 5 years ago

I was testing on ap-southeast-1 got the same issue but the ap-southeast-2 was working smoothly

dwmkerr commented 5 years ago

That is bizarre.... Let me see if I can repro in the same way

ashishxooa commented 5 years ago

I am also getting the same issue, it worked yesterday, I destroyed everything and then when I tried it again, getting the issue even after several retries, deleted the tf state files and tried everything from beginning still getting the same issue.

efess commented 5 years ago

I was getting this as well. The issue may be related to this issue:

https://github.com/openshift/openshift-ansible/issues/10741

The workaround to disable the availability check worked for me and I was able to install this cluster successfully.

FridMehdi commented 5 years ago

@dwmkerr hello any workaround for this issue ? I have the same problem

lejeczek commented 4 years ago

I too see the same with openshift-ansible-3.11.37-1.git.0.3b8b341.el7.noarch on Centos 7. First is: CHECK [memory_availability : .... followed by: Message: One or more checks failed Details: check "docker_image_availability": One or more required container images are not available: docker.io/cockpit/kubernetes:latest, docker.io/openshift/origin-deployer:v3.11, docker.io/openshift/origin-docker-registry:v3.11, docker.io/openshift/origin-haproxy-router:v3.11, docker.io/openshift/origin-pod:v3.11 Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>

check "docker_storage": Could not find imported module support code for docker_info. Looked for either AnsibleDockerClient.py or docker_common.py Traceback (most recent call last): File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/action_plugins/openshift_health_check.py", line 225, in run_check result = check.run() File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/docker_storage.py", line 53, in run docker_info = self.execute_module("docker_info", {}) File "/usr/share/ansible/openshift-ansible/roles/openshift_health_checker/openshift_checks/__init__.py", line 211, in execute_module result = self._execute_module(module_name, module_args, self.tmp, self.task_vars) File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 815, in _execute_module (module_style, shebang, module_data, module_path) = self._configure_module(module_name=module_name, module_args=module_args, task_vars=task_vars) File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 209, in _configure_module **become_kwargs) File "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1258, in modify_module environment=environment) File "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1095, in _find_module_utils py_module_cache, zf) File "/usr/lib/python2.7/site-packages/ansible/executor/module_common.py", line 764, in recursive_finder raise AnsibleError(' '.join(msg)) AnsibleError: Could not find imported module support code for docker_info. Looked for either AnsibleDockerClient.py or docker_common.py I'm on bare-metal and behind a proxy but I put proxy vars in all the places I know of. Commands: docker, skopeo succeed when ran manually.

regards, L.