microsoft / openshift-container-platform

OpenShift Container Platform on Azure
MIT License
134 stars 195 forks source link

OpenShift 3.9 Deployment fails: the playbook /openshift-node/network_manager.yml could not be found #159

Closed geda2 closed 5 years ago

geda2 commented 5 years ago

Describe the bug We deploy our OpenShift 3.9 cluster using Azure CLI with ARM. Since Feb 28 the deployment is failing.

To Reproduce Deploy cluster with following command: az group deployment create --name <> \ --template-file azuredeploy.json \ --parameters @azuredeploy.parameters.3.9.json \ --resource-group <> \ --no-wait"

Parameters file azuredeploy.parameters.3.9.txt

Expected behavior Cluster will be deployed

stdout

Thu Feb 28 11:30:04 UTC 2019  - Starting Script
Configuring SSH ControlPath to use shorter path name
Thu Feb 28 11:30:04 UTC 2019  - Cloning Ansible playbook repository
Cloning into 'openshift-container-platform-playbooks'...
 - Retrieved playbooks successfully
Thu Feb 28 11:30:09 UTC 2019  - Creating variables file for future playbooks
Thu Feb 28 11:30:09 UTC 2019  - Create variable for master cluster address based on cluster type
Thu Feb 28 11:30:09 UTC 2019  - Creating Master nodes grouping
Thu Feb 28 11:30:09 UTC 2019  - Creating Infra nodes grouping
Thu Feb 28 11:30:09 UTC 2019  - Creating Nodes grouping
Thu Feb 28 11:30:09 UTC 2019  - Creating CNS nodes grouping
Thu Feb 28 11:30:09 UTC 2019  - Create Ansible Hosts file
Thu Feb 28 11:30:09 UTC 2019  - Running DNS Hostname resolution check

PLAY [all] *********************************************************************

TASK [Wait for DNS hostname resolution - will try for up to 33 minutes] ********
ok: [poc-azure-azurerc1-infra-1]
ok: [poc-azure-azurerc1-infra-0]
ok: [poc-azure-azurerc1-master-2]
ok: [poc-azure-azurerc1-master-1]
ok: [poc-azure-azurerc1-master-0]
ok: [poc-azure-azurerc1-node-1]
ok: [poc-azure-azurerc1-node-0]
ok: [poc-azure-azurerc1-node-2]
ok: [poc-azure-azurerc1-cns-0]
ok: [poc-azure-azurerc1-cns-1]
ok: [poc-azure-azurerc1-cns-2]

PLAY RECAP *********************************************************************
poc-azure-azurerc1-cns-0   : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-cns-1   : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-cns-2   : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-infra-0 : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-infra-1 : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-master-0 : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-master-1 : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-master-2 : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-node-0  : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-node-1  : ok=1    changed=0    unreachable=0    failed=0   
poc-azure-azurerc1-node-2  : ok=1    changed=0    unreachable=0    failed=0   

Thu Feb 28 11:30:12 UTC 2019  - Creating glusterfs configuration
Thu Feb 28 11:30:15 UTC 2019  - Create Ansible Hosts file

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-master-0]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-master-0]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-master-1]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-master-1]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-master-2]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-master-2]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-cns-0]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-cns-0]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-cns-1]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-cns-1]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-cns-2]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-cns-2]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-infra-0]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-infra-0]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-infra-1]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-infra-1]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-node-0]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-node-0]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-node-1]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-node-1]

PLAY [all] *********************************************************************

TASK [make sure /etc/origin/cloudprovider exists] ******************************
changed: [poc-azure-azurerc1-node-2]

TASK [populate /etc/origin/cloudprovider/azure.conf] ***************************
changed: [poc-azure-azurerc1-node-2]

PLAY RECAP *********************************************************************
poc-azure-azurerc1-cns-0   : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-cns-1   : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-cns-2   : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-infra-0 : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-infra-1 : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-master-0 : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-master-1 : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-master-2 : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-node-0  : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-node-1  : ok=2    changed=2    unreachable=0    failed=0   
poc-azure-azurerc1-node-2  : ok=2    changed=2    unreachable=0    failed=0   

Thu Feb 28 11:30:37 UTC 2019  - Creation of Cloud Provider Config (azure.conf) completed on all nodes successfully
Thu Feb 28 11:30:37 UTC 2019  - Running NetworkManager playbook

stderr

ERROR! the playbook: /usr/share/ansible/openshift-ansible/playbooks/openshift-node/network_manager.yml could not be found

Template Information (please complete the following information):

geda2 commented 5 years ago

also applies for OpenShift 3.11

geda2 commented 5 years ago

stdout stdout on bastion node showas following message:

Fri Mar 1 08:46:24 UTC 2019  - Installing OpenShift utilities
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-
              : manager
Resolving Dependencies
--> Running transaction check
---> Package openshift-ansible.noarch 0:3.11.82-3.git.0.9718d0a.el7 will be installed
--> Processing Dependency: openshift-ansible-docs = 3.11.82-3.git.0.9718d0a.el7 for package: openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch
--> Processing Dependency: openshift-ansible-playbooks = 3.11.82-3.git.0.9718d0a.el7 for package: openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch
--> Processing Dependency: openshift-ansible-roles = 3.11.82-3.git.0.9718d0a.el7 for package: openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch
--> Processing Dependency: java-1.8.0-openjdk-headless for package: openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch
--> Processing Dependency: patch for package: openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch
--> Processing Dependency: python2-crypto for package: openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch
--> Running transaction check
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.201.b09-0.el7_6 will be installed
--> Processing Dependency: copy-jdk-configs >= 3.3 for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: tzdata-java >= 2015d for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: cups-libs(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: jpackage-utils for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: libjpeg.so.62(LIBJPEG_6.2)(64bit) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: lksctp-tools(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: pcsc-lite-devel(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
--> Processing Dependency: libjpeg.so.62()(64bit) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
---> Package openshift-ansible-docs.noarch 0:3.11.82-3.git.0.9718d0a.el7 will be installed
---> Package openshift-ansible-playbooks.noarch 0:3.11.82-3.git.0.9718d0a.el7 will be installed
---> Package openshift-ansible-roles.noarch 0:3.11.82-3.git.0.9718d0a.el7 will be installed
---> Package patch.x86_64 0:2.7.1-10.el7_5 will be installed
---> Package python2-crypto.x86_64 0:2.6.1-15.el7 will be installed
--> Processing Dependency: libtomcrypt.so.0()(64bit) for package: python2-crypto-2.6.1-15.el7.x86_64
--> Running transaction check
---> Package copy-jdk-configs.noarch 0:3.3-10.el7_5 will be installed
---> Package cups-libs.x86_64 1:1.6.3-35.el7 will be installed
--> Processing Dependency: libavahi-client.so.3()(64bit) for package: 1:cups-libs-1.6.3-35.el7.x86_64
--> Processing Dependency: libavahi-common.so.3()(64bit) for package: 1:cups-libs-1.6.3-35.el7.x86_64
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.201.b09-0.el7_6 will be installed
--> Processing Dependency: pcsc-lite-devel(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
---> Package javapackages-tools.noarch 0:3.4.1-11.el7 will be installed
--> Processing Dependency: python-javapackages = 3.4.1-11.el7 for package: javapackages-tools-3.4.1-11.el7.noarch
---> Package libjpeg-turbo.x86_64 0:1.2.90-6.el7 will be installed
---> Package libtomcrypt.x86_64 0:1.17-26.el7 will be installed
--> Processing Dependency: libtommath >= 0.42.0 for package: libtomcrypt-1.17-26.el7.x86_64
--> Processing Dependency: libtommath.so.0()(64bit) for package: libtomcrypt-1.17-26.el7.x86_64
---> Package lksctp-tools.x86_64 0:1.0.17-2.el7 will be installed
---> Package tzdata-java.noarch 0:2018i-1.el7 will be installed
--> Running transaction check
---> Package avahi-libs.x86_64 0:0.6.31-19.el7 will be installed
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.201.b09-0.el7_6 will be installed
--> Processing Dependency: pcsc-lite-devel(x86-64) for package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64
---> Package libtommath.x86_64 0:0.42.0-6.el7 will be installed
---> Package python-javapackages.noarch 0:3.4.1-11.el7 will be installed
--> Finished Dependency Resolution
Error: Package: 1:java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64 (rhel-7-server-rpms)
           Requires: pcsc-lite-devel(x86-64)
**********************************************************************
yum can be configured to try to resolve such errors by temporarily enabling
disabled repos and searching for missing dependencies.
To enable this functionality please set 'notify_only=0' in /etc/yum/pluginconf.d/search-disabled-repos.conf
**********************************************************************

 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

problem: java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6 now requires "pcsc-lite-devel(x86-64)"

since following bugfix: https://access.redhat.com/errata/RHSA-2019:0435

fix: https://access.redhat.com/solutions/3949501 At the current time, to install java-1.8.0-openjdk-headless-1.8.0.201.b09-1.el6_10.x86_64 or java-1.8.0-openjdk-headless-1.8.0.201.b09-0.el7_6.x86_64 requires enabling the optional repository for the pcsc-lite-devel(x86-64) dependency

subscription-manager repos --enable=rhel-7-server-optional-rpms

haroldwongms commented 5 years ago

Sorry for the delay. I have been working on making a major update to the templates and scripts. I have already incorporated the fix from @geda10 into the latest version of the templates and have opened a PR with a major revision. Once this gets reviewed and we merge, all should be good.

haroldwongms commented 5 years ago

Fixes now merged into master. Please try again and let me know if you still have issues.

geda2 commented 5 years ago

it's working now

haroldwongms commented 5 years ago

Thank you. Closing this issue now.