Closed gbenhaim closed 6 years ago
@gbenhaim correct the rpms for 3.10 are not yet built, am planning to work on it this week however it won't be published on official repo in the next 2 week. Once will be available as candidate i'll tag you, sorry for delay
@DanyC97 thanks.
@nellyc
@DanyC97 any updates?
@gbenhaim sorry no, just got back to work. is on my top priority list, just please bear with me a bit.
@gbenhaim : any update or way we could help ?
@ticapix Is there any official repo for 3.10 rc?
@DanyC97 Repo 3.10 for soon or not yet? Thank's
Any news here. Openshift 3.10 has been released by RedHat, however origin is not available
guys i understand the excitement but it doesn't help if everyone is adding a comment on top of another comment.
the reason i haven't released as part of PaaS SIG any origin rpm for v3.10.rc0 is because we couldn't build it due to missing code which was made in OCP but has been ported to Origin but that is now sorted
Once that was done i was waiting for the new tag in Origin repo to be created which was done few hours ago with the official v3.10 release and so i'm working on it to get the rpms out for testing.
in case you haven't seen the notification sent to dev mailing list
Let me know if there are any issues either here or writing to the mailer
The default playbooks/prerequisites.yml
and playbooks/deploy_cluster.yml
are working on the first try on a fresh Centos7 install with the default inventory/hosts.localhost
config.
The only tweak was
pip uninstall ansible || true
yum -y install yum # this will install ansible-2.4
pip install --upgrade ansible # this will install ansible-2.6
Otherwise if using ansible-2.6 installed from pip and without having the package installed from the distrib package manager, the playbook will force the install of 2.4 version via yum and it messes up the script at some point. (error about environment 'shell' var not being defined)
eh ?
to my knowledge 3.10 is working with 2.4, @sdodson am i missing something here ? only 3.11 will require 2.6 ansible
You saying that deploying 3.10 with ansible 2.4 doesn't work ?
I'm saying that if ansible 2.6 is installed via pip before running the playbook it fails. I'll try with the default 2.4 version from yum.
Le ven. 3 août 2018 à 20:19, Dani Comnea notifications@github.com a écrit :
eh ?
to my knowledge 3.10 is working with 2.4, @sdodson https://github.com/sdodson am i missing something here ? only 3.11 will require 2.6 ansible
You saying that deploying 3.10 with ansible 2.4 doesn't work ?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-410336067, or mute the thread https://github.com/notifications/unsubscribe-auth/AC8k_nAGF8vUiHp0ytocGgoUzKHex2ypks5uNJQsgaJpZM4UCqDY .
right which is what i expected @ticapix . As mentioned 2.6 is only for 3.11 which is not released, is work in progress in master(dev) branch.
I just installed but still getting error OS Version: CentOS 7.5
git clone https://github.com/openshift/openshift-ansible.git
cd openshift-ansible && git fetch && git checkout release-3.10 && cd ..
ansible-playbook -i inventory.ini openshift-ansible/playbooks/prerequisites.yml
ansible-playbook -i inventory.ini openshift-ansible/playbooks/deploy_cluster.yml
Following is error
No package matching 'origin-clients-3.10*' found available, installed or updated
@imranrazakhan which yum repo have you used ?
If you look here the rpm is present.
the promotion to the other repos have not taken place due to few people responsible with the CentOS repo traveling.
when promotion will happen i'm going to send another notification.
@imranrazakhan production repos won't work yet.
check dev mailing list for testing RPMs: http://lists.openshift.redhat.com/openshift-archives/dev/2018-August/msg00000.html
Anyone to post how to activate them in ansible?
@imranrazakhan : add the testing repo
cat > /etc/yum.repos.d/CentOS-OpenShift-Origin-CBS.repo <<EOF
[centos-openshift-origin-testing-cbs]
name=CentOS OpenShift Origin Testing CBS
baseurl=https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS
EOF
@ticapix and you saying with the above repo things still don't work ?
@DanyC97 @imranrazakhan For my 3.10 installation I used the following baseurl in the .repo file: https://cbs.centos.org/repos/paas7-openshift-origin310-candidate/x86_64/os/Packages/
The installation of the package origin-node-3.10.0 worked without problems. I used for the installation the Ansible version 2.6.2. and the openshift-ansible repo with the branch release-3.10. Only the openshift-infra deployment failed because of image pull errors of version v3.10, it seems not be build already. Version latest worked for my test without problems.
@DanyC97 I was able to upgrade to version 3.10 with the provided rpms. Is there anything I can do for you to help you to get this RPM into production? @isaldarriaga At the moment it's not possibly to activate this repo with openshift-ansible @ticapix Thanks for your repo definition. I copied this definition via ansible ad hoc command to my cluster nodes
@Reamer thank you kindly for your help, there is nothing major left now that you confirmed. I just sent out another notification with the next steps.
@Reamer FYI i set the repo in all host as per @ticapix instructions and ansible was able to pull the images without problem.
My installation is not working though.
i traced down the error to the api control plane pod:
$ master-logs api $(docker ps | grep api | awk '{print $1}')
json: cannot unmarshal array into Go value of type types.ContainerJSON
the pod's yaml looks fine, but not sure if causing the problem or not:
$ cat /etc/origin/node/pods/apiserver.yaml
https://gist.github.com/isaldarriaga/3bedaf1dfa731485b6783a5f289740d6
docker logs shows nothing.
i couldn't find anything in /var/log for /usr/bin/pod in the api control plane's container either:
$ docker exec -it $(docker ps | grep api | awk '{print $1}') ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1001 1 0.0 0.0 2764 720 ? Ssl Aug07 0:00 /usr/bin/pod
1001 44 2.0 0.0 51708 1652 ? Rs+ 00:57 0:00 ps aux
$ docker exec -it $(docker ps | grep api | awk '{print $1}') ls /var/log
btmp grubby_prune_debug lastlog tallylog wtmp yum.log
how to increase the verbosity in master-logs / docker container? already is 8 in ansible and control plane looks already set to that.
$ cat /etc/origin/master/master.env
https://gist.github.com/isaldarriaga/d9a836ef6700df0b0f43fa5b9c76b82d
This is the docker's config just in case:
$ cat /etc/sysconfig/docker
https://gist.github.com/isaldarriaga/2fcddaea8b53d74279f64236695ef55c
centos version (master node):
$ uname -a
Linux prod-master-01.example.com 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
https://gist.github.com/isaldarriaga/29397ec40caac0aa1888d9f81730ba08
=========================
Fedora version (remote ansible):
$ uname -a
Linux dev-ops 4.17.9-200.fc28.x86_64 #1 SMP Mon Jul 23 21:41:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Ansible version: https://gist.github.com/isaldarriaga/84583361107c0eb38e107ab28e3dc39c
Inventory file: https://gist.github.com/isaldarriaga/4af701c1ed8b4e8a47a752a5e48011ff
ansible log (verbosity = 8) ansible-2018-08-07T18:32:37.zip
edit: adding long output to gist
@isaldarriaga It was very helpful to start the openshift api service manual with openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=${DEBUG_LOGLEVEL:-2}
Then you should be able to look via webconsole into the output log of your failed pods in namespace kube-system
. In my case it was an old vsphere config file /etc/vsphere/vsphere.conf
- custom stuff...
@Reamer i started the api that way.
$ openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
https://gist.github.com/isaldarriaga/0f9e0a536ae6c414951609d9c2d8c877
The $DEBUG_LOGLEVEL is empty by default:
$ env
https://gist.github.com/isaldarriaga/9ff9da4c8e842d2f4aabd6f7dc28093d
$ cat /etc/origin/master/master-config.yaml
https://gist.github.com/isaldarriaga/975f86539c12825de643d4944547d625
openshift start master api
command's last output is :
Invalid MasterConfig /etc/origin/master/master-config.yaml
etcdClientInfo.ca: Invalid value: "/etc/origin/master/master.etcd-ca.crt": could not read file: stat /etc/origin/master/master.etcd-ca.crt: no such file or directory
$ ls -al /etc/origin/master/
https://gist.github.com/isaldarriaga/6bd94aec66473a15e1315a57519b2ded
Since the api didn't come up, i cannot get anything from oc:
$ oc get pods -n kube-system
The connection to the server prod-master-01.example.com:443 was refused - did you specify the right host or port?
This is what i attempted:
a) $ cp /etc/origin/master/ca-bundle.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 06:07:12.757726 25569 start_api.go:68] dial tcp 192.168.246.111:2379: getsockopt: connection refused
the master-01 (.111) attempts to reach itself over :2379
b) $ cp /etc/origin/master/ca.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 06:13:58.080386 26172 start_api.go:68] dial tcp 192.168.246.112:2379: getsockopt: no route to host
the master-01 (.111) attempts to reach master-02 (.112) over :2379
$ iptables --list-rules
https://gist.github.com/isaldarriaga/eba36ac855ce37844c7f0b10eac8615f
c) $ cp /etc/origin/master/client-ca-bundle.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 05:54:39.059864 24454 start_api.go:68] dial tcp 192.168.246.112:2379: getsockopt: no route to host
the master-01 (.111) attempts to reach master-02 (.112) over :2379
d) $ cp /etc/origin/master/frontproxy-ca.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 06:26:58.111176 27244 start_api.go:68] dial tcp 192.168.246.113:2379: getsockopt: no route to host
the master-01 (.111) attempts to reach master-03 (.113) over :2379
so there is a number of problems:
openshift start master api
verbositymaster-logs api $(docker ps | grep api | awk '{print $1}')
command's output does not correspond to
openshift start master api
command's output. maybe caused by 1)/etc/origin/master/master.etcd-ca.crt
should be in your folder. My file is quite old. Generated from openshift-ansible during the installation of 3.7. Take a look into your backup if you can recover this file.
On Port 2379 etcd is running. Is your etcd-cluster up?
so i'm a bit confused @Reamer @isaldarriaga ..
running the Ansible deployment of 3.10 what error you got ?
@DanyC97 This has nothing to do with missing packages. @isaldarriaga Please open a new issue for your problem.
okay @Reamer , thank you for confirmation. But can you confirm if you you managed to get a working/ running 3.10 cluster ?
I can confirm that i have now a running 3.10 cluster with your provided rpms using the openshift-ansible upgrade playbook.
thank you v much for confirmation @Reamer ! @gbenhaim can you please close this issue that that the rpms been provided ? i'll complete the final step on promoting to mirror.centos in the next few hours and it should get in sync by tmw.
@DanyC97, ansible detected one missing rpm (python-docker).
on TASK [Ensure openshift-ansible installer package deps are installed]:
"item": "python-docker",
"msg": "Failure talking to yum: failure: repodata/repomd.xml from centos-openshift-origin-testing-cbs: [Errno 256] No more mirrors to try.\nhttps://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/Packages/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found"
there's no such rpm in any of the 3 referenced openshift repos.
but here: https://rpmfind.net/linux/rpm2html/search.php?query=python-docker-py
I reinstalled having etcd in separate hosts this time.
inventory: https://gist.github.com/isaldarriaga/43ed5c19a32f4b2ed778a7b59952439a
install logs (verbose = 8) ansible.zip
@isaldarriaga the error you showing me is related to missing the python-docker
rpm but then you are showing me in rpmfind.net
the python-docker-py
package .
Now looking in the code i can see this so i suspect it was a typo ?
and for your info @isaldarriaga i just tagged python-docker-3.3.0-1.el7
into testing repo, any chance you can give it another run ?
you're right. i was trying to point out to the existence of the package, but typo.
My lab's hosts is centos7, my client fedora28 (ansible runs here).
Does the code mean if ansible running in fedora, no matter the host's OS, it should install python-docker instead of python3-docker?
now seeing python2-docker-3.3.0-1.el7.noarch.rpm in centos CBS/buildlogs.
thanks, i'm going to retry.
Does the code mean if ansible running in fedora, no matter the host's OS, it should install python-docker instead of python3-docker?
no, what the code does is:
if the host's OS where you are deploying OpenShift (provided by hosts: "{{ l_base_packages_hosts | default('oo_all_hosts') }}"
is:
fedora
=> then python3-docker
is installedpython-docker
is installed ok. Some dependencies are missing when yum -y update
:
Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs) Requires: python-requests >= 2.14.2 Installed: python-requests-2.6.0-1.el7_1.noarch (@base) python-requests = 2.6.0-1.el7_1 Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs) Requires: python-pyOpenSSL You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
I'm connected to https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/
Full output here: https://gist.github.com/isaldarriaga/4203a08fbd5455bc0587c52dfe75205d
It works for me
Thank's
2018-08-09 14:39 GMT+02:00 Ivan Saldarriaga notifications@github.com:
ok. Some dependencies are missing when yum -y update:
Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs) Requires: python-requests >= 2.14.2 Installed: python-requests-2.6.0-1.el7_1.noarch (@base https://github.com/base) python-requests = 2.6.0-1.el7_1 Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs) Requires: python-pyOpenSSL You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
I'm connected to https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/ https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/
Full output here: https://gist.github.com/isaldarriaga/4203a08fbd5455bc0587c52dfe75205d
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-411743000, or mute the thread https://github.com/notifications/unsubscribe-auth/ABuKD-HRYLAPvdfsvty55FvFWYuDZMm-ks5uPC2XgaJpZM4UCqDY .
ok. i'm going to try with a fresh install to confirm.
fyi i just tagged pyOpenSSL-17.3.0-3.el7
& python-requests-2.19.1-3.el7
and it should appear in next 10-15 min in the repo.
Saying that will be good if you can try 2 more fresh installed:
actually ran the installer from fresh install before the new packages to appear in the repo.
failed in the prerequisites.
"item": "python-docker", "msg": "Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs)\n Requires: python-requests >= 2.14.2\n Available: python-requests-2.6.0-1.el7_1.noarch (base)\n python-requests = 2.6.0-1.el7_1\nError: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs)\n Requires: python-pyOpenSSL\n",
looks like they are actually needed.
i'll wait an retry. many thanks.
@DanyC97 so far python2-pyOpenSSL-17.3.0-3.el7.noarch.rpm and python2-requests-2.19.1-3.el7.noarch.rpm appear in CBS, but not in buildlogs.
if i change the repo's baseurl from https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/ to https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/Packages/ and try to yum -y update
i get:
https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/Packages/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
The same error appear when running ansible's prerequisites.yaml too.
will the rpm appear in buildlogs? or better to add repodata folder in CBS?
update:
python2-idna python2-urllib3
required/missing
using: https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/
full output (ansible prerequisites): https://gist.github.com/isaldarriaga/1c3bcc0cb6a470b42d1a48e021b741e3
so far python2-pyOpenSSL-17.3.0-3.el7.noarch.rpm and python2-requests-2.19.1-3.el7.noarch.rpm appear in CBS, but not in buildlogs.
they are now
python2-idna python2-urllib3
will add them soon
@isaldarriaga here we go: python-idna-2.5-1.el7
& python-urllib3-1.16-2.el7
added to the repo, it should appear in next 10-30 min.
any chance you can give it another go and let me know how it goes ? thx
yes, this time I'll add the [lb] (haproxy) to see if more packages are needed or not.
@DanyC97, got a new error
{ "attempts": 3, "changed": false, "invocation": { "module_args": { "allow_downgrade": false, "bugfix": false, "conf_file": null, "disable_gpg_check": false, "disable_plugin": [], "disablerepo": null, "enable_plugin": [], "enablerepo": null, "exclude": null, "install_repoquery": true, "installroot": "/", "list": null, "name": [ "python-docker" ], "security": false, "skip_broken": false, "state": "present", "update_cache": false, "update_only": false, "validate_certs": true } }, "item": "python-docker", "msg": "Error: Package: python2-urllib3-1.16-2.el7.noarch (centos-openshift-origin-testing-cbs)\n Requires: python-pysocks\n", "rc": 1, "results": [ "Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n base: linux.mirrors.es.net\n extras: centos.s.uw.edu\n * updates: mirror.sjc02.svwh.net\nPackage docker-python is obsoleted by python2-docker, trying to install python2-docker-3.3.0-1.el7.noarch instead\nResolving Dependencies\n--> Running transaction check\n---> Package python2-docker.noarch 0:3.3.0-1.el7 will be installed\n--> Processing Dependency: python-websocket-client >= 0.32.0 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-six >= 1.4.0 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-requests >= 2.14.2 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-docker-pycreds >= 0.2.1 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-backports-ssl_match_hostname >= 3.5 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-pyOpenSSL for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-idna for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-cryptography for package: python2-docker-3.3.0-1.el7.noarch\n--> Running transaction check\n---> Package python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 will be installed\n--> Processing Dependency: python-backports for package: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch\n---> Package python-docker-pycreds.noarch 0:1.10.6-4.el7 will be installed\n---> Package python-six.noarch 0:1.9.0-2.el7 will be installed\n---> Package python-websocket-client.noarch 0:0.32.0-116.el7 will be installed\n---> Package python2-cryptography.x86_64 0:1.7.2-2.el7 will be installed\n--> Processing Dependency: python-pyasn1 >= 0.1.8 for package: python2-cryptography-1.7.2-2.el7.x86_64\n--> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.7.2-2.el7.x86_64\n--> Processing Dependency: python-setuptools for package: python2-cryptography-1.7.2-2.el7.x86_64\n--> Processing Dependency: python-enum34 for package: python2-cryptography-1.7.2-2.el7.x86_64\n---> Package python2-idna.noarch 0:2.5-1.el7 will be installed\n---> Package python2-pyOpenSSL.noarch 0:17.3.0-3.el7 will be installed\n---> Package python2-requests.noarch 0:2.19.1-3.el7 will be installed\n--> Processing Dependency: python2-urllib3 for package: python2-requests-2.19.1-3.el7.noarch\n--> Running transaction check\n---> Package python-backports.x86_64 0:1.0-8.el7 will be installed\n---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed\n--> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64\n---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed\n---> Package python-setuptools.noarch 0:0.9.8-7.el7 will be installed\n---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be installed\n---> Package python2-urllib3.noarch 0:1.16-2.el7 will be installed\n--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch\n--> Running transaction check\n---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed\n--> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch\n---> Package python2-urllib3.noarch 0:1.16-2.el7 will be installed\n--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch\n--> Running transaction check\n---> Package python-ply.noarch 0:3.4-11.el7 will be installed\n---> Package python2-urllib3.noarch 0:1.16-2.el7 will be installed\n--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n" ] }
@isaldarriaga sorted, python-pysocks-1.5.6-3.el7
added, give it 10-15 min pls to show up
prerequisites passed. running playbooks/deploy_cluster.yml now.
Description
Failed to install Openshift 3.10 because of missing packages.
Version
Steps To Reproduce
Run
deploy_cluster.yml
playbookExpected Results
Observed Results
Additional Information
Centos 7.5
https://gist.github.com/gbenhaim/1f816581e2b1d3d0db2dd7c6796d3ddd
I also didn't see that origin--3.10 is available in the testing repo: https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin/
I don't understand why installing any RPM packages on the masters/nodes when
containerized=true