Closed rahul334481 closed 7 years ago
Hey @rahul334481, could you check the current master serving certificate to ensure that it was signed by the current CA? Also I'm curious if there were any issues encountered with install that would have resulted in these getting out of sync with one another OR if this issue is reproducible starting with fresh VMs?
openssl verify -CAfile /etc/origin/master/ca.crt /etc/origin/master/master.server.crt
openssl verify -CAfile /etc/origin/master/ca-bundle.crt /etc/origin/master/master.server.crt
Hi @abutcher Thanks for the update. I checked and looks good:
[root@oc-master1 ~]# openssl verify -CAfile /etc/origin/master/ca.crt /etc/origin/master/master.server.crt
/etc/origin/master/master.server.crt: OK
[root@oc-master1 ~]# openssl verify -CAfile /etc/origin/master/ca-bundle.crt /etc/origin/master/master.server.crt
/etc/origin/master/master.server.crt: OK
Current oc status has below error:
[root@oc-master1 ~]# oc status
Error from server: Get https://oc-master.domain.com:8443/api/v1/namespaces/default: x509: certificate signed by unknown authority
Thanks, Rahul
Hmmm. Let's check the kubeconfig then. Here are some ugly one liners we can use to compare the md5sums of the base 64 encoded CA data.
/root/.kube/config
(default for cli operations) and /etc/origin/master/admin.kubeconfig
should match.
grep certificate-authority-data /etc/origin/master/admin.kubeconfig | awk '{ print $2 }' | base64 -d | md5sum
grep certificate-authority-data /root/.kube/config | awk '{ print $2 }' | base64 -d | md5sum
md5sum /etc/origin/master/ca-bundle.crt
Or to just examine the CA data in the kubeconfig, which we could then compare with the current bundle:
grep certificate-authority-data /etc/origin/master/admin.kubeconfig | awk '{ print $2 }' | base64 -d | openssl x509 -noout -text
Hi @abutcher
There are diff in /root/.kube/config
and /etc/origin/master/admin.kubeconfig
root@oc-master1 ~]# diff /root/.kube/config /etc/origin/master/admin.kubeconfig
5,6c5,6
< server: https://10.1.15.26:8443
< name: 10-1-15-26:8443
---
> server: https://oc-master.domain.com:8443
> name: oc-master-domain-com:8443
9c9
< cluster: 10-1-15-26:8443
---
> cluster: oc-master-domain-com:8443
11,13c11,13
< user: system:admin/10-1-15-26:8443
< name: default/10-1-15-26:8443/system:admin
< current-context: default/10-1-15-26:8443/system:admin
---
> user: system:admin/oc-master-domain-com:8443
> name: default/oc-master-domain-com:8443/system:admin
> current-context: default/oc-master-domain-com:8443/system:admin
17c17
< - name: system:admin/10-1-15-26:8443
---
> - name: system:admin/oc-master-domain-com:8443
Please find output of next 3 commands:
[root@oc-master1 ~]# grep certificate-authority-data /etc/origin/master/admin.kubeconfig | awk '{ print $2 }' | base64 -d | md5sum
fb33bfae97375e964fc5c75aec16a894 -
[root@oc-master1 ~]# grep certificate-authority-data /root/.kube/config | awk '{ print $2 }' | base64 -d | md5sum
fb33bfae97375e964fc5c75aec16a894 -
[root@oc-master1 ~]# md5sum /etc/origin/master/ca-bundle.crt
fb33bfae97375e964fc5c75aec16a894 /etc/origin/master/ca-bundle.crt
Besides, do we need to enter this in ansible file?
openshift_master_default_subdomain=apps.oc-master.domain.com
Thanks, Rahul
Okay, all of the CA data in the kubeconfigs match and we also know that the master's serving certificate was signed by that CA. We could also ensure that the cert being served by the master checks out:
openssl s_client -CAfile /etc/origin/master/ca-bundle.crt -connect oc-master.domain.com:8443
What happens if we specify the admin kubeconfig rather than using the default, since the root kubeconfig is pointing to an IP address. I doubt this will work since the installer operation would have used the admin kubeconfig for importing imagesteams.
oc status --config=/etc/origin/master/admin.kubeconfig
openshift_master_default_subdomain
sets the default subdomain used for created routes and wouldn't interfere with what we're verifying at the moment.
unsuccessful:
[root@oc-master1 ~]# openssl s_client -CAfile /etc/origin/master/ca-bundle.crt -connect oc-master.domain.com:8443
socket: Connection refused
connect:errno=111
[root@oc-master1 ~]# oc status --config=/etc/origin/master/admin.kubeconfig
Unable to connect to the server: x509: certificate signed by unknown authority
[root@oc-master1 ~]#
Please disregard last output. Getting new one
This command took time and once i hit enter, got prompt with 400 Bad Requestclosed
[root@oc-master1 ~]# openssl s_client -CAfile /etc/origin/master/ca-bundle.crt -connect oc-master.domain.com:8443
CONNECTED(00000003)
depth=1 CN = openshift-signer@1490641060
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/CN=10.1.15.26
i:/CN=openshift-signer@1490641060
1 s:/CN=openshift-signer@1490641060
i:/CN=openshift-signer@1490641060
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEozCCA4ugAwIBAgIBDTANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
...
...
TsW1HNdBavvjPEBL2HMzMznJiyD1ejWW1I3d8lWnJsah/aHR6QdjcSlY4NWn/aFI
ryyGE9nzlCH7phzpbu7JnN2eesbp9+iIzymWeEM/kQizXyKQE78C
-----END CERTIFICATE-----
subject=/CN=10.1.15.26
issuer=/CN=openshift-signer@1490641060
---
Acceptable client certificate CA names
/CN=openshift-signer@1490641060
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 2620 bytes and written 385 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: DB825253ECF48162B8EF6CBE8A084C0F5A196DD1E547AE71647A5F6EAABA385D
Session-ID-ctx:
Master-Key: 92B1BC67690E21D0DFCC88FCA8B7EFB1358C2C60E3062C10B3759A232D2226A3B4C96CD28748669E6A5721104E6DCA7D
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket:
0000 - ae eb fc 4e fb df 51 a1-19 5c cc 70 a1 4b a9 f6 ...N..Q..\.p.K..
0010 - 7c 3d 00 98 dc d1 80 64-45 66 86 f3 18 82 a5 33 |=.....dEf.....3
0020 - 66 c7 78 47 92 9c f8 65-f8 c2 34 27 a9 e0 cb 21 f.xG...e..4'...!
0030 - f0 47 6f 18 f4 8b fc 11-50 51 aa 06 2b e2 95 bb .Go.....PQ..+...
0040 - 7c c5 ba 74 cc 79 d2 3f-0c 86 4d ff 40 ad 5a ac |..t.y.?..M.@.Z.
0050 - 60 c5 ea 4f 39 5c 1b c1-56 24 5a 54 d7 34 95 ee `..O9\..V$ZT.4..
0060 - af 3d ef 6c d8 89 6c 38-8c 81 5e e8 16 a1 d6 f6 .=.l..l8..^.....
0070 - a1 92 08 1f 26 db 48 66- ....&.Hf
Start Time: 1490732648
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
HTTP/1.1 400 Bad Request
Content-Type: text/plain
Connection: close
400 Bad Requestclosed
[root@oc-master1 ~]#
[root@oc-master1 ~]# oc status --config=/etc/origin/master/admin.kubeconfig
Unable to connect to the server: x509: certificate signed by unknown authority
So if I wait for 300s (timeout), below is what i get:
Start Time: 1490733126
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
read:errno=0
[root@oc-master1 ~]#
We would expect Verify return code: 0 (ok)
from the previous section. Compare the issuer of the CA certificate with the issuer of the certificate returned from s_client
. Is it issuer= /CN=openshift-signer@1490641060
?
openssl x509 -noout -issuer -in /etc/origin/master/ca.crt
We should also verify that the CAs match on each master and that the master serving certificate per master can be verified with the CA as we checked before with openssl verify
.
Hi @abutcher,
Please find details as below:
==oc-master== <<<<< LB
Error opening Certificate /etc/origin/master/ca.crt
140711121414048:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/origin/master/ca.crt','r')
140711121414048:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
unable to load certificate
==oc-master1==
issuer= /CN=openshift-signer@1490641040
==oc-master2==
issuer= /CN=openshift-signer@1490641060
==oc-master3==
issuer= /CN=openshift-signer@1490641092
Verification:
==oc-master.domain.com==
Error loading file /etc/origin/master/ca.crt
140005027325856:error:02001002:system library:fopen:No such file or directory:bss_file.c:169:fopen('/etc/origin/master/ca.crt','r')
140005027325856:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:172:
140005027325856:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:281:
usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-attime timestamp] [-engine e] cert1 cert2 ...
recognized usages:
sslclient SSL client
sslserver SSL server
nssslserver Netscape SSL server
smimesign S/MIME signing
smimeencrypt S/MIME encryption
crlsign CRL signing
any Any Purpose
ocsphelper OCSP helper
timestampsign Time Stamp signing
==oc-master1.domain.com==
/etc/origin/master/master.server.crt: OK
==oc-master2.domain.com==
/etc/origin/master/master.server.crt: OK
==oc-master3.domain.com==
/etc/origin/master/master.server.crt: OK
==oc-master.domain.com==
Error loading file /etc/origin/master/ca-bundle.crt
140169011611552:error:02001002:system library:fopen:No such file or directory:bss_file.c:169:fopen('/etc/origin/master/ca-bundle.crt','r')
140169011611552:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:172:
140169011611552:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:281:
usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-attime timestamp] [-engine e] cert1 cert2 ...
recognized usages:
sslclient SSL client
sslserver SSL server
nssslserver Netscape SSL server
smimesign S/MIME signing
smimeencrypt S/MIME encryption
crlsign CRL signing
any Any Purpose
ocsphelper OCSP helper
timestampsign Time Stamp signing
==oc-master1.domain.com==
/etc/origin/master/master.server.crt: OK
==oc-master2.domain.com==
/etc/origin/master/master.server.crt: OK
==oc-master3.domain.com==
/etc/origin/master/master.server.crt: OK
Hi @abutcher / Team,
Any insight is appreciated.
Current status:
==oc-master== <<<<<< lb
bash: oc: command not found
==oc-master1==
Error from server: Get https://oc-master.domain.com:8443/api/v1/namespaces/default: x509: certificate signed by unknown authority
==oc-master2==
Error from server: Get https://oc-master.domain.com:8443/api/v1/namespaces/default: x509: certificate signed by unknown authority
==oc-master3==
Error from server: Get https://oc-master.domain.com:8443/api/v1/namespaces/default: x509: certificate signed by unknown authority
Thanks, Rahul
Hi @abutcher,
Current error once I ran ansible again:
TASK [openshift_hosted_templates : Create or update hosted templates] **********
fatal: [oc-master1]: FAILED! => {"changed": false, "cmd": ["oc", "create", "-f", "/usr/share/openshift/hosted", "--config=/tmp/openshift-ansible-ul8RA0/admin.kubeconfig", "-n", "openshift"], "delta": "0:00:00.206521", "end": "2017-03-29 10:58:55.697675", "failed": true, "failed_when_result": true, "rc": 1, "start": "2017-03-29 10:58:55.491154", "stderr": "Unable to connect to the server: x509: certificate signed by unknown authority", "stdout": "", "stdout_lines": [], "warnings": []}
to retry, use: --limit @/root/openshift-ansible/playbooks/byo/config.retry
PLAY RECAP *********************************************************************
localhost : ok=10 changed=0 unreachable=0 failed=0
oc-etcd1 : ok=90 changed=1 unreachable=0 failed=0
oc-etcd2 : ok=82 changed=1 unreachable=0 failed=0
oc-etcd3 : ok=82 changed=1 unreachable=0 failed=0
oc-master : ok=70 changed=0 unreachable=0 failed=0
oc-master1 : ok=314 changed=16 unreachable=0 failed=1
oc-master2 : ok=245 changed=10 unreachable=0 failed=0
oc-master3 : ok=245 changed=10 unreachable=0 failed=0
oc-node1 : ok=112 changed=2 unreachable=0 failed=1
oc-node2 : ok=112 changed=2 unreachable=0 failed=1
Hey @rahul334481, based on the CNs of those CA certificates it looks like there is a different CA certificate on each master. If this is the case then each master's serving certificate may have been signed by each individual master's CA certificate (rather than all certificates being signed by a single, common CA certificate), meaning that none of the masters can talk to one another. I'm really curious how this could have occurred.
The CA certificate on each master is identical when I configure an HA cluster using the master
branch.
[root@master1 ~]# openssl x509 -noout -issuer -in /etc/origin/master/ca.crt
issuer= /CN=openshift-signer@1490734436
[root@master1 ~]# md5sum /etc/origin/master/ca.crt
b9692b2d6f948547d70e7913c1e166aa /etc/origin/master/ca.crt
[root@master2 ~]# openssl x509 -noout -issuer -in /etc/origin/master/ca.crt
issuer= /CN=openshift-signer@1490734436
[root@master2 ~]# md5sum /etc/origin/master/ca.crt
b9692b2d6f948547d70e7913c1e166aa /etc/origin/master/ca.crt
[root@master3 ~]# openssl x509 -noout -issuer -in /etc/origin/master/ca.crt
issuer= /CN=openshift-signer@1490734436
[root@master3 ~]# md5sum /etc/origin/master/ca.crt
b9692b2d6f948547d70e7913c1e166aa /etc/origin/master/ca.crt
Are you using the master
branch of openshift-ansible? If not, which version or branch?
What procedure was used to install this cluster? Curious if a single run of playbooks/byo/config.yml
against this inventory resulted in this state or if the cluster was configured in pieces.
If this is a new cluster I would recommend running uninstall to clean up the mismatched certificates / configuration and then reinstall.
ansible-playbook -i <inventory> playbooks/adhoc/uninstall.yml
I'm on freenode IRC as abutcher if you want to reach out there.
Hi @abutcher ,
I am using ``ansible 2.2.1.0
version
Which channel are you in IRC?
Thanks, Rahul
Hi @abutcher
Thanks for the steps. I uninstalled and installed back again. Everything worked except one error as below:
TASK [openshift_metrics : fail] ************************************************
fatal: [oc-master1]: FAILED! => {"changed": false, "failed": true, "msg": "'keytool' is unavailable. Please install java-1.8.0-openjdk-headless on the control node"}
to retry, use: --limit @/root/openshift-ansible/playbooks/byo/config.retry
PLAY RECAP *********************************************************************
localhost : ok=11 changed=0 unreachable=0 failed=0
oc-etcd1 : ok=112 changed=33 unreachable=0 failed=0
oc-etcd2 : ok=103 changed=27 unreachable=0 failed=0
oc-etcd3 : ok=103 changed=27 unreachable=0 failed=0
oc-master : ok=75 changed=8 unreachable=0 failed=0
oc-master1 : ok=551 changed=136 unreachable=0 failed=1
oc-master2 : ok=337 changed=86 unreachable=0 failed=0
oc-master3 : ok=337 changed=86 unreachable=0 failed=0
oc-node1 : ok=195 changed=55 unreachable=0 failed=0
oc-node2 : ok=195 changed=55 unreachable=0 failed=0
You have mail in /var/spool/mail/root
Do you want me to run uninstall.yml, install java package on all nodes (lb, 3 masters, 2 nodes and 3 etcds) and run config.yml again?
Thanks, Rahul
Besides, current NFS PVs are as below. If I modify storage from NFS side, how will it reflect changes on host?
[root@oc-master1 ~]# oc get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
logging-volume 10Gi RWO Retain Available 6m
metrics-volume 10Gi RWO Retain Available 6m
registry-volume 5Gi RWX Retain Bound default/registry-claim 6m
TASK [openshift_logging : copy] ************************************************
task path: /root/openshift-ansible/roles/openshift_logging/tasks/generate_configmaps.yaml:9
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 443, in _execute
self._task.post_validate(templar=templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 248, in post_validate
super(Task, self).post_validate(templar)
File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 373, in post_validate
value = templar.template(getattr(self, name))
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 427, in template
disable_lookups=disable_lookups,
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 383, in template
disable_lookups=disable_lookups,
File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 583, in do_template
res = j2_concat(rf)
File "<template>", line 9, in root
File "/usr/lib/python2.7/site-packages/ansible/plugins/filter/core.py", line 198, in from_yaml
return yaml.safe_load(data)
File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load
return load(stream, SafeLoader)
File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 71, in load
return loader.get_single_data()
File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 39, in get_single_data
return self.construct_document(node)
File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 48, in construct_document
for dummy in generator:
File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 398, in construct_yaml_map
value = self.construct_mapping(node)
File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 208, in construct_mapping
return BaseConstructor.construct_mapping(self, node, deep=deep)
File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 132, in construct_mapping
"found unacceptable key (%s)" % exc, key_node.start_mark)
ConstructorError: while constructing a mapping
in "<unicode string>", line 9, column 21:
number_of_shards: {{ es_number_of_shards | default ...
^
found unacceptable key (unhashable type: 'dict')
in "<unicode string>", line 9, column 22:
number_of_shards: {{ es_number_of_shards | default ...
^
fatal: [oc-master1]: FAILED! => {
"failed": true
}
MSG:
Unexpected failure during module execution.
Noting that the metrics and logging issues encountered are addressed by https://github.com/openshift/openshift-ansible/pull/3770 and https://github.com/openshift/openshift-ansible/pull/3792.
Hi @abutcher As per verification steps, running below command from master1 node but it gives error:
etcdctl -C https://oc-etcd1:2379,https://oc-etcd2:2379,https://oc-etcd3:2379 --ca-file=/etc/origin/master/master.etcd-ca.crt --cert-file=/etc/origin/master/master.etcd-client.crt --key-file=/etc/origin/master/master.etcd-client.key cluster-health
-bash: etcdctl: command not found
http://<lb_hostname>:9000
Hey @rahul334481, you'll need to install the etcd
package on the master to provide etcdctl
.
Thanks @abutcher
That worked.
As per documentation, if I install multiple masters (which I did) using HAProxy as Load balancer, i need to browse http://<lb_hostname>:9000
and check HAProxy's status but for some reason browser page times out.
Besides, I gave cluster admin priv to a user but for some reason user is unable to run "oc" commands:
[root@oc-master1 ~]# oc adm policy add-cluster-role-to-user cluster-admin ragarwal
-bash-4.2$ oc get nodes
error: Missing or incomplete configuration info. Please login or point to an existing, complete config file:
1. Via the command-line flag --config
2. Via the KUBECONFIG environment variable
3. In your home directory as ~/.kube/config
To view or setup config directly use the 'config' command.
-bash-4.2$
Hi @abutcher
Good Morning,
For some reason, Metrics URL is not working:
https://hawkular-metrics.apps.oc-master.domain.com/hawkular/metrics
Error: 503 Service Unavailable
No server is available to handle this request.
[root@oc-master1 ~]# oc logs hawkular-cassandra-1-4cns3 -n openshift-infra
The MAX_HEAP_SIZE envar is not set. Basing the MAX_HEAP_SIZE on the available memory limit for the pod (2000000000).
The memory limit is less than 2GB. Using 1/2 of available memory for the max_heap_size.
The MAX_HEAP_SIZE has been set to 953M
THE HEAP_NEWSIZE envar is not set. Setting to 800M based on the CPU_LIMIT of 8000. [100M per CPU core]
About to generate seeds
Trying to access the Seed list [try #1]
Trying to access the Seed list [try #2]
Trying to access the Seed list [try #3]
Setting seeds to be hawkular-cassandra-1-4cns3
Creating the Cassandra keystore from the Secret's cert data
Converting the PKCS12 keystore into a Java Keystore
Entry for alias cassandra successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled
[Storing /opt/apache-cassandra/conf/.keystore]
Building the trust store for inter node communication
Certificate was added to keystore
Certificate was added to keystore
Building the trust store for client communication
Certificate was added to keystore
Certificate was added to keystore
Generating self signed certificates for the local client for cqlsh
Generating a 4096 bit RSA private key
..........................................................................++
...........................................................................................................................................................................................................................................++
writing new private key to '.cassandra.local.client.key'
-----
Certificate was added to keystore
cat: /etc/ld.so.conf.d/*.conf: No such file or directory
getopt: invalid option -- 'R'
OpenJDK 64-Bit Server VM warning: Cannot open file /opt/apache-cassandra/logs/gc.log due to No such file or directory
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
INFO [main] 2017-03-31 14:53:26,777 YamlConfigurationLoader.java:85 - Configuration location: file:/opt/apache-cassandra-3.0.12-1/conf/cassandra.yaml
INFO [main] 2017-03-31 14:53:26,893 Config.java:451 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=LZ4Compressor; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@67e2d983; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=512; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-4cns3; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=null; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-4cns3; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=hawkular-cassandra-1-4cns3}; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO [main] 2017-03-31 14:53:26,894 DatabaseDescriptor.java:323 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2017-03-31 14:53:27,008 DatabaseDescriptor.java:430 - Global memtable on-heap threshold is enabled at 218MB
INFO [main] 2017-03-31 14:53:27,008 DatabaseDescriptor.java:434 - Global memtable off-heap threshold is enabled at 218MB
INFO [main] 2017-03-31 14:53:27,116 CassandraDaemon.java:434 - Hostname: hawkular-cassandra-1-4cns3
INFO [main] 2017-03-31 14:53:27,117 CassandraDaemon.java:441 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_121
INFO [main] 2017-03-31 14:53:27,117 CassandraDaemon.java:442 - Heap size: 916455424/916455424
INFO [main] 2017-03-31 14:53:27,117 CassandraDaemon.java:445 - Code Cache Non-heap memory: init = 2555904(2496K) used = 3596224(3511K) committed = 3604480(3520K) max = 251658240(245760K)
INFO [main] 2017-03-31 14:53:27,117 CassandraDaemon.java:445 - Metaspace Non-heap memory: init = 0(0K) used = 16394024(16009K) committed = 16777216(16384K) max = -1(-1K)
INFO [main] 2017-03-31 14:53:27,118 CassandraDaemon.java:445 - Compressed Class Space Non-heap memory: init = 0(0K) used = 1901488(1856K) committed = 2097152(2048K) max = 1073741824(1048576K)
INFO [main] 2017-03-31 14:53:27,118 CassandraDaemon.java:445 - Par Eden Space Heap memory: init = 671088640(655360K) used = 174483896(170394K) committed = 671088640(655360K) max = 671088640(655360K)
INFO [main] 2017-03-31 14:53:27,118 CassandraDaemon.java:445 - Par Survivor Space Heap memory: init = 83886080(81920K) used = 0(0K) committed = 83886080(81920K) max = 83886080(81920K)
INFO [main] 2017-03-31 14:53:27,118 CassandraDaemon.java:445 - CMS Old Gen Heap memory: init = 161480704(157696K) used = 0(0K) committed = 161480704(157696K) max = 161480704(157696K)
INFO [main] 2017-03-31 14:53:27,118 CassandraDaemon.java:447 - Classpath: /opt/apache-cassandra/conf:/opt/apache-cassandra/build/classes/main:/opt/apache-cassandra/build/classes/thrift:/opt/apache-cassandra/lib/ST4-4.0.8.jar:/opt/apache-cassandra/lib/airline-0.6.jar:/opt/apache-cassandra/lib/antlr-runtime-3.5.2.jar:/opt/apache-cassandra/lib/apache-cassandra-3.0.12-1.jar:/opt/apache-cassandra/lib/apache-cassandra-clientutil-3.0.12-1.jar:/opt/apache-cassandra/lib/apache-cassandra-thrift-3.0.12-1.jar:/opt/apache-cassandra/lib/asm-5.0.4.jar:/opt/apache-cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/apache-cassandra/lib/commons-cli-1.1.jar:/opt/apache-cassandra/lib/commons-codec-1.2.jar:/opt/apache-cassandra/lib/commons-lang3-3.1.jar:/opt/apache-cassandra/lib/commons-math3-3.2.jar:/opt/apache-cassandra/lib/compress-lzf-0.8.4.jar:/opt/apache-cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/apache-cassandra/lib/disruptor-3.0.1.jar:/opt/apache-cassandra/lib/ecj-4.4.2.jar:/opt/apache-cassandra/lib/guava-18.0.jar:/opt/apache-cassandra/lib/high-scale-lib-1.0.6.jar:/opt/apache-cassandra/lib/jackson-core-asl-1.9.2.jar:/opt/apache-cassandra/lib/jackson-mapper-asl-1.9.2.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar:/opt/apache-cassandra/lib/javax.inject.jar:/opt/apache-cassandra/lib/jbcrypt-0.3m.jar:/opt/apache-cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/jna-4.0.0.jar:/opt/apache-cassandra/lib/joda-time-2.4.jar:/opt/apache-cassandra/lib/json-simple-1.1.jar:/opt/apache-cassandra/lib/jstackjunit-0.0.1.jar:/opt/apache-cassandra/lib/libthrift-0.9.2.jar:/opt/apache-cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/logback-classic-1.1.3.jar:/opt/apache-cassandra/lib/logback-core-1.1.3.jar:/opt/apache-cassandra/lib/lz4-1.3.0.jar:/opt/apache-cassandra/lib/metrics-core-3.1.0.jar:/opt/apache-cassandra/lib/metrics-jvm-3.1.0.jar:/opt/apache-cassandra/lib/metrics-logback-3.1.0.jar:/opt/apache-cassandra/lib/netty-all-4.0.44.Final.jar:/opt/apache-cassandra/lib/ohc-core-0.4.3.jar:/opt/apache-cassandra/lib/ohc-core-j8-0.4.3.jar:/opt/apache-cassandra/lib/reporter-config-base-3.0.0.jar:/opt/apache-cassandra/lib/reporter-config3-3.0.0.jar:/opt/apache-cassandra/lib/sigar-1.6.4.jar:/opt/apache-cassandra/lib/slf4j-api-1.7.7.jar:/opt/apache-cassandra/lib/snakeyaml-1.11.jar:/opt/apache-cassandra/lib/snappy-java-1.1.1.7.jar:/opt/apache-cassandra/lib/stream-2.5.2.jar:/opt/apache-cassandra/lib/thrift-server-0.3.7.jar:/opt/apache-cassandra/lib/jsr223/*/*.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar
INFO [main] 2017-03-31 14:53:27,119 CassandraDaemon.java:449 - JVM Arguments: [-Dcassandra.commitlog.ignorereplayerrors=true, -Xloggc:/opt/apache-cassandra/logs/gc.log, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms953M, -Xmx953M, -Xmn800M, -ea, -Xss256k, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:StringTableSize=1000003, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=/opt/apache-cassandra/conf/hotspot_compiler, -javaagent:/opt/apache-cassandra/lib/jamm-0.3.0.jar, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=/opt/apache-cassandra/lib/sigar-bin, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/apache-cassandra/logs, -Dcassandra.storagedir=/opt/apache-cassandra/data, -Dcassandra-foreground=yes]
WARN [main] 2017-03-31 14:53:27,153 CLibrary.java:176 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN [main] 2017-03-31 14:53:27,154 StartupChecks.java:121 - jemalloc shared library could not be preloaded to speed up memory allocations
WARN [main] 2017-03-31 14:53:27,154 StartupChecks.java:153 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN [main] 2017-03-31 14:53:27,154 StartupChecks.java:178 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO [main] 2017-03-31 14:53:27,161 SigarLibrary.java:44 - Initializing SIGAR library
WARN [main] 2017-03-31 14:53:27,171 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
WARN [main] 2017-03-31 14:53:27,175 StartupChecks.java:246 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
WARN [main] 2017-03-31 14:53:27,183 StartupChecks.java:267 - Directory /cassandra_data/data doesn't exist
ERROR [main] 2017-03-31 14:53:27,185 CassandraDaemon.java:710 - Has no permission to create directory /cassandra_data/data
[root@oc-master1 ~]#
[root@oc-master1 ~]# oc get rc -n openshift-infra
NAME DESIRED CURRENT READY AGE
hawkular-cassandra-1 1 1 0 1d
hawkular-metrics 1 1 0 1d
heapster 1 1 0 1d
[root@oc-master1 ~]# oc delete pod/hawkular-cassandra-1-4cns3 -n openshift-infra
pod "hawkular-cassandra-1-4cns3" deleted
[root@oc-master1 ~]# oc delete pod/heapster-3u9kd -n openshift-infra
pod "heapster-3u9kd" deleted
[root@oc-master1 ~]# oc delete pod/hawkular-metrics-3677c -n openshift-infra
pod "hawkular-metrics-3677c" deleted
[root@oc-master1 ~]# oc get pods -n openshift-infra
NAME READY STATUS RESTARTS AGE
hawkular-cassandra-1-r2w3h 0/1 CrashLoopBackOff 2 1m
hawkular-metrics-4cy2n 0/1 Running 0 31s
heapster-5cha3 0/1 Running 0 46s
[root@oc-master1 ~]#
[root@oc-master1 ~]# oc logs hawkular-metrics-4cy2n -n openshift-infra
Certificate was added to keystore
[Storing hawkular-metrics.truststore]
Certificate was added to keystore
[Storing hawkular-metrics.truststore]
2017-03-31 15:04:49,259 INFO [org.jboss.as.repository] (ServerService Thread Pool -- 9) WFLYDR0001: Content added at location /opt/jboss/wildfly/standalone/data/content/ac/88acbcbaa9a9b7cab9ab888db9fa801b63d8dd/content
2017-03-31 15:04:49,277 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
2017-03-31 15:04:49,293 INFO [org.xnio] (MSC service thread 1-4) XNIO version 3.4.0.Final
2017-03-31 15:04:49,301 INFO [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.4.0.Final
2017-03-31 15:04:49,353 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.21.Final
2017-03-31 15:04:49,362 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 45) WFLYCLJG0001: Activating JGroups subsystem.
2017-03-31 15:04:49,368 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0001: Activating Infinispan subsystem.
2017-03-31 15:04:49,379 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
2017-03-31 15:04:49,379 INFO [org.jboss.as.security] (ServerService Thread Pool -- 58) WFLYSEC0002: Activating Security Subsystem
2017-03-31 15:04:49,380 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 51) WFLYNAM0001: Activating Naming Subsystem
2017-03-31 15:04:49,389 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 62) WFLYWS0002: Activating WebServices Extension
2017-03-31 15:04:49,387 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 48) WFLYJSF0007: Activated the following JSF Implementations: [main]
2017-03-31 15:04:49,410 INFO [org.jboss.as.connector] (MSC service thread 1-3) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
2017-03-31 15:04:49,430 INFO [org.jboss.as.security] (MSC service thread 1-5) WFLYSEC0001: Current PicketBox version=4.9.6.Final
2017-03-31 15:04:49,435 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 36) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
2017-03-31 15:04:49,440 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0018: Started Driver service with driver-name = h2
2017-03-31 15:04:49,508 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0003: Undertow 1.4.0.Final starting
2017-03-31 15:04:49,535 INFO [org.jboss.as.naming] (MSC service thread 1-5) WFLYNAM0003: Starting Naming Service
2017-03-31 15:04:49,535 INFO [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
2017-03-31 15:04:49,702 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0014: Creating file handler for path '/opt/jboss/wildfly/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
2017-03-31 15:04:49,704 INFO [org.jboss.as.ejb3] (MSC service thread 1-8) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 128 (per class), which is derived from thread worker pool sizing.
2017-03-31 15:04:49,704 INFO [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 32 (per class), which is derived from the number of CPUs on this host.
2017-03-31 15:04:49,729 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0012: Started server default-server.
2017-03-31 15:04:49,730 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0018: Host default-host starting
2017-03-31 15:04:49,806 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080
2017-03-31 15:04:49,812 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow AJP listener ajp listening on 0.0.0.0:8009
2017-03-31 15:04:49,815 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000001: Initializing mod_cluster version 1.3.3.Final
2017-03-31 15:04:49,829 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
2017-03-31 15:04:49,870 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
2017-03-31 15:04:50,006 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/wildfly/standalone/deployments
2017-03-31 15:04:50,012 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0027: Starting deployment of "hawkular-metrics.ear" (runtime-name: "hawkular-metrics.ear")
2017-03-31 15:04:50,116 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTPS listener https listening on 0.0.0.0:8443
2017-03-31 15:04:50,301 INFO [org.jboss.ws.common.management] (MSC service thread 1-6) JBWS022052: Starting JBossWS 5.1.5.Final (Apache CXF 3.1.6)
2017-03-31 15:04:51,192 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0207: Starting subdeployment (runtime-name: "hawkular-alerts-action-email.war")
2017-03-31 15:04:51,192 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0207: Starting subdeployment (runtime-name: "hawkular-metrics.war")
2017-03-31 15:04:51,196 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0207: Starting subdeployment (runtime-name: "hawkular-alerts.war")
2017-03-31 15:04:51,196 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "hawkular-metrics-alerter.war")
2017-03-31 15:04:51,196 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0207: Starting subdeployment (runtime-name: "hawkular-alerts-action-webhook.war")
2017-03-31 15:05:19,387 ERROR [org.jgroups.protocols.TCP] (TransferQueueBundler,ee,hawkular-metrics-4cy2n) JGRP000029: hawkular-metrics-4cy2n: failed sending message to hawkular-metrics-3677c (55 bytes): java.net.SocketTimeoutException: connect timed out, headers: FD: heartbeat, TP: [cluster_name=ee]
2017-03-31 15:05:19,407 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-5,ee,hawkular-metrics-4cy2n) ISPN000094: Received new cluster view for channel server: [hawkular-metrics-4cy2n|2] (1) [hawkular-metrics-4cy2n]
2017-03-31 15:05:19,409 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-5,ee,hawkular-metrics-4cy2n) ISPN000094: Received new cluster view for channel hawkular-metrics: [hawkular-metrics-4cy2n|2] (1) [hawkular-metrics-4cy2n]
2017-03-31 15:05:19,409 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-5,ee,hawkular-metrics-4cy2n) ISPN000094: Received new cluster view for channel web: [hawkular-metrics-4cy2n|2] (1) [hawkular-metrics-4cy2n]
2017-03-31 15:05:19,411 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-5,ee,hawkular-metrics-4cy2n) ISPN000094: Received new cluster view for channel hawkular-alerts: [hawkular-metrics-4cy2n|2] (1) [hawkular-metrics-4cy2n]
2017-03-31 15:05:19,708 ERROR [org.jgroups.protocols.TCP] (TransferQueueBundler,ee,hawkular-metrics-4cy2n) JGRP000029: hawkular-metrics-4cy2n: failed sending message to hawkular-metrics-3677c (70 bytes): java.net.SocketTimeoutException: connect timed out, headers: VERIFY_SUSPECT: [VERIFY_SUSPECT: ARE_YOU_DEAD], TP: [cluster_name=ee]
2017-03-31 15:05:22,515 WARN [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 73) Could not connect to Cassandra cluster - assuming is not up yet. Cause: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/172.30.218.252:9042 (com.datastax.driver.core.exceptions.TransportException: [hawkular-cassandra/172.30.218.252:9042] Cannot connect))
2017-03-31 15:05:22,515 WARN [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 73) [11] Retrying connecting to Cassandra cluster in [3000]ms...
2017-03-31 15:07:53,306 WARN [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 65) Could not connect to Cassandra cluster - assuming is not up yet. Cause: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: hawkular-cassandra/172.30.218.252:9042 (com.datastax.driver.core.exceptions.TransportException: [hawkular-cassandra/172.30.218.252:9042] Cannot connect))
2017-03-31 15:07:53,306 WARN [org.hawkular.alerts.engine.impl.CassCluster] (ServerService Thread Pool -- 65) [2] Retrying connecting to Cassandra cluster in [3000]ms...
[root@oc-master1 ~]#
[root@oc-master1 ~]# oc logs heapster-5cha3 -n openshift-infra
Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
Could not connect to https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 7. Status Code 000
'https://hawkular-metrics:443/hawkular/metrics/status' is not accessible [HTTP status code: 000. Curl exit code 7]. Retrying.
[root@oc-master1 ~]#
[root@oc-master1 ~]# oc get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
logging-volume 500Gi RWO Retain Bound openshift-infra/metrics-1 1d
metrics-volume 100Gi RWO Retain Bound logging/logging-es-0 1d
registry-volume 500Gi RWX Retain Bound default/registry-claim 1d
TASK [openshift_master : Create the ldap ca file if needed] ********************
failed: [oc-master1.dur.lulu.com] (item={u'bindDN': u'<USER>', u'kind': u'LDAPPasswordIdentityProvider', u'name': u'lulu_ldap_provider', u'bindPassword': u'<PASS>', u'url': u'ldap://rhds.dur.lulu.com:389/ou=People,dc=lulu,dc=com?uid', u'insecure': u'false', u'ca': u'ca-bundle.crt', u'attributes': {u'id': [u'dn'], u'preferredUsername': [u'uid'], u'email': [u'mail'], u'name': [u'cn']}, u'login': u'true', u'challenge': u'true'}) => {"checksum": "491c3da8ace6eb400e2380f1f7bff88e9a62bdc0", "failed": true, "gid": 0, "group": "root", "item": {"attributes": {"email": ["mail"], "id": ["dn"], "name": ["cn"], "preferredUsername": ["uid"]}, "bindDN": "<USER>", "bindPassword": "<PASS>", "ca": "ca-bundle.crt", "challenge": "true", "insecure": "false", "kind": "LDAPPasswordIdentityProvider", "login": "true", "name": "lulu_ldap_provider", "url": "ldap://rhds.dur.lulu.com:389/ou=People,dc=lulu,dc=com?uid"}, "mode": "0600", "msg": "src file does not exist, use \"force=yes\" if you really want to create the link: /etc/origin/master/tmpzXSp8Q", "owner": "root", "path": "/etc/origin/master/ca-bundle.crt", "size": 263111, "src": "tmpzXSp8Q", "state": "hard", "uid": 0}
changed: [oc-master2.dur.lulu.com] => (item={u'bindDN': u'<USER>', u'kind': u'LDAPPasswordIdentityProvider', u'name': u'lulu_ldap_provider', u'bindPassword': u'<PASS>', u'url': u'ldap://rhds.dur.lulu.com:389/ou=People,dc=lulu,dc=com?uid', u'insecure': u'false', u'ca': u'ca-bundle.crt', u'attributes': {u'id': [u'dn'], u'preferredUsername': [u'uid'], u'email': [u'mail'], u'name': [u'cn']}, u'login': u'true', u'challenge': u'true'})
changed: [oc-master3.dur.lulu.com] => (item={u'bindDN': u'<USER>', u'kind': u'LDAPPasswordIdentityProvider', u'name': u'lulu_ldap_provider', u'bindPassword': u'<PASS>', u'url': u'ldap://rhds.dur.lulu.com:389/ou=People,dc=lulu,dc=com?uid', u'insecure': u'false', u'ca': u'ca-bundle.crt', u'attributes': {u'id': [u'dn'], u'preferredUsername': [u'uid'], u'email': [u'mail'], u'name': [u'cn']}, u'login': u'true', u'challenge': u'true'})
NO MORE HOSTS LEFT *************************************************************
to retry, use: --limit @/root/openshift-ansible/playbooks/byo/config.retry
PLAY RECAP *********************************************************************
localhost : ok=10 changed=0 unreachable=0 failed=0
oc-etcd1.dur.lulu.com : ok=95 changed=5 unreachable=0 failed=0
oc-etcd2.dur.lulu.com : ok=87 changed=5 unreachable=0 failed=0
oc-etcd3.dur.lulu.com : ok=87 changed=5 unreachable=0 failed=0
oc-master.dur.lulu.com : ok=75 changed=4 unreachable=0 failed=0
oc-master1.dur.lulu.com : ok=211 changed=17 unreachable=0 failed=1
oc-master2.dur.lulu.com : ok=196 changed=19 unreachable=0 failed=0
oc-master3.dur.lulu.com : ok=196 changed=19 unreachable=0 failed=0
oc-node1.dur.lulu.com : ok=83 changed=6 unreachable=0 failed=0
oc-node2.dur.lulu.com : ok=83 changed=6 unreachable=0 failed=0
Closing since we were able to work around the original issue. Let's create new ones for specific problems if any are encountered. Thanks!
Thanks @abutcher for all support. I will update you with configuring my host with LDAP as I tried multiple efforts still getting x509 Certificate Signed by unknown authority.
Thanks, Rahul
Some of the best OpenShift SSL debugging information I've seen yet. Thanks for posting @abutcher, helped debug issues where internal certificates expired and the renew playbook failed.
Very helpful indeed, @moortimis @abutcher
when i use system:admin to login ,the oc need me to input password ,i already check my ca things,then not works ,when i input oc status Unable to connect to the server: x509: certificate signed by unknown authority
Hi Team,
Trying to implement OpenShift on 3 masters/2 nodes/3 etcds/1 LB
Ansible Script looks like below:
note: tbielawa edited above block to use code styling
ansible --version ansible 2.2.1.0 config file = /etc/ansible/ansible.cfg configured module search path = Default w/o overrides
Error while running ansible:
$ cat /etc/redhat-release
) OS: RHEL 7.3 running on vmware