Closed brianotte closed 4 years ago
NOTE: I did need to apply https://github.com/openshift/okd/issues/310 to all nodes to get fluentd pods to start.
More data for related to MASTER_NODE_0 (edited name) and fluentd logs included
OKD 4_5_cluster]$ oc get node MASTER_NODE_0
NAME STATUS ROLES AGE VERSION
MASTER_NODE_0 Ready master 55d v1.18.3
OKD 4_5_cluster]$ oc get pods --selector component=fluentd -o wide -n openshift-logging | grep MASTER_NODE_0
fluentd-2kfzj 1/1 Running 0 4m44s 10.128.0.32 MASTER_NODE_0 <none> <none>
OKD 4_5_cluster]$ oc logs fluentd-2kfzj
Setting each total_size_limit for 2 buffers to 41190554112 bytes
Setting queued_chunks_limit_size for each buffer to 4910
2020-09-01 18:38:51 +0000 [warn]: parameter 'pos_file_compaction_interval' in <source>
@type tail
@id container-input
path "/var/log/containers/*.log"
exclude_path ["/var/log/containers/fluentd-*_openshift-logging_*.log","/var/log/containers/elasticsearch-*_openshift-logging_*.log","/var/log/containers/kibana-*_openshift-logging_*.log"]
pos_file "/var/log/es-containers.log.pos"
pos_file_compaction_interval 1800
refresh_interval 5
rotate_wait 5
tag "kubernetes.*"
read_from_head true
@label @CONCAT
<parse>
@type "multi_format"
<pattern>
format json
time_format "%Y-%m-%dT%H:%M:%S.%N%Z"
keep_time_key true
time_type string
</pattern>
<pattern>
format regexp
expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
keep_time_key true
</pattern>
</parse>
</source> is not used.
NOTE: the following was required to get audit logs to ElasticSearch -- but the logs are not viewable in Kibana (yet):
Identified that "Pipelines" may be required so followed steps to enable....Section 5.3.2 SOURCE: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.5/html-single/logging/index#cluster-logging-viewing
Verification that data is getting to ElasticSearch: oc exec elasticsearch-cdm-f7d47soc-1-69c7659545-6hf6m -n openshift-logging -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca https://localhost:9200/_cat/indices?v
Defaulting container name to elasticsearch. Use 'oc describe pod/elasticsearch-cdm-f7d47soc-1-69c7659545-6hf6m -n openshift-logging' to see all of the containers in this pod.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 JOfd0dA2SGq8pb_PssUfNQ 1 1 0 0 522b 261b
green open .kibana_1892964590_zaphod_beeblebrox 6nTyxh4vQzqfPjP0l6R1TQ 3 1 1 0 8.4kb 4.2kb
green open .security IMQJdgIkTvKPUwgHR9oPaQ 1 1 5 2 54.9kb 27.5kb
green open audit-000001 NzLa1rQaQ9KoEGUvpedhOw 3 1 247377 0 455.1mb 227.5mb
Kibana only indicates: Couldn't find any Elasticsearch data
Re-configured all to add app, infra, and audit (as is default) and elasticsearch creates indexes -- but kibana only indicates that it sees app-* index or .kibana_1892964590_zaphod_beeblebrox.
There is no option to add infra- or audit-
Is this by design?
oc exec elasticsearch-cdm-bcl4thcz-1-78686f4cc6-7rvpq -n openshift-logging -- indices
Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-bcl4thcz-1-78686f4cc6-7rvpq -n openshift-logging' to see all of the containers in this pod.
Tue Sep 1 20:31:32 UTC 2020
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open audit-000001 PltlAIsPTZegNWFExmvGAw 3 1 151869 0 252 126
green open app-000001 YkkTcgpnRjCyZBksnoC8qA 3 1 9265 0 10 5
green open infra-000001 B5YGKoYqSXO5gEpeVU8MeQ 3 1 6108216 0 9238 4244
green open .kibana_1 kzLafkb2TNmeN9qXGNCyLw 1 1 0 0 0 0
green open .security xtF2MqjRQk-xPQ3VkOLEcQ 1 1 5 0 0 0
green open .kibana_1892964590_zaphod_beeblebrox kAsZThv-S-KWf4kIJCUxcw 1 1 1 0 0 0
Here is a screen capture for what is available (after sending audit, app, infra logs to elasticsearch). NOTE: There is no audit or infra to index.
Changed the title for this issue to accurately define the issue.
Uninstalled and working towards Splunk Connect for Openshift Logging: https://www.openshift.com/blog/splunk-connect-for-openshift-logging-part
Thank you.
Installed cluster logging as directed from this URL: https://docs.okd.io/latest/logging/cluster-logging-deploying.html#cluster-logging-deploy-cli_cluster-logging-deploying.
In OKD deployment -- the audit.log locations on masters exists in both locations:
Here is modified step 5b ( clo-instance.yaml ) that should only collect audit logs from https://docs.okd.io/latest/logging/cluster-logging-deploying.html#cluster-logging-deploy-cli_cluster-logging-deploying (NOTE: I tried a default clo-instance.yaml -- and app, infra, audit logs sent to elasticsearch -- see below examples -- but index not create-able in Kibana) :
cat step_5b_clo-instance.yaml
Describe the bug Audit logs got to Elasticsearch but cannot add index in Kibana.
Version Server Version: 4.5.0-0.okd-2020-08-12-020541 AWS Installer Provisioned Infrastructure.
How reproducible 100%
Log bundle Please let me know if needed (some or all -- and I can provide) -- but it is a large file:
du -s -h ./must-gather.local.1265493575915701287/ 2.8G ./must-gather.local.1265493575915701287/
Please let me know if more information is needed.
Thank you for your help.
Brian