cluster logging deployment does not allow viewing captured audit logs in Kibana

[brianotte]

Installed cluster logging as directed from this URL: https://docs.okd.io/latest/logging/cluster-logging-deploying.html#cluster-logging-deploy-cli_cluster-logging-deploying.

In OKD deployment -- the audit.log locations on masters exists in both locations:

• /var/log/kube-apiserver
• /var/log/openshift-apiserver

Here is modified step 5b ( clo-instance.yaml ) that should only collect audit logs from https://docs.okd.io/latest/logging/cluster-logging-deploying.html#cluster-logging-deploy-cli_cluster-logging-deploying (NOTE: I tried a default clo-instance.yaml -- and app, infra, audit logs sent to elasticsearch -- see below examples -- but index not create-able in Kibana) :

cat step_5b_clo-instance.yaml

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: "openshift-logging"
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    retentionPolicy:
      audit:
        maxAge: 21d
    elasticsearch:
      nodeCount: 3
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      storage:
        storageClassName: "gp2"
        size: 512G
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      replicas: 1
  curation:
    type: "curator"
    curator:
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      schedule: "30 3 * * *"
  collection:
    logs:
      type: "fluentd"
      fluentd: {}

Describe the bug Audit logs got to Elasticsearch but cannot add index in Kibana.

Version Server Version: 4.5.0-0.okd-2020-08-12-020541 AWS Installer Provisioned Infrastructure.

How reproducible 100%

Log bundle Please let me know if needed (some or all -- and I can provide) -- but it is a large file:

du -s -h ./must-gather.local.1265493575915701287/ 2.8G ./must-gather.local.1265493575915701287/

Please let me know if more information is needed.

Thank you for your help.

Brian

[brianotte]

NOTE: I did need to apply https://github.com/openshift/okd/issues/310 to all nodes to get fluentd pods to start.

[brianotte]

More data for related to MASTER_NODE_0 (edited name) and fluentd logs included

OKD 4_5_cluster]$ oc get node MASTER_NODE_0

NAME                                         STATUS   ROLES    AGE   VERSION
MASTER_NODE_0   Ready    master   55d   v1.18.3

OKD 4_5_cluster]$ oc get pods --selector component=fluentd -o wide -n openshift-logging | grep MASTER_NODE_0 fluentd-2kfzj 1/1 Running 0 4m44s 10.128.0.32 MASTER_NODE_0 <none> <none>

OKD 4_5_cluster]$ oc logs fluentd-2kfzj

Setting each total_size_limit for 2 buffers to 41190554112 bytes
Setting queued_chunks_limit_size for each buffer to 4910
2020-09-01 18:38:51 +0000 [warn]: parameter 'pos_file_compaction_interval' in <source>
  @type tail
  @id container-input
  path "/var/log/containers/*.log"
  exclude_path ["/var/log/containers/fluentd-*_openshift-logging_*.log","/var/log/containers/elasticsearch-*_openshift-logging_*.log","/var/log/containers/kibana-*_openshift-logging_*.log"]
  pos_file "/var/log/es-containers.log.pos"
  pos_file_compaction_interval 1800
  refresh_interval 5
  rotate_wait 5
  tag "kubernetes.*"
  read_from_head true
  @label @CONCAT
  <parse>
    @type "multi_format"
    <pattern>
      format json
      time_format "%Y-%m-%dT%H:%M:%S.%N%Z"
      keep_time_key true
      time_type string
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr)( (?<logtag>.))? (?<log>.*)$/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      keep_time_key true
    </pattern>
  </parse>
</source> is not used.

[brianotte]

NOTE: the following was required to get audit logs to ElasticSearch -- but the logs are not viewable in Kibana (yet):

Identified that "Pipelines" may be required so followed steps to enable....Section 5.3.2 SOURCE: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.5/html-single/logging/index#cluster-logging-viewing

Verification that data is getting to ElasticSearch: oc exec elasticsearch-cdm-f7d47soc-1-69c7659545-6hf6m -n openshift-logging -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca https://localhost:9200/_cat/indices?v

Defaulting container name to elasticsearch. Use 'oc describe pod/elasticsearch-cdm-f7d47soc-1-69c7659545-6hf6m -n openshift-logging' to see all of the containers in this pod.

health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1                   JOfd0dA2SGq8pb_PssUfNQ   1   1          0            0       522b           261b
green  open   .kibana_1892964590_zaphod_beeblebrox 6nTyxh4vQzqfPjP0l6R1TQ   3   1          1            0      8.4kb          4.2kb
green  open   .security                   IMQJdgIkTvKPUwgHR9oPaQ   1   1          5            2     54.9kb         27.5kb
green  open   audit-000001                NzLa1rQaQ9KoEGUvpedhOw   3   1     247377            0    455.1mb        227.5mb

Kibana only indicates: Couldn't find any Elasticsearch data

[brianotte]

Re-configured all to add app, infra, and audit (as is default) and elasticsearch creates indexes -- but kibana only indicates that it sees app-* index or .kibana_1892964590_zaphod_beeblebrox.

There is no option to add infra- or audit-

Is this by design?

oc exec elasticsearch-cdm-bcl4thcz-1-78686f4cc6-7rvpq -n openshift-logging -- indices

Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-bcl4thcz-1-78686f4cc6-7rvpq -n openshift-logging' to see all of the containers in this pod.
Tue Sep  1 20:31:32 UTC 2020
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   audit-000001                PltlAIsPTZegNWFExmvGAw   3   1     151869            0        252            126
green  open   app-000001                  YkkTcgpnRjCyZBksnoC8qA   3   1       9265            0         10              5
green  open   infra-000001                B5YGKoYqSXO5gEpeVU8MeQ   3   1    6108216            0       9238           4244
green  open   .kibana_1                   kzLafkb2TNmeN9qXGNCyLw   1   1          0            0          0              0
green  open   .security                   xtF2MqjRQk-xPQ3VkOLEcQ   1   1          5            0          0              0
green  open   .kibana_1892964590_zaphod_beeblebrox kAsZThv-S-KWf4kIJCUxcw   1   1          1            0          0              0

[brianotte]

Here is a screen capture for what is available (after sending audit, app, infra logs to elasticsearch). NOTE: There is no audit or infra to index.

[brianotte]

Changed the title for this issue to accurately define the issue.

[brianotte]

Uninstalled and working towards Splunk Connect for Openshift Logging: https://www.openshift.com/blog/splunk-connect-for-openshift-logging-part

Thank you.