Errors uploading dashboard to Openshift 3.11

[MastanaGuru]

I am trying to upload the kibana5_traffic.json to OCP 3.11 but getting errors In openshift the index pattern is project., I changed logstash- to project.* in the json and uploaded the file. In OCP we have EFK stack and Kibana version is 5.6.16.

In liberty(20.0.0.8) server.xml added below lines

<accessLogging logFormat='%R{W} %u %{my_cookie}C %s'/>
 <logging consoleSource="message,trace,accessLog,audit" consoleFormat="json" jsonAccessLogFields="logFormat"/>

In Kibana, on refresh in Management->Index Patterns -> project.* I see only below fields

ibm_datetime                    
ibm_datetime.raw                
ibm_messageId               
ibm_messageId.raw               
ibm_sequence    
ibm_sequence.raw    
ibm_serverName          
ibm_serverName.raw
ibm_threadId
ibm_threadId.raw
ibm_userDir
ibm_userDir.raw 

When I upload the json dashboard, I see below errors in Kibana

Saved Objects: Importing Liberty Message (liberty_messageGraph) failed: Could not locate that index-pattern-field (id: loglevel.keyword)
Saved Objects: Importing Liberty Access Log Response Code (liberty_accessLogResponseCodeGraph) failed: Could not locate that index-pattern-field (id: ibm_responseCode)
Saved Objects: Importing Liberty Access Log Elapsed Time (liberty_accessLogElapsedTimeGraph) failed: Could not locate that index-pattern-field (id: ibm_elapsedTime)
Saved Objects: Importing Liberty Access Log Average Elapsed Time (liberty_accessLogAverageElapsedTimeGraph) failed: Could not locate that index-pattern-field (id: ibm_elapsedTime)
Saved Objects: Importing Liberty Slow Hits (liberty_slowHits) failed: Could not locate that index-pattern-field (id: ibm_elapsedTime)
Saved Objects: Importing Liberty Slowest URLs (liberty_slowestURLs) failed: Could not locate that index-pattern-field (id: ibm_uriPath.keyword)
Saved Objects: Importing Liberty Trace (liberty_traceGraph) failed: Could not locate that index-pattern-field (id: loglevel.keyword)
Saved Objects: Importing Liberty Top URLs (liberty_topURLs) failed: Could not locate that index-pattern-field (id: ibm_uriPath.keyword)
Saved Objects: Importing Liberty Servers (Traffic) (AWYhv5KDN1XQo0uIQwxD) failed: Could not locate that index-pattern-field (id: ibm_serverName.keyword)
Saved Objects: Importing Liberty User Directories (Traffic) (AWYhvzNUN1XQo0uIQwwV) failed: Could not locate that index-pattern-field (id: ibm_userDir.keyword)
Saved Objects: Importing Liberty Access Log Metrics (liberty_accessLogMetrics) failed: Could not locate that index-pattern-field (id: ibm_elapsedTime)
Saved Objects: Importing Liberty Hosts Table (Traffic) (AWYhvu0tN1XQo0uIQwv4) failed: Could not locate that index-pattern-field (id: host.keyword)

[MastanaGuru]

Message as displayed in Kibana Discover

{
  "_index": "project.liberty--sbx--rest.8df738cb-c91d-11ea-940d-0050569ebd6a.2020.08.03",
  "_type": "com.redhat.viaq.common",
  "_id": "OGE1MTk5YjktMjhlOS00MmRkLWI1NWMtNzA4MjcxMjJlODYw",
  "_version": 1,
  "_score": null,
  "_source": {
    "type": "liberty_message",
    "host": "microprofile-1-vm889",
    "ibm_userDir": "/opt/ol/wlp/usr/",
    "ibm_serverName": "defaultServer",
    "message": "[Default Executor-thread-1] INFO com.xyz.liberty.EntityDetailsResource - EntityNm - 561118388A",
    "ibm_threadId": "00000024",
    "ibm_datetime": "2020-08-03T17:25:28.485-0400",
    "module": "SystemErr",
    "loglevel": "SystemErr",
    "ibm_sequence": "1596489928485_00000000001AD",
    "ext_thread": "Default Executor-thread-1",
    "docker": {
      "container_id": "e5343e1c4c472742799e1b98768dd71f9769e119b8d451374fa728cf728bf80f"
    },
    "kubernetes": {
      "container_name": "microprofile",
      "namespace_name": "liberty--sbx--rest",
      "pod_name": "microprofile-1-vm889",
      "container_image": "docker-registry.default.svc:5000/liberty--sbx--rest/microprofile@sha256:2328101fa8957933eb4b9cf2ac2960b92c3407dd318fc123282ab00cb6cf3ed2",
      "container_image_id": "docker-pullable://docker-registry.default.svc:5000/liberty--sbx--rest/microprofile@sha256:2328101fa8957933eb4b9cf2ac2960b92c3407dd318fc123282ab00cb6cf3ed2",
      "pod_id": "522a9623-d5cb-11ea-a453-0050569ebd6a",
      "labels": {
        "app": "microprofile",
        "deployment": "microprofile-1",
        "deploymentconfig": "microprofile"
      },
      "host": "vlrhoaq112.eqx10.iad.xyz.com",
      "master_url": "https://kubernetes.default.svc.cluster.local",
      "namespace_id": "8df738cb-c91d-11ea-940d-0050569ebd6a"
    },
    "level": "info",
    "hostname": "vlrhoaq112.eqx10.iad.xyz.com",
    "pipeline_metadata": {
      "collector": {
        "ipaddr4": "10.41.6.97",
        "ipaddr6": "fe80::14ca:3cff:fe83:2dda",
        "inputname": "fluent-plugin-systemd",
        "name": "fluentd",
        "received_at": "2020-08-03T21:25:28.509845+00:00",
        "version": "0.12.43 1.6.0"
      }
    },
    "@timestamp": "2020-08-03T21:25:28.486199+00:00",
    "viaq_msg_id": "OGE1MTk5YjktMjhlOS00MmRkLWI1NWMtNzA4MjcxMjJlODYw"
  },
  "fields": {
    "@timestamp": [
      1596489928486
    ],
    "pipeline_metadata.collector.received_at": [
      1596489928509
    ]
  },
  "highlight": {
    "type": [
      "@kibana-highlighted-field@liberty_message@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1596489928486
  ]
}

[Yushan-Lin]

Hi @MastanaGuru , If you're setting jsonAccessLogFields="logFormat" the JSON access log fields will only include the properties you specify in the accessLogging logFormat attribute. To make it work with our existing dashboards, you can either change jsonAccessLogFields="default" to only include, or change your logFormat to: <accessLogging logFormat='%h %H %A%B %m %p %q %{R}W %s %U'/>. For more info about jsonAccessLogFields: https://openliberty.io/blog/2020/07/30/json-logging-open-liberty-20008.html Information about accessLogFormat: https://openliberty.io/docs/20.0.0.8/access-logging.html Let me know if you still have questions

[donbourne]

@MastanaGuru , I think you may 3 problems.

• based on fact your dashboard json refers to logstash- and not project- , and the fact you're posting this issue in WASdev/sample.dashboards, I think you're using our standalone (non Kubernetes) dashboard. For OKD/OCP you should use the dashboards at https://github.com/OpenLiberty/open-liberty-operator/blob/master/deploy/dashboards/logging .
• if after refresh of your index pattern you don't see the fields related to the access log json it likely means you haven't yet generated any access log events on the Liberty side. That means Elasticsearch hasn't seen/mapped those fields yet, so Kibana doesn't know about them. Try hitting a url and check to see if you have records in your access.log file
• as @Yushan-Lin says, even if you had created access log fields you'd be missing the default ones that the traffic dashboard depends on. You could create at least one access log record with the default set of fields included, or remove the visualizations that rely on those fields from the dashboard.

[MastanaGuru]

@donbourne , @Yushan-Lin , Thanks that works, I used the RHOCP dashboards. But accesslogs is really filling up the storage, wondering if there is a way we can exclude some urls in accesslog setting, specifically health check and readiness probe.

[donbourne]

@MastanaGuru , I don't have a simple answer for how to filter the access logs. My first thought would be to set a grep filter in the Fluentd configuration, but cluster logging doesn't expose that in its configuration.

One thing you could do, which I know will sound like a hack, is to filter the output from Liberty's stdout itself. This would probably need to be a change to your Open Liberty image's command that starts the server and would look something like this:

./server run server1 | grep --line-buffered -v "/metrics" | grep --line-buffered -v "/health"

This should work because Liberty's JSON log is output to console with each JSON event on a separate line.