Kibana 7 dashboard with updated templates partially working

[mbuccarello]

Hi

with the new version is partially working.

From my point of view the error is here

Inside the buckets window a parameter is wrong or missing.

For example the traffic dashbaord to make it working I changed the value in the json template from

uriPath to uriPath.keyword, see the screnshot below

I'm going to check all dashboard and I update soon all the final results.

Thanks

Michele

[mbuccarello]

This is the summary about my test for traffic problems and memory.

Traffic and problem dashboard they need .keyword in the file and change "a" to "auto"

About memory dashboard I repeat, I have the same exact issue I told you about the index pattern field and I refreshed the list of item. Can be possible liberty is not emitting this information?

[donbourne]

@mbuccarello , I'm wondering if your elasticsearch index has fields that were created before you put the new template in place. If possible, stop your ELK stack, delete your logstash index from the elasticsearch/data directory, and restart. You also need to refresh your index in Kibana to make sure it sees the correct field definitions after they are in place.

Looking at your individual points...

1. When I look at the same visualization in my environment (with the template/dashboards currently posted in the repo) I see:

In your index pattern, if the field was created by the template, you should see:

2. Traffic and problem dashboard they need .keyword in the file and change "a" to "auto" -- I think at least the first part (.keyword) will be fixed if you get the right field definitions into elasticsearch/kibana. What problem are you seeing regarding your "a" to "auto" comment? I'm seeing all of the visualizations/searches appear as expected in my Kibana 7 traffic dashboard (including the "Liberty Access Logs Search")

3. memory data only is available if you're using an IBM JVM (it's not available if you're using HotSpot). Take a look in your data on the discover tab to see if you have any GC events...

[mbuccarello]

These was my steps before update logstash and other stuff:

1. stop logstash
2. delete the index from kibana
3. delete the previously imported dashboard and related object
4. delete index pattern
5. start logstash
6. create index pattern
7. wait 5 min refresh object

I have websphere liberty and IBM J9 logstash collector 1,0 , the type:liberty_gc no result. About uripath from my side it works only if I use uriPath.keyword. About auto it works if I use only auto instead of a.

[donbourne]

You also need to get elasticsearch to "forget" the fields that have been created with the wrong settings. If this is a sandbox environment I would recommend just deleting the directories under elasticsearch/data.

We've tried simulating having the wrong template, running with it, then having the right template (but not cleaning up the messed up fields from elasticsearch first), and we run into the same issue you're seeing (the need for .keyword) -- so I think the cleanup is still the issue.

re the garbage collection, which version of the JVM are you running with? On my test system I see this at the top of my console log from Liberty: Launching defaultServer (WebSphere Application Server 19.0.0.10/wlp-1.0.33.cl191020191002-0300) on IBM J9 VM, version 8.0.5.41 - pxa6480sr5fp41-20190919_01(SR5 FP41) (en_US)

re auto instead of a, I'll have a look -- I recall this issue in one of our dashboard level migrations in the past, but as mentioned the dashboards are working for me without change.

[donbourne]

also re the garbage collection, please ensure your time range you are looking at is long enough to encompass when you started the liberty server. Once the server is started, if there is no traffic, you may not be having any GC events being triggered.

[mbuccarello]

Hi,

just an update, the warning below could be the problem memory events arent sent?

TRAS4352W: Health Center agent was not found, so JVM monitoring and diagnostic information will not be available.

[donbourne]

Definitely -- the garbage collection events come from the health center agent. If that's not loading then you won't get GC events.

[donbourne]

@mbuccarello are you able to get the problems/traffic dashboards running as expected now (other than the memory events)?

[mbuccarello]

By manually modifying every json not starting from scratch

[mbuccarello]

What is the right way to enable health center agent?

[donbourne]

By manually modifying every json not starting from scratch

ok, I realize that if you can't start from scratch then modifying json can compensate. In future you may want to reindex the data (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) to be able to take advantage of dashboards or other updates we post.

[donbourne]

What is the right way to enable health center agent?

It should automatically be enabled. An issue was addressed in 19.0.0.5 ( https://www-01.ibm.com/support/docview.wss?uid=swg1PH11801 ) -- but if you are already using a more recent version of Liberty I would suggest opening a support ticket to get to the bottom of why the health center isn't loading as it should just work without special config.

[mbuccarello]

By manually modifying every json not starting from scratch

ok, I realize that if you can't start from scratch then modifying json can compensate. In future you may want to reindex the data (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) to be able to take advantage of dashboards or other updates we post.

I did a clean installation and tested the dashboards. I don't know why I have this problem with ELK 7.4

[donbourne]

@mbuccarello , do you mean you still have problems with the dashboard when you do a clean install on ELK 7.4, or were you meaning that the clean install works?

[mbuccarello]

I mean with a clean ELK installation I still have problems.