Closed julienlim closed 6 years ago
@sankarshanmukhopadhyay @brainfunked @r0h4n @nthomas-redhat @Tendrl/qe @Tendrl/tendrl_frontend @japplewhite @rghatvis@redhat.com @mcarrano
This dashboard proposal is ready for review. Note: API impact, module impact, etc. has to be filled out by someone else -- maybe @cloudbehl, @anmolbabu, or @anivargi.
Suggested Labels (for folks who have permissions to label the spec):
Row 1 Panel 3: Disks No platform support for disk status as such. This won't be supported now
Panel 6: IO Size I havn't seen a mention of this in the MVP. Also this stats is not provided by collectd plugin. So I prefer to defer this
Row 2 Panel 8: CPU Available? does this really make any sense?
Row 3 Panel 13: Growth Rate Panel 14: Time Remaining (Weeks) Does this really make any sense to display at the host level? Also MVP just talks about the projections at volume level only.
Panel 15: Disk Load Trend This is already covered in Row4 - typo?
Panel 16: Services Trend Can we get some clarity around this? Is it part of MVP?
Row 4 (Assumption: all stats in row 4 are aggregated on per host)
Panel 18: Disk Operations Trend Isn't this same as IOPs mentioned in Row 1? is there a difference?
Row 5 (Assumption: collected and displayed only for cluster network)
Panel 21: Bytes Sent and Received Trend Throughput is calculated from Bytes Sent and Received. Do we need to graph this again?
@nthomas-redhat
Row 1 Panel 3: Disks - Marked as FUTURE
Panel 6: IO Size - Marked as FUTURE
Row 2 Panel 8: CPU Available - This was for consistency with the other (memory, swap, capacity). It’s not critical and can be removed or made optional. I’ll make it as Optional.
Row 3 Panel 13: Growth Rate and Panel 14: Time Remaining (Weeks) Since disks are typically on a host (that get consumed for use by a volume), knowing when the volume runs out of space is not sufficient in helping Admins know where to plan to add capacity to and what kind of runway is needed. This is also related to forecast / projections for bricks that I mentioned in https://github.com/Tendrl/specifications/issues/230#issuecomment-321587511.
Panel 15: Disk Load Trend - typo (removed)
Panel 16: Services Trend I raised this a few times in BLR, and I'm suggesting this to have parity with the old Console (via Nagios plugin). This was the only we didn't address. The use scenario is that there's not easy way for Admins to know if their services/daemons die today or are still ok, and this is a means for monitoring their health. I will defer this to @japplewhite if this is part of the MVP.
Row 4 (Assumption: all stats in row 4 are aggregated on per host)
Panel 18: Disk Operations Trend - Isn't this same as IOPs mentioned in Row 1? is there a difference?
Row 5 - Network stats If we assume collected and displayed only for cluster network, is the same true when another network is being used by Cluster, e.g. replication network, CTDB, etc.? Why can’t we show it for all the interfaces and user can filter out what they don’t want (vs. making them have to manually add it in)? It’s easier for user to remove than add interfaces+metrics.
Panel 21: Bytes Sent and Received Trend Typo / Redundant not needed. Removed.
I've updated the spec above based on comments.
@sankarshanmukhopadhyay @brainfunked @r0h4n @nthomas-redhat @Tendrl/qe @Tendrl/tendrl_frontend @japplewhite @rghatvis@redhat.com @mcarrano
Here's a rough mockup of what this might look like:
@julienlim Would it make sense to merge Brick Capacity and Brick Capacity Used panels together and provide some information like 2.0 GiB / 19.0 GiB
? It would free a lot of space.
@julienlim any updates on this? need to close this issue
@r0h4n @fbalak @cloudbehl @nthomas-redhat @mbukatov
In reviewing the current Host Dashboard, there are multiple panels show brick utilization and capacity, i.e. Total Brick Capacity Utilization Trend, Total Brick Capacity Utilization, Total Brick Capacity Available, Brick Utilization, Brick Capacity, Brick Capacity Used -- making it confusing as it appears that some of the data may be redundant if a Tendrl host has only 1 brick.
In looking at monitoring-integration/etc/tendrl/monitoring-integration/grafana/dashboards/tendrl-gluster-hosts.json, it isn't clear what the 3 Panels (Brick Utilization, Brick Capacity, and Brick Capacity Used) panels are trying to actually showcase (i.e. what problems they are trying to solve) based on current labels used. Specifically, are they intended to show all the bricks on the host, or the top n bricks on the host, or something else? The labels for the panels don't tell a user one way or either what the intent was, nor is there accompanying description to help explain it.
My thoughts is that it is probably the Top Brick Consumers that should be shown (and in the following left-to-right order):
With this updated definition of the 3 bricks panels, it no longer makes sense to combine the 2 panels as @fbalak suggested, as the results will vary based on the criteria.
Having made this proposal on how to address these 3 bricks panels, is this considered a RFE, a Bug, or both (which I believe this to be)? This should be a fairly straightforward fix. Can we get this fixed for the upcoming release (if not the next one, the following one)?
@r0h4n I created a separate issue for the suggested changes to the 3 brick panels mentioned in my previous comments -- this is more of a bug and RFE that probably should be tracked as a separate issue. With the creation of the https://github.com/Tendrl/monitoring-integration/issues/324 issue, you can probably close this spec/issue.
Thanks, Ill close this one
Dashboard Spec - Host Dashboard
Display a default dashboard for a single Gluster storage node present in Tendrl that provides at-a-glance information about a single Gluster node that is part of a Gluster trusted storage pool that includes health and status information, key performance indicators (e.g. IOPS, throughput, etc.), and alerts that can highlight the Tendrl user's (e.g. Gluster Administrator) attention to potential issues in the host, volume, brick, and disk.
Problem description
A Gluster Administrator wants to be able to answer the following questions by looking at the cluster dashboard:
Use Cases
Uses Cases in the form of user stories:
As a Gluster Administrator, I want to view at-a-glance information about my Gluster host that includes health and status information, key performance indicators (e.g. IOPS, throughput, etc.), and alerts that can highlight my attention to potential issues in the host, volume, brick, and disk.
As a Gluster Administrator, I want to compare 1 or more metrics (e.g. IOPS, CPU, Memory, Network Load) across bricks within the host
Compare utilization (e.g. IOPS, capacity, etc.) across bricks within a host
Proposed change
Provide a pre-canned, default host dashboard in Grafana (that is initially launchable from the Tendrl UI, and eventually embed it into the Tendrl UI) that shows the following metrics rendered either in text or in a chart/graph depending on the type of metric being displayed below:
The Dashboard is composed of individual Panels (dashboard widgits) arranged on a number of Rows.
Note: The cluster and host name or unique identifier should be visible at all times, and user should be able to switch to another host.
Row 1
Panel 1: Health
Panel 2: Bricks
[FUTURE] Panel 3: Disks
Panel 4: Connections Trend
Panel 5: IOPS Trend
[FUTURE] Panel 6: IO Size
Row 2
Panel 7: CPU Utilization Trend
[OPTIONAL] Panel 8: CPU Available
Panel 9: Memory Used Trend
Panel 10: Memory Free
Panel 11: Swap Used Trend
Panel 10: Swap Free
Row 3
Panel 11: Capacity Utilization
Panel 12: Capacity Available
Panel 13: Growth Rate
Panel 14: Time Remaining (Weeks)
[FUTURE] Panel 15: Services Trend
Panel 15A: LVM thin pool metadata %
Panel 15B: LVM thin pool data usage %
Row 4 (Disk Metrics) - combined into Row 3
= Row 4 assumptions (for the disk metrics) - aggregation should depend on how many disks we expect on the host. If we expect <=8 disks per host, then no aggregation is needed, but if we expect higher, then aggregation per host is recommended.
Panel 16: Disk Load Trend
Panel 17: Disk Operations Trend
Panel 18: Disk IO Trend - Line Chart / Spark
Row 5 (Network Metrics)
Panel 20: Throughput Trend
Panel 22: Dropped Packets Trend
Panel 23: Errors Trend
Panel 24: Overruns Trend
Note: The dashboard layout for the panels and panels within the rows may need to alter based on implementation and actual visualization especially when certain metrics may need to be aligned together whether vertically or horizontally.
Alternatives
Create similar dashboard using PatternFly (www.patternfly.org) or d3.js components to show similar information within the Tendrl UI.
Data model impact:
TBD
Impacted Modules:
TBD
Tendrl API impact:
TBD
Notifications/Monitoring impact:
TBD
Tendrl/common impact:
TBD
Tendrl/node_agent impact:
TBD
Sds integration impact:
TBD
Security impact:
TBD
Other end user impact:
User will mostly interact with this feature via the Grafana UI, though access via Grafana API and Tendrl API is possible, but would require API calls to provide similar information.
Performance impact:
TBD
Other deployer impact:
Plug-ins required by Grafana will need to be packaged and installed with tendrl-ansible.
This (default) host dashboard will need to be automatically generated whenever a cluster is imported to be managed by Tendrl.
Developer impact:
TBD
Implementation:
TBD
Assignee(s):
Primary assignee: @cloudbehl
Other contributors: @anmolbabu, @anivargi, @julienlim, @japplewhite
Work Items:
TBD
Estimate:
TBD
Dependencies:
TBD
Testing:
Test whether health, status, and metrics displayed for a given host is correct and that the information is up-to-date as failures or other changes are observed on a given host.
Documentation impact:
Documentation should include information related to what's being displayed and explained for clarity if not immediately obvious from looking at the dashboard. This may include but not be limited to what the metrics refers to, the measurement unit, how to use or apply it to solving troubleshooting problems, e.g. healing / split brain issues, lost of quorum, etc.
References and Related GitHub Links: