FAT Approval Test Summary for Publish a standalone Open Liberty Grafana dashboard to grafana.com

[kabicin]

1) Describe the test strategy & approach for this feature, and describe how the approach verifies the functions delivered by this feature. The description should include the positive and negative testing done, whether all testing is automated, what manual tests exist (if any) and where the tests are stored (source control). Automated testing is expected for all features with manual testing considered an exception to the rule.

To preface, the Grafana dashboard visualizes mpMetrics-2.x data for standalone Open Liberty servers. In our specific scenario we have chosen manual testing over automated testing for the following reasons.

Firstly, we have no effective way to manage or create automated tests for the Grafana dashboard since it is not a part of the Liberty product. In addition, automatic tests would be hard to control. Performing equality operations on time-series data that is often system-dependent (i.e. Process CPU Time could be lower for faster CPUs), would be difficult to analyze and makes such tests hard to create.

The manual test factors:

• Scalability: Viewed data on Grafana from more than one Liberty server. This helped us assess the validity of information being consumed by Prometheus, and thus, the integrity of our sample dashboard. A minimum of 3 cases were analyzed:
  • viewing metrics data from an Open Liberty instance built from source only
  • viewing metrics data from a Daily UBI Docker Open Liberty instance only
  • and lastly, viewing metrics data from all instances.
• Security: Ensure Prometheus is able to consume from both insecure and secure Liberty endpoints by simply toggling <mpMetrics authentication="false" /> in the server.xml.
• Connection Pool: The above tests will generate data for all metrics except for the Connection Pool ones. Using the Sample Daytrader App, this gives us connection pool data we need. There is one caveat however, to constrict the size of the connection pool to a low of 1 and high of 1 in order to force generation of "queued connection pool requests". We have confidence in results from this forced scenario, after discussion across the team.
• Stress: We ran multiple apps such as Daytrader and our own custom apps to view that graph lines increased or decrease according to the load of the Liberty servers. In some cases, we expected graph lines to stay relatively the same even when stress was applied.

2) Collectively as a team you need to assess your confidence in the testing delivered based on the values below. This should be done as a team and not an individual to ensure more eyes are on it and that pressures to deliver quickly are absorbed by the team as a whole.

0 - No automated testing delivered

We are confident that our manual testing is sufficient to maintain the current Grafana dashboard overtime. Our team collectively assessed the reliability of our custom Prometheus queries and overall dashboard through multiple live demos within meetings and also through email discussion. Showing the dashboard to the team in this case means sharing the dashboard to IBMers who have worked with or are currently working with MicroProfile Metrics and to the WebSphere Observability architect. We are aware that we will be the first to use the IBM ID, "openlibertyproject" through grafana.com and so the update process of this dashboard has been adjusted within our team's internal repository.

[dave-waddling]

Hi @kabicin,

Thanks for completing the Feature Test Summary. I agree that having no automated testing makes sense here and the manual testing seems more than sufficient. I'll add the FAT Focal Approval to the parent epic at this time.