Bad performance of the Dashboard page with big metrics data

[marla-singer]

If I go to apinf.io and open any of the following pages, loading takes 1-2 seconds: front page, catalog, organizations, API profile If I open Dashboard overview, loading takes 29 seconds If I open Details view for example for APInf Catalog REST API, loading takes 36 seconds

/cc: @bajiat

Goal:

• Research a possible problem of current situation with Dashboard page and the View Analytic page

[marla-singer]

Result

Now for getting data it sends one big request to Elasticsearch with a few sub-requests in the request body. After discussion with @shaliko we think up a couple scenarios how to improve the current situation.

As the first step is to separate one big request to several

1. Table "Errors statistic"

   • Make a separate request
   • Set the records limit as 6
   • Order by data
   • Take number of all errors for current period
   • When a user clicks on the link "Download all errors" then send an additional request to get all data

2. Table "Most frequent users"

   • Make a separate request
   • Set the records limit as 6
   • Order by calls
   • Take number of all users for current period
   • When a user clicks on the link "Download all users" then send an additional request to get all data

3. Charts "API Request Timeline" and "API Response time"

   • Make a separate request

4. Overview charts

   • Separate request for each chart OR
   • Request to get data for chart and another request to get data about total numbers (sum requests, general medial time and unique users number)

[marla-singer]

As the second step is to make pre-aggregated data for each day but It will touch some UI changes:

1. Change option of date range picker "Last 24 hours" to "Today". The charts will display data from midnight to current hour.

2. Change UI of Errors statistic table:

   • Instead of calls count use relation of success and unsuccess calls
   • Rethink about aggregation of one minute. The possible solution is to get last 6-10 records

3. Rethink about chart Unique users API. With pre-aggregated data we would have data for charts per day but not for a total number of a unique user. It would be a high calculated operation and would be better to display Average of unique users number per week

[marla-singer]

Follow up issues #3071 #3072

[bajiat]

Closing since follow-up issues have been created.